Video processing method, server device, and computer program

Abstract

To provide a video processing method for increasing gifting opportunities of a viewing user, and a server device and a program for controlling video.SOLUTION: In a video processing method in a video distribution system 1, a server device 20 is configured to: generate video including animation of an avatar object of a distribution user based on motion of the distribution user; receive, via a communication line from a terminal device of a viewing user, a request signal generated on the basis of operation of the viewing user who is viewing the video, and for requesting giving of a first gift object to the avatar object; display, on the video, the first gift object given to the avatar object on the basis of the request signal; and display, on the basis of motion of the distribution user, when a determination is made that the avatar object has executed a predetermined action on one or more first gift objects displayed on the video, a second gift object on the video.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology disclosed in the present application relates to a video processing method for a video being viewed by a viewing user, The present invention relates to a server device and a computer program for controlling the same. [Background technology]

[0002] Conventionally, video distribution services that distribute videos to terminal devices via a network have been known. In this type of video distribution service, the avatar of the distribution user who distributes the video is The avatar object is displayed, and the viewing user and the distribution user who are watching the video are An environment where users can communicate with each other through messages and chats Boundaries are provided.

[0003] Furthermore, as a communication tool between viewing users and distribution users, The item that the viewing user acquired is assigned to the avatar object (broadcast user) in the video. It is disclosed that the act of giving gifts (gifting) will be realized electronically. .

[0004] In addition, video distribution services such as VirtualCast (registered trademark) The gift is then sent to the avatar object (broadcast user) by the user. An environment where an avatar object can grasp a video based on the user's movements. is provided (Non-Patent Document 1).

[0005] The above-mentioned Patent Document 1 and Non-Patent Document 1 are incorporated herein by reference in their entirety. can be done. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2012-120098 A [Non-patent literature]

[0007] [Non-Patent Document 1] "VirtualCast", [online], [searched September 10, 2019], Internet (URL: https: / / virtualcast.jp / ) Summary of the Invention [Problem to be solved by the invention]

[0008] The gifting disclosed in Patent Document 1 and Non-Patent Document 1 is a service that is provided by a viewing user and a distribution user ( Avatar objects through direct communication between the However, the above-mentioned patent document 1 and the non-patent document 2 are not applicable to the above-mentioned cases. The technology disclosed in Patent Document 1 displays gifts corresponding to gifting on a video. or an environment in which an avatar object can hold the gift on video. As such, gifting opportunities for viewers are still limited.

[0009] Gifting is a method in which viewers give money to service providers who provide video streaming services. After purchasing a gift by paying a fee, the gift is given to the distribution user (avatar object) In terms of the management of video distribution services, From this perspective, increasing the gifting opportunities for viewers is a great advantage for service providers. This can be said to be a challenge.

[0010] Therefore, some embodiments disclosed in this application may be used to provide a gift to a viewing user. A video processing method, a server device, and a computer program for increasing video opportunities do. [Means for solving the problem]

[0011] According to one aspect, a video processing method includes: A video processing method executed by a plurality of processors, A generating step of generating a video including an animation of an avatar object of the broadcast user. and the avatar object is generated based on an operation of a user who watches the video. a request signal for requesting the viewing user to grant a first gift object to the viewing user; a receiving step of receiving the request signal from a terminal device of the user via a communication line; The first gift object given to the avatar object is displayed on the video. a first display step of displaying the video on the basis of an action of the broadcast user; The avatar object is assigned to one or more of the first gift objects. When it is determined that the action has been executed, the first A second gift object different from the gift object is displayed on the video. A display step.

[0012] According to one embodiment, a server device includes one or more processors, the processors being configured to: A video including animation of the avatar object of the broadcast user based on the user's actions. The avatar image is generated based on an operation of a user who watches the video. a request signal for requesting that a first gift object be given to the first object; The request signal is received from the terminal device of the viewing user via a communication line, and the avatar is The first gift object given to the target object is displayed on the video. and displaying one or more of the above-mentioned on the video based on the operation of the distribution user. The avatar object performed a given action on a gift object. When it is determined by the terminal device of the distribution server that the predetermined action is executed, A second gift object different from the first gift object is displayed on the video. It is meant to show.

[0013] According to one embodiment, a server device includes one or more processors, the processors being configured to: A video including animation of the avatar object of the broadcast user based on the user's actions. The information for generating the moving image is transmitted to the terminal device of the distribution user who generates the moving image or the viewing user. the video is generated based on an operation of the viewing user to the terminal device of the viewing user, Requesting the granting of a first gift object to the avatar object. A request signal is received from the terminal device of the viewing user via a communication line, and based on the request signal Regarding the first gift object granted to the avatar object based on the The information on the first gift object to be displayed on the video of the broadcast user a terminal device of the viewing user, and based on the operation of the distribution user, The avatar is displayed on the video with respect to one or more of the first gift objects. The terminal device of the distribution server determines that the object has executed a predetermined action. When the predetermined action is performed on the first gift object, the second gift object is different from the first gift object. Information regarding a second gift object is displayed on the video. The terminal device of the distribution user or the terminal device of the viewing user to be displayed. do.

[0014] According to one embodiment, a computer program is provided that is executed by one or more processors. Thus, the animation of the avatar object of the broadcast user based on the action of the broadcast user is A video including the video is generated based on an operation of a user who watches the video, A request to grant a first gift object to an avatar object. A signal is received from a server device or a terminal device of the viewing user via a communication line, and the request The first gift object granted to the avatar object based on the request signal. The video is displayed on the basis of the action of the broadcast user. The avatar object is displayed for one or more of the first gift objects. When it is determined that a predetermined action has been executed, Displaying a second gift object different from the first gift object on the video. The present invention causes the processor to function in this manner.

[0015] According to one embodiment, a computer program is provided that is executed by one or more processors. Thus, the animation of the avatar object of the broadcast user based on the action of the broadcast user is The information for generating a video including the video is sent to a server device that generates the video or to a user who views the video. The video is generated based on an operation of the user who watches the video, and the video is transmitted to a terminal device. A request to grant a first gift object to an avatar object. A signal is received from a server device or a terminal device of the viewing user via a communication line, and the request The first gift object granted to the avatar object based on the request signal. The first gift object is displayed on the video. a server device or a terminal device of the viewing user, and based on the operation of the distribution user, The avatar is displayed on the video with respect to one or more of the first gift objects. When it is determined that the object has executed a predetermined action, the predetermined action is executed. information about a second gift object that is different from the first gift object that was executed; The server device or the and transmitting the information to a terminal device of the viewing user.

[0016] According to one embodiment, a computer program is provided that is executed by one or more processors. Thus, the animation of the avatar object of the broadcast user based on the action of the broadcast user is In order to generate a video including the video, information about the video is transmitted to a server device or the distribution user. and is generated based on an operation of a viewing user who views the video; A request to grant a first gift object to an avatar object. A signal is transmitted via a communication line, and the avatar object is notified based on the request signal. to display the first gift object on the video. The information regarding the first gift object is transmitted to the server device or the terminal device of the broadcast user. and based on the action of the broadcast user, one or more front images displayed on the video. The avatar object executes a predetermined action on the first gift object. When the terminal device of the distribution server determines that the predetermined action has been performed, A second gift object different from the first gift object is displayed on the video. information regarding the second gift object to be displayed on the server device or and causing the processor to function to receive from the terminal device of the broadcast user. .

[0017] According to one embodiment, a computer program is provided that is executed by one or more processors. Thus, the animation of the avatar object of the broadcast user based on the action of the broadcast user is A video including the video is received from a server device or a terminal device of the broadcast user, and the video is viewed. A first gimmick is generated based on an operation of a viewing user, and the first gimmick is applied to the avatar object. a request signal for requesting that a soft object be provided is transmitted via a communication line; The first gift object granted to the avatar object based on the request signal. The video displaying the object is received from the server device or the terminal device of the broadcast user. and based on the action of the distributor user, one or more of the first When the avatar object performs a predetermined action on the gift object, When it is determined by the terminal device of the distribution server that the predetermined action has been executed, The animation showing a second gift object different from the first gift object. The processor receives the image from the server device or the terminal device of the broadcast user. This is what makes the system function. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a video distribution system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the terminal device (server device) illustrated in FIG. [Diagram 3] FIG. 3 is a block diagram illustrating an example of functions of the terminal device illustrated in FIG. [Figure 4] FIG. 4 is a diagram showing an example in which a first gift object is displayed on a video including animation of a broadcast user's avatar object. [Diagram 5] FIG. 5 is a diagram showing a case where a first gift object is displayed on a video including animation of a broadcast user's avatar object, and two first collision regions overlap. [Figure 6] FIG. 6 is a diagram showing an example in which a second gift object is displayed on a video including animation of a broadcast user's avatar object. [Figure 7] FIG. 7 is a diagram showing an example in which a second gift object is displayed on a video including animation of a broadcast user's avatar object. [Figure 8] FIG. 8 is a diagram showing an example in which an effect object is displayed on a video including animation of a broadcast user's avatar object. [Figure 9] FIG. 9 is a block diagram illustrating an example of the functions of the server device illustrated in FIG. [Figure 10] FIG. 10 is a flow diagram showing an example of the operation performed in the video distribution system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Various embodiments of the present invention will be described below with reference to the accompanying drawings. Components throughout the drawings are labeled with the same reference numerals. Please note that some elements may be omitted in different drawings for ease of illustration. Furthermore, the accompanying drawings are not necessarily drawn to scale. Furthermore, the term application refers to software or programs. A program is a command to a computer that will produce a certain result. It is sufficient that the combination of the above-mentioned components is capable of producing the desired effect.

[0020] 1. Video distribution system configuration The video distribution system 1 disclosed in the present application is, in brief, a distribution unit that distributes The user uses the terminal device to access data regarding the operation of the broadcast user and / or the broadcast user's The avatar is generated based on the voice data of the user's speech and singing. A user watches a video including animation of an object via a communication line. This enables the content to be distributed to the user's terminal device.

[0021] FIG. 1 is a block diagram showing an example of a configuration of a video distribution system 1 according to an embodiment. As shown in FIG. 1, a video distribution system 1 includes a plurality of video distribution systems connected to a communication line (not shown). The system includes a terminal device 10 and one or more server devices 20 connected to a communication line. Each terminal device 10 is connected to one or more server devices 20 via a communication line. To be continued.

[0022] In addition, communication lines include mobile phone networks, wireless LANs, fixed telephone networks, the Internet, and This includes, but is not limited to, EtherNet / IP and / or EtherNet / IP. can.

[0023] 1-1. Terminal device 10 As shown in FIG. 1, a plurality of terminal devices 10 are terminal devices 1 of one or more broadcast users. 0 (here, terminal device 10A) and one or more viewing user terminal devices 10 (here, Each terminal device 10 has a common configuration. By having such a configuration, a terminal device (terminal device 10 of a distribution user) for distributing videos and The terminal device 10 may be either a terminal device for viewing a video or a terminal device for viewing a video (a terminal device 10 of a viewing user). That is why.

[0024] When each terminal device 10 operates as a broadcast user's terminal device 10, By executing the video distribution application, the user's actions can be monitored. Acquire data and / or audio data regarding speech or singing uttered by the broadcast user A virtual character (avatar object) that changes according to the acquired data A moving image including the animation of the above is generated, and the moving image is transmitted to the server device 20 via a communication line. The video can be transmitted to the server device 20 (distribution server device 20C). The server device 20C) transmits the installed video viewing application via a communication line. The terminal device 10 for receiving the video (the terminal device 10 of the viewing user) executes the application. It will be distributed.

[0025] Furthermore, when each terminal device 10 operates as a terminal device 10 of a broadcast user, The server device 20 transmits the broadcast user (avatar) to the broadcast user (avatar) via a communication line based on the operation of the viewing user. Give the gift object (first gift object) to the target object (second gift object) One or more request signals (one viewer user terminal device 1 When multiple request signals are sent from the viewer user terminal device 10, and when multiple request signals are sent from the viewer user terminal device 10, The video signal is received from the terminal device 10 of the viewing user who is viewing the video. When the request is received, each terminal device 10 receives the request from the server device 20 (information processing server device 20B). and receiving a request signal, the first A gift object can be displayed. A first gift object can be displayed on the video. A specific method for displaying will be described later.

[0026] Furthermore, when each terminal device 10 operates as a terminal device 10 of a broadcast user, Based on the action of the broadcast user, one or more first gift objects displayed on the video are The terminal device 10 determines that the avatar object has performed a predetermined action on the object. In this case, the terminal device 10 performs the predetermined action on the first gift object. A second gift object different from the original gift object can be displayed on the video. Details of the specified action, details of how the specified action is judged, and the second gift of the A specific method for displaying the object will be described later.

[0027] On the other hand, each terminal device 10 is a terminal device for viewing videos (a terminal device of a viewing user). If the application is installed, it may be necessary to run the video viewing application. Thus, the video distributed by the terminal device 10 of the distribution user is transmitted to the server device 20 (distribution server The signal can be received via the server device 20C.

[0028] Furthermore, when each terminal device 10 operates as a terminal device of a viewing user, By executing the application, the broadcast user (the application) can Gifting the first gift object to the butter object ) to the server device 20 (gift API server device 20A). This can be done.

[0029] Furthermore, when each terminal device 10 operates as a terminal device of a viewing user, By executing the application, the viewing user can change to a broadcasting user (avatar object) The server device 20 (comment API server device 2 0D).

[0030] The video distribution application and video viewing application mentioned above are integrated Either as one combined application or as separate applications. It can be installed and executed on each terminal device 10.

[0031] Each terminal device 10 may be any terminal device capable of performing such operations, Smartphones, tablets, mobile phones (feature phones) and / or personal computers The present invention can include, but is not limited to, a computer, etc.

[0032] 1-2. Server device 20 As shown in FIG. 1, the server device 20 is a dynamic system in which a plurality of server devices (to be described later) are combined. It can also be called an image distribution system.

[0033] The server device 20 receives the avatar object transmitted from the broadcast user's terminal device 10. Videos including animations can be distributed to the terminal devices 10 of each viewing user.

[0034] Furthermore, the server device 20 provides the avatar object with a first gift object. One or more request signals (from one viewing user terminal) requesting the granting of a gift (gifting) When multiple request signals are sent from the end device 10, or when multiple request signals are sent from the viewer user terminal devices 10, (including both cases where a request signal is sent from the terminal device 10 of each viewing user) In addition, the server device 20 can transmit information about the request signal to the broadcast user. After transmitting the message to the terminal device 10, the broadcast user's terminal device 10 uploads the animation of the avatar object. and displaying a first gift object corresponding to the request signal on a video including the animation. The server device 20 generates a video in which the first gift object is displayed. The broadcast user can also receive the broadcasted video) from the terminal device 10 of the broadcast user.

[0035] Furthermore, based on the action of the broadcast user, one or more first graphics displayed on the video are displayed. When the avatar object executes a predetermined action on the soft object, the terminal device When the terminal device 10 (the broadcast user's terminal device 10) judges that A second gift object that is different from the first gift object that has had a given action performed on it. When the second gift object is displayed on the video, the server device 20 The terminal device 10 can also receive the recorded video.

[0036] Furthermore, the server device 20 receives a notification from the viewing user to the distributor user (avatar object). The information regarding the comment to be posted can be received from the terminal device 10 of the viewing user, and The comment can be distributed to viewing users who are viewing the same video.

[0037] To perform such operations, the server device 20, in one embodiment, Gift A is connected via a communication line (including a wireless line and / or a wired line, not shown). A PI server device 20A, an information processing server device 20B, a distribution server device 20C, and a comment server device 20C. The API server device 20D may be included.

[0038] The gift API server device 20A creates an avatar object based on the operation of the viewing user. one or more requests to grant a first gift object to A number of request signals can be received from the terminal device 10 of each viewing user. The gift API server device 20A detects from the request signal the viewing user who performed the gifting act. User identification information (serial number of the terminal device 10 of each viewing user and ID information, etc.), information about the first gift object that was gifted (type, name Information on the name, color, size, etc. of the gifted avatar object, Identification information of the broadcast user (serial number of the terminal device 10 of the broadcast user and and at least read out the personal information (ID information, etc.) and process the read out information automatically or through an information processing server. When an inquiry is received from the information processing server device 20B, the can be done.

[0039] The information processing server device 20B automatically or by itself inquires about the gift API service. The gift API server device 20A receives the information read by the gift API server device 20A from the gift API server device 20A. The information can be transmitted to the terminal device 10 of the broadcast user. 0B can have a table (database) for each delivery user, and the Gift API service The server device 20A reads out information about the first gift object, and stores it in the It may be stored in a table.

[0040] Furthermore, the information processing server device 20B stores the identification information of the viewing user and the first gate. For the avatar object (broadcast user), refer to the information about the All or part of the information regarding which viewing user gave what first gift object can transmit a part of the comment information to the comment API server device 20D.

[0041] Furthermore, the information processing server device 20B generates an avatar for the first gift object. - The object is determined to have performed a predetermined action and the second gift object is The information displayed above is received from the broadcast user's terminal device 10, and the second gift Comment information about the object being displayed on the video is sent to the comment API server. It may also include functionality such as transmitting to device 20D.

[0042] The distribution server device 20C receives the video (first gift) transmitted from the terminal device 10 of the distribution user. a video in which a gift object is displayed and a video in which a second gift object is displayed, The content (including the content) can be distributed to the terminal device 10 of each viewing user.

[0043] The comment API server device 20D receives comments of the viewing user from the terminal device 10 of the viewing user. The comment can be sent to all viewers who are watching the same video. It can be distributed to users.

[0044] In addition, the comment API server device 20D receives from the information processing server device 20B, Which viewing user gave what first gift object to the avatar object (broadcast user)? Comments corresponding to all or part of the information about whether the object was given to the same video It can be distributed to each viewing user who is viewing.

[0045] Furthermore, the comment API server device 20D receives from the information processing server device 20B Based on the comment information regarding the second gift object appearing on the video, The comment corresponding to the comment information is sent to each viewing user who is viewing the same video. It can also be distributed to

[0046] In the example shown in FIG. 1, for the purpose of distributing the load, the server device 20 A gift API server device 20A, an information processing server device 20B, a distribution server device 20C, and The configuration shown has four server devices, namely, a comment API server device 20D and a comment API server device 20C. However, at least one of these four servers is not the same as the remaining servers. For example, the information processing server device 20B may be integrated with any one of the information processing server devices. It may be integrated with distribution server device 20C.

[0047] 2. Hardware configuration of each device Next, an example of the hardware configuration of each of the terminal device 10 and the server device 20 will be described. He explains.

[0048] 2-1. Hardware configuration of terminal device 10 An example of the hardware configuration of each terminal device 10 will be described with reference to FIG. 1 is a block diagram showing an example of a hardware configuration of the terminal device 10 (server device 20) shown in FIG. (Note that in FIG. 2, the reference numerals in parentheses indicate the server device 2 as described later.) 0.)

[0049] As shown in FIG. 2, each terminal device 10 mainly comprises a central processing unit 11, a main memory device 12, and , an input / output interface 13, an input device 14, an auxiliary storage device 15, and an output device 16. These devices are connected to each other by a data bus and / or a control bus. It is being done.

[0050] The central processing unit 11 is called a "CPU" and is stored in the main memory 12. Calculation is performed on the instructions and data stored in the main memory 12. Furthermore, the central processing unit 11 receives input data from an input device via an input / output interface 13. The terminal device 10 can control the terminal 14, the auxiliary storage device 15, and the output device 16. , may include one or more such central processing units 11.

[0051] The main memory device 12 is referred to as a "memory", and the input device 14 and the auxiliary memory device 1 5 and the communication network 30 (server device 20, etc.) via the input / output interface 13. The main memory stores the instructions and data executed by the central processing unit 11, as well as the results of the operations performed by the central processing unit 11. The device 12 includes a RAM (random access memory), a ROM (read only memory), and and / or flash memory, etc., without being limited thereto.

[0052] The auxiliary storage device 15 is a storage device having a larger capacity than the main storage device 12. Specific applications (video distribution applications, video viewing applications, etc.) Stores instructions and data (computer programs) that make up the web browser, etc. These instructions and data (common data) are stored in the memory and controlled by the central processing unit 11. A program (computer program) is transmitted to the main memory device 12 via the input / output interface 13. The auxiliary storage device 15 may be a magnetic disk device and / or an optical disk device. These may include, but are not limited to:

[0053] The input device 14 is a device for inputting data from the outside, and includes a touch panel, buttons, keys, etc. This includes, but is not limited to, a board, a mouse, and / or a sensor. As described below, one or more cameras and / or one or more microphones The sensors may include, but are not limited to:

[0054] The output device 16 may include a display device, a touch panel, and / or a printer device. These may include, but are not limited to:

[0055] In such a hardware configuration, the central processing unit 11 stores the data in the auxiliary storage device 15. The instructions and data that make up a specific application stored in the The input data is sequentially loaded into the main memory 12 and the loaded instructions and data are operated. Controlling an output device 16 via an output interface 13 or via an input / output interface The device 13 and other devices (for example, the server device 20 and other terminal devices 11 and 12) are connected to the communication network 30 via the communication network 30. It is possible to send and receive various information between the device and the Internet (such as ISP, ISP, etc.).

[0056] As a result, the terminal device 10 executes the specific application that has been installed. This allows you to perform the operations necessary to generate videos, send (distribute) and / or receive videos. The actions required for the gifting act, the actions required for the gifting act, and the specified actions described below are performed. The judgment regarding the quality of the comment, the actions required to send the comment (and various other actions detailed later) (including operations) can be performed.

[0057] The terminal device 10 may be used in place of the central processing unit 11 or together with the central processing unit 11. or more microprocessors and / or graphics processing units The device may include a graphics processing unit (GPU).

[0058] 2-2. Hardware configuration of server device 20 An example of the hardware configuration of each server device 20 will be described with reference to FIG. Server device 20 (gift API server device 20A, information processing server device 20B, distribution server The hardware configuration of the device 20C and the comment API server device 20D is, for example, For example, the same hardware configuration as that of each terminal device 10 described above can be used. Therefore, the reference numerals for the components of each server device 20 are collectively shown in FIG. is shown in the arc.

[0059] As shown in FIG. 2, each server device 20 mainly includes a central processing unit 21 and a main memory device 22. an input / output interface 23, an input device 24, an auxiliary storage device 25, and an output device 26 These devices are connected to each other by a data bus and / or a control bus. It has been done.

[0060] Central processing unit 21, main memory device 22, input / output interface 23, input device 24, auxiliary The storage device 25 and the output device 26 are respectively included in the terminal device 10 described above. A processing unit 11, a main memory unit 12, an input / output interface 13, an input unit 14, an auxiliary memory unit The input device 15 and the output device 16 may be substantially the same.

[0061] In such a hardware configuration, the central processing unit 21 stores the data in the auxiliary storage device 25. The instructions and data that make up a specific application stored in the The input data is sequentially loaded into the main memory 22 and the loaded instructions and data are operated. Controlling an output device 26 via an output interface 23 or alternatively, an input / output interface Through the device 23 and the communication line 30, various communication devices (such as the terminal devices 10, etc.) can be connected to each other. It is possible to send and receive information.

[0062] As a result, the server device 20 receives the data generated and transmitted by the terminal device 10 of each broadcast user. The operations required to distribute the video to the terminal device 10 of each viewing user, Operation required to receive a gifting action transmitted from the device 10, The operations required to generate the required information for generating the video, and the transmission (distribution) of the video ) and / or perform operations necessary to receive the information (including various operations described in detail below) It is possible.

[0063] In addition, the server device 20, instead of the central processing unit 21 or together with the central processing unit 21, One or more microprocessors and / or graphics processing units The processor may include a graphics processing unit (GPU).

[0064] 3. Functions of each device Next, an example of the detailed functions of the terminal device 10 and the server device 20 will be described. do.

[0065] 3-1. Functions of terminal device 10 An example of the functions of the terminal device 10 will be described with reference to FIGS. 3 to 8. FIG. 4 is a block diagram showing an example of the functions of the terminal device 10 shown in FIG. The first gift object appears on the video containing the animation of the avatar object. FIG. 5 is a diagram showing an example of a display. The first gift object appears on the video containing the first collision object and the second collision object. 6 and 7 are diagrams showing cases where the regions overlap. FIG. 1 shows an example of a second gift object displayed on a video including a gift animation. Figure 8 shows the effect of adding an animation of the broadcast user's avatar object to a video. FIG. 13 is a diagram showing an example in which an effect object is displayed.

[0066] As shown in FIG. 3, the terminal device 10 mainly includes a video generating unit 100, a display unit 110, a determination unit 120, storage unit 130, user interface unit 140, gift storage unit 150, and communication Includes unit 160.

[0067] (1) Video generation unit 100 The video generation unit 100 generates data on the broadcast user's actions and / or Based on the voice data of the spoken or sung utterances, the avatar object of this broadcast user is It is possible to generate videos that include animations of the objects. The image generating unit 100 may include, for example, an acquisition unit (not shown) and a processing unit (not shown). Cut.

[0068] The acquisition unit acquires data on the face, hands, feet, etc. (whole body) of the broadcast user. The first sensor (not shown) and the utterance and / or singing uttered by the broadcast user and one or more second sensors (not shown) for acquiring audio data corresponding to the can.

[0069] In a preferred embodiment, the first sensor is an RGB camera that captures visible light and a near-infrared and a near-infrared camera for capturing an image of the line. , which will be described later, may include a motion sensor, a tracking sensor, etc. As a camera or near-infrared camera, for example, the iPhone X (registered trademark) TrueDepth It is possible to use the one included in the True Depth camera. The sensor may include a microphone for recording audio.

[0070] First, regarding the first sensor, the acquisition unit is placed close to the face, hands, feet, etc. of the broadcast user. The first sensor (a camera included in the first sensor) is used to capture the face, hands, feet, etc. of the broadcasting user. This allows the acquisition unit to record the image acquired by the RGB camera as a time code ( The data recorded over a unit of time (e.g. MP EG file). Furthermore, the acquisition unit can generate the near-infrared image data acquired by the near-infrared camera. The number (e.g., floating-point number) indicating the depth of the specified number (e.g., 51) is added to the above type. Data recorded over a unit of time (e.g., TSV file [data It is possible to generate a file in a format that records multiple data by separating each data with a tab. do.

[0071] Regarding near-infrared cameras, specifically, a dot projector projects a dot pattern. The system emits an infrared laser onto the face, hands, feet, etc. of the broadcasting user, who is the performer, and a near-infrared camera captures the The infrared dot projected on the user's face and reflected is captured, and the The acquisition unit generates an image by acquiring dots emitted by a dot projector that has been registered in advance. By comparing the image of the grid pattern with the image captured by the near-infrared camera, The position of each point (feature point) in the image (for example, each of 51 points / feature points) The depth of each point (each feature point) is calculated using the deviation (each point, each feature point and the near-infrared camera). The acquisition unit calculates the depth (the distance between the object and the target object). As above, the data recorded over a unit time is generated by associating it with the time code. It is possible.

[0072] Next, regarding the second sensor, the acquisition unit acquires the second sensor located in the vicinity of the distribution user. Using the above, speech and / or singing voice uttered by the broadcast user is acquired. As a result, the acquisition unit can acquire data (e.g. In one embodiment, the acquisition unit may generate a first section. The second sensor is used to obtain data on the face, hands, and feet of the broadcasting user. Acquire voice data regarding speech and / or singing uttered by a broadcast user using the In this case, the acquisition unit may acquire an image acquired by the RGB camera and a second Using the sensor, voice data regarding speech and / or singing uttered by the distribution user and The data recorded over a unit time period in correspondence with the same time code (e.g. MPE G file).

[0073] The acquisition unit then extracts the data on the face, hands, feet, etc. of the broadcast user (MPEG file) generated in this manner. files and TSV files, and / or utterances and / or can output voice data (such as an MPEG file) relating to singing to the processing unit.

[0074] In this embodiment, the first sensor includes an RGB camera and an infrared camera. As explained above, the first sensor is, for example, (A) an infrared laser irradiated to detect the distribution user's (B) A camera that detects infrared rays reflected from the face, limbs, etc.; (C) Multiple RGB cameras that capture visible light (C) a single camera that captures visible light; In the case of (A) above, the acquisition unit may acquire the distribution information by the same method as described above. The depth of each feature point on the user's face, hands, feet, etc. can be calculated. In the case of (1), the acquisition unit performs alignment using a plurality of images captured by a plurality of RGB cameras. It is possible to calculate the depth for each feature point on the user's face, hands, feet, etc. In the above case of (C), the acquisition unit acquires the distribution user's Calculate the depth of each feature point on the face, hands, feet, etc. using deep learning etc. In the above case (C), the acquisition unit may be configured to capture images using a single camera. By analyzing the captured images (image processing, etc.), each characteristic of the broadcaster's face, limbs, etc. is identified. The depth of the feature may be calculated.

[0075] The acquisition unit also uses a motion sensor and a tracking sensor included in the first sensor. and acquiring detection information regarding the position and orientation of each part of the broadcasting user's body, and The processing unit can output the information using a motion sensor or tracking sensor. Details of the detection information regarding the position and orientation of each part of the broadcasting user's body that is acquired will be described later. do.

[0076] The processing unit receives data on the face, limbs, etc. of the distribution user from the acquisition unit (as well as detection information, etc.) Based on the various information related to the It is possible to generate videos that include animations. The processing unit processes various information (e.g., Geometry information, bone information, texture information, shader information and blendshape information By using the above, the rendering unit (not shown) executes rendering, and the avatar is It is also possible to generate animations of target objects.

[0077] In addition, data regarding the actions of the broadcast user and / or utterances made by the broadcast user Based on the voice data of the user's voice and singing, the animation of the avatar object of this user is Any other known techniques may be used to generate video including animation. do.

[0078] By the way, the communication unit 160 described later responds to gifting from a viewing user by sending an avatar The first gift object is displayed on a video containing an animation of the target object. When the request information to that effect is received from the server device 20 (information processing server 20B), the video generating unit The communication unit 100 receives the request information via the communication unit 160 and Identification information (name, shape, etc.) regarding the first gift object stored in the gift storage unit 150 First, place the first gear on the video containing the animation of the avatar object by referring to the shape, color, etc. Display the gift object (generate a video displaying the first gift object) ).

[0079] Specifically, the first gift object is generated in response to a gifting action from a viewing user. In order to display the gift on the video, each of the first gift objects is assigned a corresponding A first collision area is preset, and each first gift object and its corresponding The information on the first collision area is stored in the gift storage unit 150, which will be described later, for each first gift. The image is stored along with information about the soft object (size, shape, color, etc.). , the first gift object is displayed at a position on the video. Each gift object has a position coordinate in a three-dimensional Cartesian coordinate system that is set at the time of gifting. When setting the position coordinates, the avatar object already displayed on the video is A position that does not overlap with various objects such as the first gift object and the first gift object on the video. The coordinates are set. The coordinates of positions that do not overlap on the video are the coordinates of multiple positions displayed on the video. A plurality of first collision regions set for each of the first gift objects overlap each other. There is no avatar object (each part of the avatar object) in the video. The second collision area or other object (e.g. the second gift object) is set to This means that the first collision area does not overlap with the third collision area set in the In order to satisfy this condition, the video generator 100 generates each of the first gift objects. The position coordinates in three-dimensional orthogonal coordinates (X axis is horizontal, Y axis is depth, and Z axis is vertical) The position coordinates are set to correspond to the center position of the first gift object. The position coordinates of the first gift object in FIG. 5 are the X-axis, Y-axis, and Z-axis. (position coordinates corresponding to the center position Z0 of the object 410A) and / or outside the first collision area Position coordinates on the X-axis, Y-axis, and Z-axis corresponding to a plurality of positions forming an edge (e.g., 5, the outer edge of the first collision area corresponding to the first gift object 410A is formed. Positions Z1 to Z 12 The position coordinates set in this way correspond to the position Based on the target (and the first collision area), each first gift object is The animation of the object is displayed on the video. Based on the coordinates, it is possible to determine whether multiple first collision regions overlap. The position coordinates and the first collision area set for each of the first gift objects are , and the second collision area are set in a three-dimensional Cartesian coordinate system. When expressed in terms of elements, they may be set in a two-dimensional Cartesian coordinate system.

[0080] In addition, the determination unit 120, which will be described later, determines the first image displayed on the video based on the broadcast user's action. It is judged that the avatar object has performed a given action on the gift object of 1. When the determination is made, the video generating unit 100 receives a signal related to the determination from the determining unit 120, The first gift object is placed on a video containing an animation of the avatar object. Display a different second gift object (display a second gift object) Generate a video).

[0081] Specifically, the first gift object displayed on the video based on the broadcast user's action The determination unit 120 determines that the avatar object has performed a predetermined action on the target. In order to display the second gift object on the video, Each object has a third collision area that corresponds to its shape. Information about the 2nd gift object and its corresponding 3rd collision area will be explained later. In the gift storage unit 150, information about the second gift object (size, shape, The second gift object is stored along with the color of the object. Each of the second gift objects has a three-dimensional Cartesian coordinate system. The position coordinates of the target system are set. When setting the position coordinates, the Various objects such as the avatar object and the first gift object are displayed on the video. The position coordinates that do not overlap on the video are set. Similarly, a plurality of first gift objects displayed on the video are set to each of the plurality of first gift objects. The first collision area and the second collision area set for the avatar object are 3 means that the collision regions do not overlap. To satisfy this condition, the video generator 10 0 is a three-dimensional Cartesian coordinate system (X axis is horizontal, Y axis is depth) for each of the second gift objects. The position coordinates are set with respect to the first axis (vertical direction, Z axis vertical direction). The X axis corresponds to the center position of the second gift object, and the Y axis corresponds to the center position of the second gift object. and / or a plurality of coordinates of the position of the third collision area on the Z-axis and the Z-axis. This means the position coordinates on the X-axis, Y-axis, and Z-axis corresponding to the position. Based on the position coordinates (and the third collision area) of the second gift object, The second gift object is displayed on the video containing the animation of the target object. The position coordinates and the third collision area for each object are set in three-dimensional Cartesian coordinates. However, when a video is displayed in two dimensions, it is displayed in a two-dimensional Cartesian coordinate system. May be set.

[0082] As described above, the determination unit 120 determines that a predetermined action has been executed. When the second gift object is displayed on the video as a trigger, a predetermined action is performed. Various methods can be adopted for the first gift object that is executed. For example, A first gift object is displayed on the screen when a predetermined action is performed. The gift object is erased from the video. The first gift object on which the action was performed is not erased from the video, and the first gift object is The morphology (e.g., shape, size, color, etc.) of the object gradually changes over time ( For example, if only the size of the first gift object changes, When the color of a part of the object changes, etc., a second gift object is generated (the first The gift object of 2 is displayed on the video. A configuration may be adopted (e.g., the first gift on which a predetermined action is performed If there are two objects, the first gift object of one of the two is The first gift object disappears from the video, and the shape of the other first gift object gradually changes to become the second gift object. (It may be configured so that a soft object is generated.)

[0083] In addition, the video generating unit 100 generates the first gift option when a predetermined condition is satisfied. The object is recognized as being grasped by the avatar object, as shown in Figure 4. A video of the first gift object being held by an avatar object. Here, the predetermined condition is a condition that the first gear is generated based on the operation of the broadcast user. The first collision area set for the soft object and the hand (or The second collision area set for the finger overlaps at least partially in the video. If the hand (or fingers) of the avatar object is bent, This determination may be made by the video generator 100. Alternatively, each decision result may be determined by at least one of the video generating units 100. In addition, when the hand (or fingers) of the avatar object is bent, The method for determining whether the sensor is in a high-speed state or not can be performed based on the controller signal, which will be described later. In addition, the second gift object can be grasped by the avatar object. The same is true for the second gift object. The area and the second collision area overlap at least partially in the video. If the avatar object's hand (or fingers) are bent, If it is determined that the second gift object is grasped by the avatar object, will be done.

[0084] In addition, whether or not a plurality of first collision regions overlap, and the first collision region and the second collision region are The determination of whether or not a collision area overlaps with a region is made by using the X This is performed based on multiple position coordinates for the Y-axis and the Z-axis.

[0085] As described above, when the determination unit 120 determines that a predetermined action has been executed, When displaying a second gift object on a video, what kind of second gift object should be displayed on the video? The animation generating unit 100 determines whether to display the gift object by referring to the gift storage unit described later. The setting conditions are determined by referring to the setting conditions stored in 150. The setting conditions are, for example, first, The type of the first gift object on which the predetermined action was performed, The number of first gift objects given for each type and the specific position (see FIG. 4) to be described later Depending on the various combinations of specific positions Y1 or Y2), any second gear can be formed. The setting conditions are the conditions that relate whether the object corresponds to the object. When at least one gift object touches another first gift object, In the case of the first gift object for which a predetermined action has been executed, the type and number of gift objects (number of gifts) Depending on the various combinations of This is information that associates

[0086] As a specific example of the first case of the above-mentioned setting conditions, the case shown in FIG. Then, the type of the first gift object for which the specified action was performed is "Boy's stuffed toy". "(reference number 410A in FIG. 4), the first gift on which a predetermined action is performed The number of objects given per type ("Boy's stuffed animal" 410A) is "1", and the specific position When the location is "Y1", the second gift object is multiple " "(Little Teddy Bear)" (reference number 420 in FIG. 6). In the case of FIG. 4, the type of the first gift object on which the predetermined action was performed "Boy's stuffed toy" (reference number 410B in FIG. 4), a predetermined action is executed. The number of first gift objects given for each type ("Boy's stuffed toy" 410B) If "One" and the specific location is "Y2", the second gift object is one "Big The settings for the "big elephant stuffed animal" (not shown) are preset. The various conditions can be changed as appropriate and stored in the gift storage unit 150 in the form of a table, for example. will be done.

[0087] On the other hand, as a specific example of the second case of the above-mentioned setting conditions, referring to the case shown in FIG. For example, the type and number of the first gift object on which a specific action has been performed is "male "Child's stuffed animal" 410A is "1", and "Boy's stuffed animal" 410B is "1 In the case of ", the second gift object is one "little man's "(reference number 430 in FIG. 7)" is set in advance. The various conditions related to these setting conditions can be changed as appropriate. For example, the first gift option The contact time between the first and second gift objects and the collision speed when the first gift objects come into contact with each other are set as conditions. (Specifically, when the collision speed is αkm / h or more, the second gift object When the speed is less than αkm / h, the second gear is displayed. The "turtle plush toy" is displayed as a soft object) and the specified action is executed. The relative positions of the first gift objects when they are touched (for example, in the case shown in FIG. 5) At the time of contact, "a boy's stuffed animal" 410A is "a boy's stuffed animal" The type of the second gift object to be displayed in the case above and below 410B These setting conditions may be, for example, the shape of the table. The gift is stored in the gift storage unit 150 in the form:

[0088] In addition, the first and second setting conditions include the second gift object displayed on the video. 6 and 7. The question of where the "Little Boy's Stuffed Animal" 430 will appear in the video Specifically, in the case of FIG. 6, the second gift object Each of the multiple "small teddy bears" 420 as objects has multiple locations on the video. The information about the individual position coordinates for the X-axis, Y-axis, and Z-axis is distributed separately across the The information is preset (i.e., the position where the second gift object is displayed is determined based on the predetermined action). The first gift object for which the action was performed is displayed in a different location from the one in which it was located. On the other hand, in the case of Figure 7, the second gift object is a "small The first gift object (e.g., "Boy's stuffed animal") 430 is Information regarding the same location coordinates as the location coordinates where the "Personal Body" 410A) was located is set in advance. can be.

[0089] Furthermore, the video generating unit 100 determines that the predetermined action has been executed by the determining unit 120. When the second gift object is displayed on the video as a trigger, as shown in Figure 8, Then, the second gift object is displayed on the video (second gift object) Display the effect object 500 on the video, which shows the state just before the object is displayed. In this case, the video generating unit 100 may cause the predetermined action ( The first gift object, “Boy’s stuffed toy” 410A, is transferred to another first gift object, The robot touches the soft object "Boy's stuffed toy" 410B. When the determining unit 120 determines that the effect object 500 is displayed on the video, The effect is started when the second gift object is actually displayed on the video. The object 500 is erased from the video. Regions (e.g., first collision regions 411A and 411B in FIG. 5) overlap Refer to the overlapping position coordinates and add effect object 50 to the overlapping position coordinates. It is possible to display 0.

[0090] (2) Display section 110 The display unit 110 displays a video (a video displaying a first gift object and a video displaying a second gift object). Various information required for the distribution and / or viewing of videos (including videos showing objects) For example, the display unit 110 can display a distributed video, a received video, and The display unit 110 can display the video and / or the generated video. Various comments distributed from the terminal device 20 to each terminal device 10 are also displayed together with the received video. Furthermore, the gift information (first gift, It is also possible to display information about the content (information about the content object) appropriately according to the operation of the viewing user. do.

[0091] (3) Judgment unit 120 The determination unit 120 determines whether the animation of the avatar object is generated by the video generation unit 100. A state in which one or more first gift objects are displayed on a video including In the state where the video displaying the first gift object has been generated, Based on the action, an avatar object is assigned to the first gift object displayed on the video. It is determined whether the object has executed a predetermined action. For example, the avatar object places the first gift object at a specific position in the video. Actions that move or when an avatar object is moving at least one primary gift object The action of touching, sliding, or colliding the first gift object with another first gift object. Note that all of the above actions are based on the actions of the broadcasting user. do.

[0092] As mentioned above, each first gift object has a unique (first gift object The first collision area (corresponding to the shape of the object) is preset, and the avatar object's hand is (or a finger) is preset with a second collision area, and when the above-mentioned predetermined conditions are met, The first gift object is held by the avatar object. The object must hold two primary gift objects in each hand. For example, an avatar object can have both hands. In the video, three or more first gift objects may be held. In cases where multiple avatar objects are displayed, four or more primary gift objects The object may be grasped by the avatar object.

[0093] As shown in FIG. 4, the determination unit 120 determines whether the avatar object 400 is a first gift object. Object 410A (or 410B) is placed at first position X1 (or first position X1'). Starting from a state where the user holds the object, the user maintains the state based on the user's actions. While keeping the gift object 410A (or 410B), move the first gift object 410A (or 410B) to the first position X1 (or The first position Y1 (or the second position Y2) is different from the first position X1'. In this case, it is determined that an action to move to a specific position in the video has been executed. In this case, the second position Y1 (or the second position Y2) is the specific position. Or the first position X1') is appropriately changed to any position on the moving image. , the second position Y1 (or the second position Y2) is X It is sufficient to set it in advance in the YZ coordinate system.

[0094] Also, as shown in FIG. 4, the avatar object 400 has two first gift objects. As shown in FIG. 5, the delivery is performed by holding the objects 410A and 410B in both hands. The two first gift objects 410A and 410B are brought into contact with each other based on the user's actions. When the first action is executed, the determination unit 120 determines that the predetermined action has been executed. Gift objects 410A and 410B come into contact when the first gift objects 410A and 410B come into contact with each other. First collision areas 411A and 411B are set in the respective projects 410A and 410B. 1B overlap (overlap) at least in part. When two gift objects 410A and 410B come into contact with each other, the two objects slide or collide with each other. Aspects are also included.

[0095] By the way, the two first gift objects 410A and 410B are set to The first collision regions 411A and 411B overlap at least partially (overlap As explained above, the first collision area 411A and the first collision area 411B are The position is determined based on the corresponding X-axis, Y-axis, and Z-axis position coordinates.

[0096] (4) Storage section 130 The storage unit 130 stores videos (a video displaying a first gift object and a video displaying a second gift object) Various information required for the distribution and / or viewing of videos (including videos showing objects) can be stored.

[0097] (5) User interface unit 140 The user interface unit 140 displays a moving image (a moving image showing a first gift object) and the second gift object) are necessary for distribution and / or viewing. The various pieces of information to be displayed can be input through operations by the distributor user or the viewing user. For example, the user interface unit 140 may be configured to perform a gifting function for an avatar object. The operation of the viewing user with respect to the first gift object (selection from among a plurality of first gift objects prepared in advance) (including operations such as selecting one of them and selecting the command to assign to the avatar object) Operation data showing the content of the broadcast and the operation data from the viewing user to the broadcast user (avatar object) Comment information (for example, a viewer's comment on a distribution user's "Two men holding in both hands" A comment that gives a hint for a specific action: "Try hitting Noko's stuffed animal!" The viewing user inputs data indicating the contents of the information, and the various data are transmitted to the communication unit 160. The above-mentioned operation data can be, for example, the data that a viewer sends to a distribution user. What kind of first gift object (type of first gift object) should be given to the user? It may include information about whether gifting occurred.

[0098] (6) Gift memory unit 150 The gift storage unit 150 stores a video distribution application and a video viewing application. When the applications are installed on the terminal device 10, the The gift information can store the first gift object and the second gift object. The identification information of the first gift object, the name, shape, color, etc. of the second gift object, and the identification information of the second gift object, The first gift amount to be charged to the viewing user when the viewing user gifts the viewing object. This may include information about the amount per object.

[0099] In addition, the above-mentioned identification information includes each of the first gift object and the second gift object. The information also includes information about the first and third collision regions corresponding to the target. .

[0100] In addition, the gift storage unit 150 stores a When the second gift object is displayed on the video in response to the trigger, the video generating unit 100 The above-mentioned setting conditions to which the reference is made are stored.

[0101] (7) Communications unit 160 The communication unit 160 transmits a video (a video displaying a first gift object and a video displaying a second gift object) to the Various information required for the distribution and / or viewing of videos (including videos showing objects) For example, the communication unit 160 can communicate with the server device 20. When the terminal device 10 operates as a broadcast user's terminal device 10, the avatar object and request information for displaying a first gift object on a video including an animation of The video is received from the server device 20 (information processing server device 20B) and generated. The received signal can be transmitted to distribution server device 20 (distribution server device 20C).

[0102] In addition, the communication unit 160 receives a command from the determination unit 120 that the predetermined action has been executed. When the video generating unit 100 displays a second gift object on the video, the second gift object The video displaying the gift object is sent to the server device 20 (distribution server device 20C). In this way, the communication unit 160 can transmit various The video is received from the video generating unit 100, and the server device 20 (a distribution server) The received signal can be transmitted to the server device 20C.

[0103] On the other hand, when the terminal device 10 operates as the terminal device 10 of the viewing user, the communication unit 160 In this case, a video generated by a terminal device 10 of a broadcast user is received from a server device 20. Similarly, when the terminal device 10 operates as the terminal device 10 of the viewing user, In this case, the viewing user will be asked to give a gift to the distribution user (avatar object). The information provided by viewers and comments on the broadcast user (avatar object) The information on the comment can be transmitted to the server device 20 (comment API server device 20D). do.

[0104] (8) Studio Unit The above-mentioned video generating unit 100, display unit 110, determination unit 120, storage unit 130, and user interface A studio unit including a face unit 140, a gift storage unit 150, and a communication unit 160 (see FIG. (not shown) can also be used as the terminal device 10.

[0105] The studio unit can be implemented using an information processing device such as a personal computer. As with the terminal device 10 and the server device 20 described above, the main components are a central processing unit and A main memory device, an input / output interface, an input device, an auxiliary memory device, and an output device. These devices are connected to each other by a data bus and / or a control bus. The Studio Unit also runs certain installed applications. By functioning as an information processing device, Data regarding the actions of broadcast users in a broadcast system (not shown) and / or data issued by broadcast users The animation of the avatar object is based on the speech and singing data provided. It is possible to generate a video including the above.

[0106] Many of the components of the video generator 100 included in the studio unit are located in the studio room. In addition, in the studio room, broadcast users can perform various performances. The studio unit detects the movements, facial expressions, and speech (including singing) of the broadcast user. is doing.

[0107] The distribution user is a person who is in charge of the production of the video by various sensors included in the studio unit (video production unit 100). The objects that are captured are movements, facial expressions, and speech (including singing). In this case, the broadcast user does not necessarily have to be a human being, but may be, for example, a person capable of autonomous movement. The broadcast user in the studio room may be a single person or a robot. , may be 2 or more.

[0108] The video generating unit 100 in the studio unit generates the video of multiple parts of the broadcast user (e.g., hands, Various motion sensors (not shown) are attached to the neck, instep, waist, top of the head, etc. The distribution user is provided with a controller (not shown) held in the user's hand and a wearable device (not shown). and a camera (not shown) attached to the head of the camera. The unit is equipped with the aforementioned equipment and studio room to obtain the voice data of the broadcasting user. A microphone is provided somewhere in the system (e.g., on a wall, floor, ceiling, etc.). In addition to the above configuration, the studio unit also includes multiple base stations (not shown), It may have a racking sensor (not shown) and a display (not shown).

[0109] The motion sensor cooperates with the base station to determine the location of the broadcast user. In one embodiment, the base stations can detect the position and orientation of the It is a multi-axis laser emitter that emits a synchronizing flashing light and then synchronizes with one base station. The base stations may, for example, scan the laser beam around a vertical axis, while the other base stations may, for example, scan the laser beam around a horizontal axis. The motion sensor is configured to scan the laser light around the axis. The device is equipped with multiple optical sensors that detect the incidence of blinking light and laser light from the the time difference between the timing and the incidence timing of the laser light, the light reception time at each optical sensor, The motion sensor can detect the angle of incidence of the laser light detected by the sensor. For example, Vive Tracker provided by HTC Corporation r or Xsen provided by ZERO C SEVEN Inc. s MVN Analyze may also be used.

[0110] The video generator 100 in the studio unit calculates the motion of the The motion sensors acquire detection information indicating the position and orientation of each motion sensor. The device is attached to the user's wrist, instep, waist, top of the head, etc., and functions as a motion sensor. By detecting the position and orientation, the movement of each part of the body of the broadcasting user can be detected. The detection information indicating the position and orientation of the motion sensor is stored in the video (including the virtual The position coordinates of each part of the user's body in the XYZ coordinate system (in the virtual space) The X axis is the horizontal direction in the video, the Y axis is the depth direction in the video, and the Z axis is corresponds to the vertical direction in the video. Therefore, the movement of each part of the body of the broadcasting user are all detected as position coordinate values ​​in the XYZ coordinate system.

[0111] In one embodiment, multiple motion sensors are equipped with multiple infrared LEDs. The light from the external LEDs is detected by infrared cameras installed on the floor and walls of the studio room. The position and orientation of the motion sensor may be detected by using a light source instead of an infrared LED. A visible light LED is used, and the light from this visible light LED is detected by a visible light camera. The position and orientation of the motion sensor may be detected.

[0112] In one embodiment, the motion sensor is replaced by a number of reflective markers. The reflective marker is attached to the broadcast user with adhesive tape or the like. A broadcast user with a shooting marker attached is photographed to generate shooting data, and the shooting data is used to display a By processing the image, the position and orientation of the reflective marker (as described above, in the XYZ coordinate system) can be determined. The present invention may be configured to detect the position coordinate value at the target position.

[0113] The controller sends control signals in response to the user's finger movements, such as bending the user's fingers. This is output and acquired by the video generator 100.

[0114] The tracking sensor records the setting information of the virtual camera to construct the virtual space included in the video. The tracking information is used to determine the three-dimensional orthogonal coordinate system. The motion picture generator 100 calculates the position in the coordinate system and the angle around each axis. Get information.

[0115] The camera attached to the broadcast user's head is positioned so as to capture the broadcast user's face. The camera continuously captures the face of the broadcast user, thereby capturing an image of the broadcast user's face. The video generator 100 acquires the captured image data from the camera. The camera may be a 3D camera capable of detecting the depth of the broadcast user's face.

[0116] The display functions as the display unit 110 and is viewable by the broadcast user. The distribution user obtains various information and uses the video generated by the video generating unit 100. It is possible to view the recorded video through the display.

[0117] The video generating unit 100 includes the above-mentioned motion sensor, tracking sensor, various cameras, and It obtains information from the controller and other devices, including the animation of the avatar object. Videos can be generated.

[0118] In addition to the components and functions described above, the studio unit also has the The moving image generating unit 100, the display unit 110, the determination unit 120, the storage unit 130, and the user interface The interface unit 140, the gift storage unit 150, and the communication unit 160 (or a unit having these functions) This allows the decision regarding the predetermined action to be The first gift object is placed on a video containing the settings and animation of the avatar object. and / or displaying a second gift object. The components of the studio unit described above are merely examples, and various other applicable components may be used. It may include components.

[0119] 3-2. Functions of the server device 20 An example of the function of the server device 20 will be described with reference to FIG. FIG. 2 is a block diagram illustrating an example of the functions of the server device 20.

[0120] As shown in FIG. 9, the server device 20 mainly includes a storage unit 200, a communication unit 210, a gift processing unit, and a A processing unit 220 and a comment processing unit 230 may be included.

[0121] (1) Storage section 200 The storage unit 200 stores a video (a video displaying a first gift object and a video displaying a second gift object) Various information required for the distribution and / or viewing of videos (including videos showing objects) can be stored.

[0122] (2) Communications unit 210 The communication unit 210 transmits a video (a video displaying a first gift object and a video displaying a second gift object) to the Various information required for the distribution and / or viewing of videos (including videos showing objects) between the terminal device 10 of the broadcasting user and / or the terminal device 10 of the viewing user. For example, the communication unit 210 may display a video (a first gift object) (including a video in which the second gift object is displayed and a video in which the second gift object is displayed) on the distribution user's terminal The content can be received from the device 10 and distributed to the terminal device 10 of each viewing user.

[0123] In addition, the communication unit 210 receives a message from the viewing user to the distributor user (avatar object). Information about gifting, and from viewing users to broadcasting users (avatar objects) When the information regarding the comment on the gift is received from the terminal device 10 of the viewing user, The information about is sent to the gift processing unit 220, and the information about the comment is sent to the comment processing unit 230. and output respectively.

[0124] Furthermore, the communication unit 210 transmits the first gift object read by the gift processing unit 220. The information about the gift is obtained from the gift processing unit 220, and the animation of the avatar object is generated. A user transmits a request for displaying the first gift object on a video including the first gift object. The communication unit 210 can transmit the gift to the terminal device 10 of the user. The information on the first gift object obtained from the terminal device 10 of the broadcast user is The video generating unit 100 generates a plurality of first gift objects stored in the gift storage unit 150. Which first gift object information should be referenced from the information about the gift object? For example, the first gate is transmitted from the terminal device 10 of the viewing user. When a "Boy's Stuffed Animal" is gifted as a gift object, the communication department 21 0 is a first gift object, which is the "boy's stuffed toy", from the gift processing unit 220. It may be just the name of the object (or a character string similar to a name).

[0125] Furthermore, the communication unit 210 receives a first gift object from the terminal device 10 of the broadcast user. It is determined that the avatar object has performed a predetermined action on the object, and the second gift is Receive information that an object has been displayed on a video and process the information as a comment The signal may also be output to unit 230.

[0126] (3) Gift Processing Unit 220 When the gift processing unit 220 receives the information about the gifting from the communication unit 210, Identification information of the viewing user who committed the footing act (serial number of the terminal device 10 of each viewing user) The first gift of the gift (such as the ID number and ID information assigned to each viewing user) Information about the object (identification information such as the name, size, shape, color, etc. of the first gift object) information), and the identification information of the distribution user who is the avatar object to be gifted (distribution user The serial number of the terminal device 10 of the broadcast user, ID information given to the broadcast user, etc. are read out.

[0127] In addition, regarding the first gift object, the gift processing unit 220 The animation generating unit 100 in the device 10 generates a number of first gifts stored in the gift storage unit 150. From among the information about the objects, which first gift object information should be referenced? Information sufficient to identify the intended recipient (e.g., the name of the first gift object, as mentioned above) Only the name or a character string similar to a name needs to be read out, and the size of the first gift object The identification information such as size, shape, color, etc. does not necessarily have to be read. The information on the first collision area corresponding to the object is read by the gift processing unit 220. The gift is not displayed, but is stored in the gift storage unit 150 described above.

[0128] The various information read by the gift processing unit 220 is output to the storage unit 200. The result is stored in the storage unit 200.

[0129] (4) Comment processing unit 230 The comment processor 230 processes comments from viewing users to a distributor user (avatar object). Comment information (for example, a viewer's comment on a distribution user's "two things in both hands" A comment that gives a hint for a specific action: "Try hitting the boy's stuffed animal!" The content information (including the content information) is received from the terminal device 10 of the viewing user via the communication unit 210. The comment information from the viewing user can be shared among all viewing users who are viewing the same video. can be distributed to.

[0130] In addition, the comment processing unit 230 processes various information ( With reference to the information stored in the storage unit 200, which viewing user has given which first gift object All or part of the information regarding whether the user gifted the object to the avatar object Comment information corresponding to the information (for example, "Viewer X gifted a "boy's stuffed toy" "I watched the video!") and watch the same video with that comment information It can also be distributed to each viewing user who is viewing the content.

[0131] Furthermore, the comment processing unit 230 receives a second gift object from the communication unit 210. When the information is received that a comment was displayed on the video, the corresponding comment information (e.g. For example, create a comment like "Lots of "small teddy bears" have appeared!" The comment information can also be distributed to each viewing user who is viewing the same video. Cut.

[0132] Furthermore, the comment processor 230 receives comments from viewing users to distribution users (avatar objects) The comment information on the comment (object) is received from the terminal device 10 of the viewing user via the communication unit 210. Regardless of whether the gift has been received or not, the gift processing unit 220 reads out various information and A hint for the given action for the first gift object that is gifted. Comment information that makes the difference (for example, "Two "boy stuffed animals" are stuck together vertically. "Let's line them up!" comment information) and then use that comment information in the same video. It can also be distributed to each viewing user who is viewing the content.

[0133] 4. Overall operation of video distribution system 1 Next, the overall operation of the video distribution system 1 having the above configuration will be described as follows: The following description will be given with reference to FIG. 10. FIG. 10 shows the flow of the process performed in the video distribution system 1 shown in FIG. FIG. 13 is a flow diagram showing an example of the operation.

[0134] First, in step (hereinafter referred to as "ST") 300, according to the operation of the broadcast user, Then, the terminal device 10 (here, terminal device 10A) starts a video distribution application. and execute.

[0135] Next, in ST301, terminal device 10A receives data on the operation of the broadcast user and / or the distribution user based on voice data regarding speech or singing uttered by the distribution user This generates a video containing the animation of the user's avatar object. The display unit 110 of the device 10A displays an animation of the avatar object 400 of the broadcast user. For example, in FIG. 4, the first gift objects 410A and 410B are The video before it is displayed will be displayed.

[0136] Furthermore, the terminal device 10A transmits the generated video to the distribution server device 20C of the server device 20. The distribution server device 20C transmits the video received from the terminal device 10A to the video viewing application. The terminal device 10 of each viewing user (here, the terminal device 10B , terminal device 10C, and terminal device 10D). The same video is also displayed on the display unit 110 of the device 10.

[0137] Next, in ST302, the terminal device 10 of the viewing user controls the user interface unit 1 Based on the viewer's operation on 40, The server device 20 (gift The request is sent to the API server device 20A.

[0138] Next, in ST303, the first gift object is received from the terminal device 10 of the viewing user. The server device 20 (gift API server device 20A) that receives the request signal for the gift Identification information of the viewing user to whom the gifting action was performed, the first gift object that was gifted, Information about the gifting project, and the identity of the distribution user who is the avatar object to be gifted Read out other information.

[0139] Then, in ST304, the server device 20 (information processing server device 20B) reads Based on the information provided, the relevant part will be added to the video containing the animation of the avatar object. 1. A request for displaying the gift object is transmitted to the terminal device 10 of the broadcast user. At this time, the information processing server device 20B receives the first At least a portion of the information about the gift object (e.g., the first gift object, as described above) Obtain the name of the avatar object or a string similar to the name, and Request information for displaying the first gift object on a video including an animation Generate.

[0140] Next, in ST305, the terminal device 10A of the broadcast user performs the server installation in ST304. Based on the request information sent from the device 20, the animation of the avatar object is included. The first gift object corresponding to the request information is displayed on the video being distributed. The terminal device 10A of the user displays a first gift object corresponding to the request information on a video. When displaying the gift in the gift storage unit 150, the terminal device 10A of the broadcast user stores the gift in the gift storage unit 150. The first gift object to be created is identified by its name, shape, color, first collision area, etc. See:

[0141] Next, in ST306, the broadcast user's terminal device 10A receives one or more The avatar object performs a predetermined action on a plurality of first gift objects. When it is determined that the second gift object has been performed, the second gift object is displayed on the video. As described above, the decision regarding the action is made by the decision unit 120 of the broadcast user's terminal device 10A. Also, the processing of the first gift object on which the predetermined action has been performed is The method and the method of displaying the second gift object on the video are as described above. It is.

[0142] As described above, the step is completed when the second gift object appears on the video. The group ends.

[0143] 5. Modifications In the above embodiment, the animation (the animation showing the first gift object and The generation of the video (including the video showing the second gift object) is performed by the terminal device 10 of the distribution user. However, in another embodiment, for example, the server device 20 or the viewing user In this case, the server device 20 and / or the terminal device 10 may execute the process. Alternatively, a rendering unit (not shown) may be provided in the terminal device 10 of the viewing user, and The terminal device 10 is a server device 20 and / or a terminal device 10 of a viewing user that is receiving a video (first A video showing a gift object and a video showing a second gift object Various information (e.g., a first gift object, Various identifying information about the first gift object and various identifying information about the second gift object) The configuration may be such that

[0144] In order to realize such a configuration, a server device 20, a terminal device 10 of a viewing user, and The functions of the broadcast user's terminal device 10 may be modified as appropriate from the above-described embodiment. For example, in one embodiment, the video generator 100 is provided in the terminal device 10. However, it may be provided in the server device 20 or at the terminal device 10 of the viewing user. The terminal device 10 may also be provided with the above-mentioned.

[0145] In addition, the determination as to whether the above-mentioned predetermined action has been performed may be performed in accordance with the above-mentioned embodiment. In the above embodiment, the process is executed by the terminal device 10 of the broadcast user, but in another embodiment, the process is executed by the server. The execution may be performed by the server device 20 or the terminal device 10 of the viewing user. In this case, the server device 20 or the terminal device 10 of the viewing user is provided with a function similar to that of the determination unit 120. will be prepared.

[0146] Therefore, according to various embodiments, it is possible to increase the gifting opportunities for viewing users. It is possible to provide a video processing method, a server device, and a computer program. Then, the first gift object is transformed into a second gift object, creating a You can increase the variety of gift objects displayed in the. In one embodiment, a plurality of first gift objects are converted into one second gift object. In the case of converting the image into a single image, the image is displayed on the display unit 110 of the terminal device 10 of the viewing user in a limited area. The number of first gift objects displayed in the area can be smoothly reduced. (In this case, the size of the second gift object displayed on the video is the same as that of the first gift object.) (Preferably smaller than the gift object), and the gifting motivation of viewers Furthermore, the first gift object can be delivered. The second gift object is created through a predetermined action of the user (avatar object). By changing the embodiment, some action can be taken by the broadcast user. Furthermore, according to various embodiments, the viewing user can ,For the broadcast user, advice related to a given action is ,provided as comment information (e.g., A viewer's response to a broadcast user was "crash the two boy stuffed toys he is holding in both hands." "Try it out!") can be sent as comment information, so the distribution user and the viewing user can This will stimulate communication between users and the company.

[0147] 6. Various Aspects The video processing method according to the first aspect is a method for executing computer-readable instructions. A video processing method executed by one or more processors, A video including an animation of an avatar object of the broadcast user is generated based on the A process of generating the avatar of the video based on the operation of a user who watches the video. a request signal requesting that a first gift object be granted to the object; A receiving step of receiving the request signal from a terminal device of a viewing user via a communication line; The first gift object granted to the avatar object by the A first display step of displaying on the video, and displaying on the video based on the action of the broadcast user. The avatar object is displayed for one or more of the first gift objects. When it is determined that the user has performed a predetermined action, A second gift object different from the first gift object is displayed on the video. and a second display step of displaying the same.

[0148] The video processing method according to the second aspect is the same as the first aspect, except that "the predetermined action is: Based on the action of the broadcast user, the avatar object changes the first gift object. The action of moving a target object to a specific position is called "the action of moving a target object to a specific position."

[0149] The video processing method according to the third aspect is the same as the first aspect, except that "the predetermined action is: Based on the action of the broadcast user, the avatar object changes at least one of the the action of touching one gift object to another said first gift object; "

[0150] The video processing method according to the fourth aspect is the same as the third aspect, except that "the first gift object A first collision area is set for each of the objects, and at least two of the first collision areas are set. If it is determined that the image areas overlap at least partially in the moving image, "Both of the first gift objects are recognized as being in contact."

[0151] The moving image processing method according to the fifth aspect is the same as the moving image processing method according to the fourth aspect, except that "at least two of the The determination of whether or not the collision regions overlap at least partially in the video is made based on the Each of the first positions based on the horizontal X-axis, the depth Y-axis, and the vertical Z-axis in the video. It is executed based on the position coordinates of the collision area.

[0152] The video processing method according to the sixth aspect is the same as the second aspect, except that "the first gift object A first collision area is set for each of the objects, and the hand of the avatar object is , a second collision area is set, and the first collision area and the second collision area are When it is determined that there is at least a partial overlap in the video, and the distribution user When it is determined that the finger of the avatar object is bent based on the user's movement, the first gift object is held by the avatar object. and the first gift is displayed by the avatar object at a first position in the video. a state in which the hand object is grasped starts, and a second position in the video different from the first position is started; When the gripped state is maintained until the position of 2, an action for moving to the specific position is performed. It is recognized that an action has been taken.

[0153] The video processing method according to the seventh aspect is the same as the sixth aspect, except that "the first collision area and determining whether the second collision area overlaps at least partially within the moving image. The setting is based on the horizontal X-axis, the depth Y-axis, and the vertical Z-axis in the video. The collision area is calculated based on the position coordinates of the first collision area and the position coordinates of the second collision area. It is something that is carried out.

[0154] The video processing method according to the eighth aspect of the present invention is the same as the second aspect of the present invention, except that "the second gift object The project includes the specific location, the type of the first gift object, and the first gift object. The number of objects added per type is determined and displayed on the video. .

[0155] The video processing method according to the 9th aspect is the same as the video processing method according to the 3rd or 4th aspect. The second gift object is a gift object that is a combination of the type of the first gift object and the first gift object. The number of soft objects added to each type is determined based on the number of soft objects added to each type and displayed on the video. be.

[0156] The video processing method according to the 10th aspect is the same as the 9th aspect, except that "the second gift of The object is further determined based on the contact time between the first gift objects. This is what is displayed on the video.

[0157] The video processing method according to the 11th aspect is the same as the video processing method according to the 9th or 10th aspect. The second gift object is further arranged so that the first gift objects come into contact with each other. The speed of the collision is determined based on the actual collision speed and displayed on the video.

[0158] A moving image processing method according to a twelfth aspect of the present invention is any one of the first to eleventh aspects of the present invention. "The second display step is a step of displaying the avatar object performing the predetermined action. When it is determined that the predetermined action has been performed, the first gift object The purpose of this is to "erase the image from the video."

[0159] A moving image processing method according to a thirteenth aspect of the present invention includes any one of the first to eleventh aspects. "The second display step is a step of displaying the avatar object performing the predetermined action. When it is determined that the predetermined action has been performed, the first gift object transforming the gift into the second gift object, and It is something that is displayed on the video.

[0160] A moving image processing method according to a 14th aspect of the present invention includes any one of the first aspect to the 11th aspect. "The second display step is a step of displaying the avatar object performing the predetermined action. When it is determined that the predetermined action has been performed, the plurality of first gift objects on which the predetermined action has been performed are At least one of the first gift objects is erased from the video. and at least one other first gift object is associated with the second gift object. It transforms the information into a project.

[0161] A moving image processing method according to a fifteenth aspect is any one of the above twelfth to fourteenth aspects. In the above, "the second gift object is before being erased from the video or The first gift object is present in the video before being changed into the second gift object. "appears in a different location from where it was."

[0162] A moving image processing method according to a 16th aspect of the present invention is any one of the first aspect to the 15th aspect of the present invention. "The second display step is a step of displaying the avatar object starting the predetermined action. and an effect showing a state in which the second gift object is being displayed on the video. The method includes a step of displaying a touch object on the video.

[0163] A moving image processing method according to a seventeenth aspect of the present invention is any one of the first to sixteenth aspects of the present invention. "The generating step, the first display step, and the second display step are performed by the distribution user "The program is executed by a terminal device of the program."

[0164] The moving image processing method according to the 18th aspect is the moving image processing method according to the 17th aspect, A terminal device of the viewer user and / or a terminal device of the distributor user is connected via a communication line. The first gift object is displayed on the video. The avatar object performs a predetermined action on the broadcast user. The determination is made by the terminal device.

[0165] A moving image processing method according to a 19th aspect of the present invention is any one of the first aspect to the 16th aspect of the present invention. "The generating step, the receiving step, the first display step, and the second display step are Connected to the terminal device of the viewing user and / or the terminal device of the broadcasting user via a communication line The program is executed by a server device that is connected to the program.

[0166] The video processing method according to the 20th aspect is the same as the 19th aspect, except that "the video is displayed on the video." The avatar object performs a predetermined action on the first gift object. The broadcast user's terminal device determines whether or not the broadcast user has executed the above-mentioned command.

[0167] A moving image processing method according to a 21st aspect of the present invention is any one of the first aspect to the 16th aspect of the present invention. "The generating step, the first displaying step, and the second displaying step are performed by the viewing user "The program is executed by a terminal device of the program."

[0168] The moving image processing method according to the 22nd aspect is the moving image processing method according to the 21st aspect, A terminal device of the viewer user and / or a terminal device of the distributor user is connected via a communication line. The first gift object is displayed on the video. The avatar object performs a predetermined action on the broadcast user. The determination is made by the terminal device.

[0169] A moving image processing method according to a 23rd aspect of the present invention is any one of the first aspect to the 22nd aspect of the present invention. "The processor is a central processing unit (CPU), a microprocessor or a graph processor. The chip is a graphics processing unit (GPU).

[0170] A moving image processing method according to a 24th aspect of the present invention is any one of the first aspect to the 23rd aspect of the present invention. "The communication line includes the Internet."

[0171] A server device according to the twenty-fifth aspect includes one or more processors, However, animation of the avatar object of the broadcast user based on the broadcast user's movement A video including the above is generated based on an operation of a user who watches the video, A request signal requesting that the first gift object be given to the butter object. a request signal from the terminal device of the viewing user via a communication line, The first gift object given to the avatar object is and displaying one or more of the images displayed on the video based on the action of the broadcast user. The avatar object performs a predetermined action on the first gift object. When the terminal device of the distribution server determines that the predetermined action has been executed, A second gift object different from the first gift object executed is executed. The function is to display the information on the screen.

[0172] A server device according to the twenty-sixth aspect includes one or more processors, However, animation of the avatar object of the broadcast user based on the broadcast user's movement The information for generating a moving image including the above is transmitted to a terminal device of the distribution user that generates the moving image, or The video is transmitted to a terminal device of the viewing user, and is generated based on the operation of the viewing user who is viewing the video. and granting a first gift object to the avatar object. A request signal is received from the terminal device of the viewing user via a communication line, and the request The first gift object granted to the avatar object based on the signal. The distribution of the first gift object on the video. The broadcast user's terminal device or the viewing user's terminal device is transmitted to the broadcast user's terminal device, and the broadcast user's terminal device or the viewing user's terminal device is and for one or more of the first gift objects displayed on the video, When the avatar object executes a predetermined action, the terminal device of the distribution server If it is determined that the first gift object has been subjected to the predetermined action, information about a different second gift object, said second gift object being Transmitting the video to the terminal device of the distribution user or the terminal device of the viewing user to be displayed on the video " It is something like that.

[0173] The server device according to the 27th aspect is the server device according to the 25th aspect or the 26th aspect. The processor may be a central processing unit (CPU), a microprocessor, or a graphics processor. The graphics processing unit (GPU) is a

[0174] A server device according to a 28th aspect of the present invention is any one of the 25th to 27th aspects of the present invention. "The communication line includes the Internet."

[0175] The computer program according to the twenty-ninth aspect is "executed by one or more processors." By this, animation of the avatar object of the broadcast user based on the action of the broadcast user is A video including the animation is generated based on an operation of a user who is viewing the video. and requesting that a first gift object be given to the avatar object. A request signal for requesting the same is received from a server device or a terminal device of the viewing user via a communication line. The first gift given to the avatar object based on the request signal. a broadcast user's action based on the broadcast object displayed on the video, The avatar object is displayed on one or more of the first gift objects. When it is determined that the object has executed a predetermined action, A second gift object different from the first gift object is displayed on the video. "causing the processor to function so as to display the

[0176] The computer program according to the 30th aspect is "executed by one or more processors." By this, animation of the avatar object of the broadcast user based on the action of the broadcast user is The information for generating a video including animation is transmitted to a server device for generating the video or a viewing device. The video is generated based on the operation of the user who is viewing the video and transmitted to the user's terminal device. and requesting that a first gift object be given to the avatar object. A request signal for requesting the same is received from a server device or a terminal device of the viewing user via a communication line. The first gift given to the avatar object based on the request signal. and displaying information about the first gift object on the video. to the server device or the terminal device of the viewing user, and and for one or more of the first gift objects displayed on the video, When it is determined that the avatar object has performed a predetermined action, The second gift object is different from the first gift object on which the action was performed. the server device that displays the second gift object on the video. or to the terminal device of the viewing user. be.

[0177] A computer program according to a 31st aspect of the present invention is a computer program according to the 29th aspect or the 30th aspect of the present invention. In the above, "the processor is a central processing unit (CPU), a microprocessor or a graphic It is a "fixed processing unit (GPU)".

[0178] A computer program according to a 32nd aspect of the present invention is any one of the 29th to 31st aspects of the present invention. In any one of the above, "the communication line includes the Internet."

[0179] The computer program according to the 33rd aspect is "executed by one or more processors." By this, animation of the avatar object of the broadcast user based on the action of the broadcast user is In order to generate a video including animation, information about the video is stored in a server device or the distribution device. The video is received from a terminal device of a user who is a user of the video and is generated based on an operation of a user who is a user of the video. and requesting that a first gift object be given to the avatar object. A request signal for requesting the avatar object is transmitted via a communication line, and the avatar object is displayed based on the request signal. to display the first gift object provided to the project on the video. Then, information regarding the first gift object is sent to the server device or the distribution user. 1 or 2 displayed on the video based on the action of the broadcast user received from the terminal device. The avatar object performs a predetermined action on a plurality of the first gift objects. When the terminal device of the distribution server determines that the user has executed the predetermined action, A second gift object different from the first gift object on which the operation was performed is The server transmits information about the second gift object to the server to display on the video. and causing the processor to receive the broadcast user's broadcast information from the broadcast user's terminal device or the broadcast user's terminal device. It is something.

[0180] The computer program according to the 34th aspect is "executed by one or more processors." By this, animation of the avatar object of the broadcast user based on the action of the broadcast user is A video including the animation is received from a server device or a terminal device of the broadcast user, and the video is The avatar object is generated based on an operation of a user who views the image. A request signal is transmitted via a communication line to request that a first gift object be granted. The first grant given to the avatar object based on the request signal is The video displaying the soft object is transmitted to the server device or the terminal device of the broadcast user. and receiving from the broadcast user one or more of the broadcasting users' actions displayed on the video. The avatar object performs a predetermined action on the first gift object. When the terminal device of the distribution server determines that the user has performed the predetermined action, the terminal device of the distribution server executes the predetermined action. Displaying a second gift object different from the first gift object executed. The video is received from the server device or a terminal device of the distribution user. It "makes the processor work."

[0181] A computer program according to the 35th aspect of the present invention is a computer program according to the 33rd aspect or the 34th aspect of the present invention. In the above, "the processor is a central processing unit (CPU), a microprocessor or a graphic It is a "fixed processing unit (GPU)".

[0182] A computer program according to a 36th aspect of the present invention is any one of the 33rd to 35th aspects of the present invention. In any one of the above, "the communication line includes the Internet."

[0183] According to the 37th aspect, the terminal device of the broadcast user "displays a broadcast user's behavior based on the broadcast user's actions." Generate a video including an animation of the user's avatar object, and watch the video. A first gift is generated based on an operation of a viewing user and is given to the avatar object. A request signal for requesting the addition of an object is sent to a server device or the terminal of the viewing user. The request signal is received from the terminal device via a communication line, and the avatar object is The first gift object is displayed on the video, and the distribution Based on the user's action, one or more of the first gift objects displayed on the video are When it is determined that the avatar object has performed a predetermined action on the object, Then, a second gift object different from the first gift object on which the predetermined action is performed is The present invention relates to a method for displaying soft objects on the video.

[0184] According to the 38th aspect, a terminal device of a broadcast user is configured to "instruct the broadcast user to perform a broadcast operation based on the broadcast user's operation." The information for generating a video including an animation of the user's avatar object is The video is transmitted to a server device that generates the video or a terminal device of a user who views the video. A first gift is generated based on an operation of a viewing user and is given to the avatar object. A request signal for requesting the addition of an object is sent to a server device or the terminal of the viewing user. The request signal is received from the terminal device via a communication line, and the avatar object is The information regarding the first gift object granted to the first gift object is stored in the The server device transmits the object to the terminal device of the viewing user, which displays the object on the video. and based on the action of the distributor user, one or more of the first When the avatar object performs a predetermined action on the gift object, If it is determined that the first gift object has been subjected to the predetermined action, information about a different second gift object, said second gift object being The video is displayed on the server device or the terminal device of the viewing user. do.

[0185] The terminal device of the viewing user according to the 39th aspect is By this, animation of the avatar object of the broadcast user based on the action of the broadcast user is performed. In order to generate a video including a motion picture, information about the video is transmitted to a server device or the distribution device. The video is received from a user's terminal device and generated based on the operation of a user who is viewing the video. a request to grant a first gift object to the avatar object; A request signal is transmitted via a communication line, and the avatar object is generated based on the request signal. and displaying the first gift object given to the gift on the video. The information regarding the first gift object is transmitted to the server device or the distribution user's terminal. and receiving from the terminal device one or more of the broadcast user's actions displayed on the video. The avatar object performs a predetermined action on the number of first gift objects. When it is determined by the terminal device of the distribution server that the predetermined action has been executed, A second gift object different from the first gift object on which the The server device transmits information about the second gift object to the server device to display the second gift object on the video. The broadcast user's terminal device or the broadcast user's terminal device.

[0186] According to the 40th aspect, the terminal device of the viewing user "distributor user's operation based on the operation of the distributor user" A video including an animation of the avatar object of the user is transmitted to a server device or the distribution user. the video is received from a terminal device of a user and is generated based on an operation of a viewing user who views the video; Requesting the granting of a first gift object to the avatar object. A request signal is transmitted via a communication line, and the avatar object is The video displaying the first gift object given to the customer is transmitted to the server. and based on the operation of the broadcast user, The avatar is displayed on the video with respect to one or more of the first gift objects. The terminal device of the distribution server determines that the object has executed a predetermined action. When the predetermined action is performed on the first gift object, the second gift object is different from the first gift object. The video displaying the gift object 2 is transmitted to the server device or the distribution user. "Received from a terminal device."

[0187] 7. Fields to which the technology disclosed in this application is applicable The technology disclosed in this application can be applied, for example, in the following fields: It is something. (1) An application server that distributes live videos featuring avatar objects bis (2) An app that can communicate using text and avatar objects Application services (chat applications, messengers, email applications (cations, etc.) [Explanation of symbols]

[0188] 1. Video distribution system 10(10A~10D) Terminal device 20 Server device 20A Gift API server device 20B Information processing server device 20C Distribution server device 20D Comment API server device 100 Video Generation Unit 110 Display section 120 Judgment section 130 Storage section 140 User Interface Section 150 Gift Memory Section 160 Communications Department 400 Avatar Objects 410A, 410B First Gift Object 411A, 411B First collision area 420, 430 Second Gift Object 500 Effect Objects

Claims

1. Computer, A first means for displaying a video on a viewer terminal; a second means for displaying a first gift object on the video in response to a request from the viewer terminal; and A program that functions as a third means for displaying the second gift object in the video when a distributor performs a specified action for displaying a second gift object determined by the first gift object.

2. 2. The program of claim 1, wherein the second gift object is determined according to a type of the first gift object.

3. 2. The program of claim 1, wherein the number of said second gift objects is determined by the number of said first gift objects.

4. The program according to claim 1 , wherein the second gift object is determined by a display position of the first gift object in the video.

5. 5. The program according to claim 1, wherein the third means displays the second gift object in the video at a position not overlapping with the first gift object.

6. 6. The program according to claim 1, wherein said third means erases said first gift object from said video.

7. 6. The program according to claim 1, wherein said third means changes a form of said first gift object.

8. 8. The program according to claim 7, wherein said third means changes the form of said first gift object into said second gift object.

9. 9. The program according to claim 1, wherein the third means displays the second gift object in the video when the distributor performs the predetermined action on the first gift object.

10. The program according to any one of claims 1 to 9, wherein the third means determines whether or not the distributor has performed the specified action, and if it is determined that the distributor has performed the specified action, displays the second gift object in the video.

11. A first means for displaying a video on a viewer terminal; a second means for displaying a first gift object on the video in response to a request from the viewer terminal; and The server includes a third means for displaying the second gift object in the video when a distributor performs a predetermined action for displaying a second gift object determined by the first gift object.

12. Display the video on the viewer's device, displaying a first gift object on the video in response to a request from the viewer terminal; The video processing method includes, when a distributor performs a predetermined action for displaying a second gift object determined by the first gift object, displaying the second gift object in the video.

13. Viewer terminal, A first means for displaying a video; second means for causing the animation to display the first gift object by transmitting a request to cause the animation to display the first gift object; and A program that functions as a third means for displaying the second gift object in the video when a distributor performs a specified action for displaying a second gift object determined by the first gift object.

14. A first means for displaying a video; second means for causing the animation to display the first gift object by transmitting a request to cause the animation to display the first gift object; and A terminal device comprising a third means for displaying the second gift object in the video when a distributor performs a specified action for displaying a second gift object determined by the first gift object.

15. Display the video, causing said animation to display said first gift object by transmitting a request to cause said animation to display said first gift object; The video processing method includes, when a distributor performs a predetermined action for displaying a second gift object determined by the first gift object, displaying the second gift object in the video.

16. The distributor's terminal, A first means for displaying a video on a viewer terminal and a distributor terminal; a second means for displaying a first gift object in the video displayed on the distributor terminal in response to a request from the viewer terminal; and A program that functions as a third means for displaying the second gift object in the video displayed on the broadcaster terminal when the broadcaster performs a specified action to display a second gift object determined by the first gift object.

17. A first means for displaying a video on a viewer terminal and a distributor terminal; a second means for displaying a first gift object in the video displayed on the distributor terminal in response to a request from the viewer terminal; and A terminal device comprising a third means for displaying the second gift object in the video displayed on the broadcaster terminal when the broadcaster performs a specified action to display a second gift object determined by the first gift object.

18. Display the video on the viewer's device and the distributor's device, displaying a first gift object on the video displayed on the distributor terminal in response to a request from the viewer terminal; A video processing method, in which, when a broadcaster performs a specified action to display a second gift object determined by the first gift object, the second gift object is displayed in the video displayed on the broadcaster terminal.

Images