Computer program, server device, terminal device, and method

Abstract

To provide a computer program, server device, terminal device, and method capable of supporting interactive applications.SOLUTION: A method causes a processor to acquire a first stream of a first data type including information related to motion, sound, comments and / or gifting, acquire a second stream of a second data type including video, acquire a third stream related to the first data type including sound, synthesize the data related to the first stream, the data related to the second stream, and the data related to the third stream, and provide the synthesized data to a user. The first data type includes intermediate generation information related to video, and the intermediate generation information includes depth information related to the video.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The technology disclosed in this application relates to computer programs, server devices, terminal devices, and methods.

Background Art

[0002] In recent years, users tend to desire highly interactive applications, and there is a need for a system that can support such interactive applications.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technology described in Patent Document 1 synthesizes images of the background and a 3DCG space. In addition, the technology described in Patent Document 2 enables communication between players and participants. Moreover, the technology described in Patent Document 3 synthesizes a real image and a virtual image. In addition, the technology described in Patent Document 4 relates to video distribution.

[0005] It should be noted that the above Patent Documents 1 to 4 are incorporated herein by reference in their entirety.

[0006] ​ Some embodiments disclosed in this application provide a computer program, a server device, a terminal device, and a method that can support interactive applications. **Means for Solving the Problem**

[0007] A computer program according to one aspect, when executed by a processor, obtains a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtains a second stream of a second data type including video, combines the data related to the first stream and the data related to the second stream, and provides the combined data.

[0008] A server device according to one aspect includes a processor, and when the processor executes instructions readable by a computer, it obtains a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtains a second stream of a second data type including video, combines the data related to the first stream and the data related to the second stream, and provides the combined data.

[0009] A terminal device according to one aspect, when executed by a processor, includes a processor, and when the processor executes instructions readable by a computer, it obtains a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtains a second stream of a second data type including video, Combine the data related to the first stream and the data related to the second stream, and the combined data is provided.

[0010] A method according to one aspect is a method executed by a processor that executes instructions readable by a computer, wherein the processor, by executing the instructions, obtains a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtains a second stream of a second data type including video, combines the data related to the first stream and the data related to the second stream, and provides the combined data. data is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. Note that components shown in one drawing may be omitted in another drawing for convenience of explanation. Furthermore, note that the attached drawings do not necessarily show the present invention in exact scale. Also, in this document, the term "application" may refer to software, or what is called a program, and may be a command for a computer, as long as it is combined so that a result can be obtained.

[0013] 1. Example of the System FIG. 1 is a block diagram showing an example of the configuration of a system according to an embodiment. As shown in FIG. 1, system 1 includes one or more server devices 20 connected to communication network 10, and It may include one or more terminal devices 30 connected to the communication network 10. Note that in FIG. 1, , as an example of the server device 20, three server devices 20A to 20C are illustrated, and as an example of the terminal device 3 0, three terminal devices 30A to 30C are illustrated. However, as the server device 20 , one or more server devices 20 other than these may be connected to the communication network 10, and as the terminal device 30, one or more terminal devices 30 other than these may be connected to the communication network 10 . Note that in this application document, the term "system" is also used as a term indicating only the server, or only the terminal device, in some cases including both the server and the terminal device. That is , the system may be in any form of only the server, only the terminal device, or both the server and the terminal device .

[0014] Also, the system may be an information processing device on the cloud. Also, the system may be one that constitutes an imaginary information processing device and is logically configured as a single information processing device . Also, the owner and administrator of the system may be different .

[0015] The communication network 10 may be a mobile phone network, a wireless LAN, a fixed phone network, the Internet, an intranet , Ethernet (registered trademark), and / or a combination thereof, etc., and is not limited thereto. In the communication network 10, various data disclosed in this application document may be communicated . For example, a stream including the first data type described later, a stream including the second data type , control information, etc. may be communicated from the server device 20 to the terminal device 30 , or may be communicated from the terminal device 30 to the server device 20 .

[0016] ​The server device 20 may execute operations such as obtaining a first stream of a first data type including information related to motion, sound, comments, and / or gifting by executing a specific installed application, obtaining a second stream of a second data type including video, and synthesizing the data related to the first stream and the data related to the second stream to provide the synthesized data. 、a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtaining a second stream of a second data type including video, synthesizing the data related to the first stream and the data related to the second stream, and providing the synthesized data.

[0017] Alternatively, the terminal device 30 may execute operations such as obtaining a first stream of a first data type including information related to motion, sound, comments, and / or gifting by executing an installed web browser or a specific application, obtaining a second stream of a second data type including video, synthesizing the data related to the first stream and the data related to the second stream, and providing the synthesized data. 、obtaining a first stream of a first data type including information related to motion, sound, comments, and / or gifting by executing an installed web browser or a specific application, obtaining a second stream of a second data type including video, synthesizing the data related to the first stream and the data related to the second stream, and providing the synthesized data. That is, the terminal device 30 may be any terminal device capable of executing such operations, such as a smartphone, tablet, mobile phone (feature phone), and / or personal computer, but is not limited thereto.

[0018] Next, an example of the hardware configuration of the server device 20 will be described.

[0019] 2. Hardware Configuration of Each Device

[0020] An example of the hardware configuration of the server device 20 will be described with reference to FIG. 2. FIG. 2 schematically shows an example of the hardware configuration of the server device 20 (terminal device 30) shown in FIG. 1.

[0020] 2-1. Hardware Configuration of Server Device 20 An example of the hardware configuration of the server device 20 will be described with reference to FIG. 2. FIG. 2 schematically shows an example of the hardware configuration of the server device 20 (terminal device 30) shown in FIG. 1. shows an example of the hardware configuration of the server device 20 (terminal device 30) shown in FIG. 1 in a schematic block diagram. It is a block diagram (in FIG. 2, the reference numerals in parentheses are those described in relation to each terminal device 3 as described later). (.)

[0021] As shown in FIG. 2, the server device 20 can mainly include an arithmetic device 21, a main storage device 22, and an input / output interface device 23. The server device 20 can further include an input device 24 and an auxiliary output device 26. These devices can be connected to each other by a data bus and / or a control bus. The arithmetic device 21 performs calculations using the instructions and data stored in the main storage device 22, and stores the results of the calculations in the main storage device 22. Further, the arithmetic device 21 can control the input device 24, the auxiliary storage device 25, the output device 26, etc. via the input / output interface device 23. The server device 20 may include one or more arithmetic devices 21. The arithmetic device 21 may include one or more central processing units (CPUs), microprocessors, and / or graphics processing units (GPUs).

[0022]

[0023] The main storage device 22 has a storage function, and stores instructions and data received from the input device 24, the auxiliary storage device 25, the communication network 10, etc. (server device 20, etc.) via the input / output interface device 23, as well as the calculation results of the arithmetic device 21. The main storage device 22 can include, without being limited to, RAM (random access memory), ROM (read only memory), and / or flash memory, etc.

[0024] The auxiliary storage device 25 is a storage device. The above specific applications and web browsers Instructions and data (computer programs) that make up the like may be stored, and by being controlled by the arithmetic unit 21 these instructions and data (computer programs) may be loaded into the main storage device 22 via the input / output interface device 23. The auxiliary storage device 25 may be a magnetic disk device and / or an optical disk device, a file server, etc., and is not limited thereto.

[0025] The input device 24 is a device that captures data from the outside, and may be a touch panel, buttons, a key board, a mouse and / or a sensor, etc.

[0026] The output device 26 may include, without being limited thereto, a display device, a touch panel and / or a printer device, etc. Also, the input device 24 and the output device 26 may be integrated with each other.

[0027] In such a hardware configuration, the arithmetic unit 21 sequentially loads instructions and data (computer programs) that make up a specific application stored in the auxiliary storage device 25 into the main storage device 22, and by computing the loaded instructions and data, controls the output device 26 via the input / output interface device 23, or alternatively, via the input / output interface device 23 and the communication network 10, various information may be transmitted and received with other devices (for example, the server device 20 and other terminal devices 30, etc.). When the server device 20 has such a configuration and executes a specific installed application, motion, sound, comment, and / or gift defined below

[0028] ​​​​​​​Obtain a first stream of a first data type including information related to wings, and obtain a second stream of a second data type including video. Then, synthesize the data related to the first stream and the data related to the second stream, and provide the synthesized data. Some or all of the operations (including various operations to be described in detail later) may be executed. Also, such operations may be performed when the user gives an instruction to the system of an example of the invention disclosed in this application using the input device 24 or the input device 34 related to the terminal device 30 to be described later. Further, when the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. Obtain a second stream of a second data type, and synthesize the data related to the first stream and the data related to the second stream, and provide the synthesized data. Some or all of the operations (including various operations to be described in detail later) may be executed. Also, such operations may be performed when the user gives an instruction to the system of an example of the invention disclosed in this application using the input device 24 or the input device 34 related to the terminal device 30 to be described later. Further, when the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. ream, and provide the synthesized data. Some or all of the operations (including various operations to be described in detail later) may be executed. Also, such operations may be performed when the user gives an instruction to the system of an example of the invention disclosed in this application using the input device 24 or the input device 34 related to the terminal device 30 to be described later. Further, when the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. Including various operations to be described in detail later), some or all of which may be executed. Also, such operations may be performed when the user gives an instruction to the system of an example of the invention disclosed in this application using the input device 24 or the input device 34 related to the terminal device 30 to be described later. Further, when the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. The user may operate by giving an instruction to the system of an example of the invention disclosed in this application using the input device 24 or the input device 34 related to the terminal device 30 to be described later. Also, when the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. The user may operate by giving an instruction to the system of an example of the invention disclosed in this application using the input device 24 or the input device 34 related to the terminal device 30 to be described later. Also, when the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. When the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. When the program is executed on the arithmetic unit 21, it may be displayed by the output device 26 of the server device 20 as the system used by the user, or may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user. It may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user.

[0029] 2-2. Hardware Configuration of Terminal Device 30 The hardware configuration example of the terminal device 30 will be described with reference to FIG. 2 as well. As for the hardware configuration of each terminal device 30, for example, the same configuration as that of each server device 20 described above can be used. Therefore, the reference numerals for the constituent elements of each terminal device 30 are shown in parentheses in FIG. 2. As the hardware configuration of each terminal device 30, for example, the same configuration as that of each server device 20 described above can be used. Therefore, the reference numerals for the constituent elements of each terminal device 30 are shown in parentheses in FIG. 2. Therefore, the reference numerals for the constituent elements of each terminal device 30 are shown in parentheses in FIG. 2. Therefore, the reference numerals for the constituent elements of each terminal device 30 are shown in parentheses in FIG. 2.

[0030] As shown in FIG. 2, each terminal device 30 mainly includes an arithmetic unit 31, a main storage device 32, an input / output interface device 33, an input device 34, an auxiliary storage device 35, and an output device 36. These devices are connected to each other by a data bus and / or a control bus. As shown in FIG. 2, each terminal device 30 mainly includes an arithmetic unit 31, a main storage device 32, an input / output interface device 33, an input device 34, an auxiliary storage device 35, and an output device 36. These devices are connected to each other by a data bus and / or a control bus. As shown in FIG. 2, each terminal device 30 mainly includes an arithmetic unit 31, a main storage device 32, an input / output interface device 33, an input device 34, an auxiliary storage device 35, and an output device 36. These devices are connected to each other by a data bus and / or a control bus. These devices are connected to each other by a data bus and / or a control bus.

[0031] The arithmetic unit 31, the main storage device 32, the input / output interface device 33, the input device 34, the auxiliary The memory device 35 and the output device 36 may be substantially the same as, or different from, the arithmetic unit 21, the main memory device 22, the input / output interface device 23, the input device 24, the auxiliary storage device 25, and the output device 26 included in each of the above-described server devices 20. Also, the capacities and capabilities of the arithmetic unit and the memory device may be different. In particular, the input device 34 and the output device 36 will be described in detail below. The arithmetic unit, main memory device, input / output interface device, input device, auxiliary storage device, and output device may be substantially the same as, or different from, those of the server device 20 described above. Also, the capacities and capabilities of the arithmetic unit and the memory device may be different. In particular, the input device 34 and the output device 36 will be described in detail below. In such a hardware configuration, the arithmetic unit 31 loads the instructions and data (computer program) that make up a specific application stored in the auxiliary storage device 35 into the main memory device 32 in sequence, and by calculating the loaded instructions and data, controls the output device 36 via the input / output interface device 33, or alternatively, via the input / output interface device 33 and the communication network 10, can perform various information transmissions and receptions with other devices (for example, each server device 20, etc.). The terminal device 30 may be configured as described above and may execute some or all of the operations such as obtaining a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtaining a second stream of a second data type including video, combining the data related to the first stream and the data related to the second stream, and providing the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document.

[0032] In such a hardware configuration, the arithmetic unit 31 loads the instructions and data (computer program) that make up a specific application stored in the auxiliary storage device 35 into the main memory device 32 in sequence, and by calculating the loaded instructions and data, controls the output device 36 via the input / output interface device 33, or alternatively, via the input / output interface device 33 and the communication network 10, can perform various information transmissions and receptions with other devices (for example, each server device 20, etc.). In such a hardware configuration, the arithmetic unit 31 loads the instructions and data (computer program) that make up a specific application stored in the auxiliary storage device 35 into the main memory device 32 in sequence, and by calculating the loaded instructions and data, controls the output device 36 via the input / output interface device 33, or alternatively, via the input / output interface device 33 and the communication network 10, can perform various information transmissions and receptions with other devices (for example, each server device 20, etc.). In such a hardware configuration, the arithmetic unit 31 loads the instructions and data (computer program) that make up a specific application stored in the auxiliary storage device 35 into the main memory device 32 in sequence, and by calculating the loaded instructions and data, controls the output device 36 via the input / output interface device 33, or alternatively, via the input / output interface device 33 and the communication network 10, can perform various information transmissions and receptions with other devices (for example, each server device 20, etc.).

[0033] The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. The terminal device 30 having such a configuration may obtain a first stream of a first data type including information related to motion, sound, comments, and / or gifting, obtain a second stream of a second data type including video, combine the data related to the first stream and the data related to the second stream, and provide the combined data (including various operations to be described in detail later). Also, by executing the installed web browser, a web page can be received and displayed from the server device 20, and the same operations can be performed. Also, such operations, etc., may be operated by the user using the input device 34 to give an instruction to the system of an example of the invention disclosed in this application document. Also, when the program is being executed on the arithmetic unit 31, it may be configured to be displayed on the output device 36 of the terminal device 30 as the system used by the user.

[0034] In the present application documents, the terminal device may also be referred to as a "user terminal", and a person who uses the user terminal is referred to as a user (when the application to be supported is a game, the user may be a spectator of the game, a player of the game, a referee of the game, a character appearing in the game, etc., who are related to the game. These spectators of the game, referees of the game, characters appearing in the game, etc., who are related to the game are subordinate concepts of the user.). Also, among the user terminals, a terminal capable of providing an information source may also be referred to as an "information source terminal" (therefore, if it is an information source terminal, it is a user terminal). A person who uses the information source terminal may also be referred to as an information source provider (when the application to be supported is a game, the information source provider may be a player of the game. The player of the game may be a subordinate concept of the information source provider, and the information source provider may be a subordinate concept of the user.). Therefore, for a single information processing device, this device may be referred to as an information source terminal from the perspective of the information source provider who uses such a device, and at the same time, may be referred to as a user terminal from the perspective of using such a device as a user. Note that in the system according to the present application, the user may be managed by an ID for identifying the user. Such an ID for identifying the user may be stored in the system according to the present application and may be managed within the system according to the present application. A single user may have one or more IDs. In the present application, the user terminal used with the ID for identifying user A is, for example, the user terminal used by user A, the user terminal related to user A, may be referred to as etc. Similarly, the information used as the ID for identifying the information source provider A The source terminal may be referred to as the information source terminal used by the information source provider A, the information source terminal related to the information source provider A, etc. and so on.

[0035] The user terminal may include various devices as the input device 34 that can input information and acquire information. Further, the user terminal may generate information by the arithmetic device 31. The information acquired (hereinafter, "acquired" may also include the case of acquiring the generated information) within such a user terminal may be transmitted to the communication network 10 via the input / output interface device 23.

[0036] For example, the user terminal may have a keyboard, a touch panel, etc. as the input device 34 that can input a comment and acquire the comment as text information. Further, the user terminal may have a function of generating a certain comment by the arithmetic device 34. These acquired comments may similarly be transmitted to the communication network 10 via the input / output interface device 23.

[0037] Note that the comment according to the present invention may be, in addition to the comment input by the user terminal, one automatically generated by the system. Such a comment may be transmitted to the user terminal and displayed at the user terminal. As the comment automatically generated by the system, for example, it may be a comment explaining the situation. The comment explaining the situation may be a comment caused by the user's behavior or a comment not caused by the user's behavior. As the comment caused by the user's behavior, for example, when a certain user opens the use it may be a comment caused by the user's behavior or a comment not caused by the user's behavior. As the comment caused by the user's behavior, for example, when a certain user opens the use ​​​​​​​​​When the system starts, it can say something like "Mr. A has entered the room." Or, when the user is a specific person, If it's more than the number, you can say something like "The number of viewers has exceeded X." In cases where the number of users is greater than the specified number, the number of users is limited to a certain number of games, applications, videos, etc. The number of viewers of a stream can be the total number of viewers since the beginning of the stream, or the number of viewers of the current stream. This may be the total number of viewers from the beginning, or this total minus those who did not watch. When the total number of people currently viewing the current broadcast is greater than the specified number In addition, in cases where gifting is done as a comment resulting from the user's actions, In this case, a comment such as "Person A gave item B" may be used.

[0038] The user terminal may also include a microphone capable of acquiring sound as the input device 34. The human voice may be captured by a microphone as the input device 34 and treated as sound. After being acquired by a microphone as an input device 34, the speech is recognized by the computing device 31. Alternatively, the user terminal may convert the input data into text and treat it as a comment. These captured sounds or comments may also be used to The information may be transmitted to the communication network 10 via the interface device 23.

[0039] In addition, the information source terminal, which is one aspect of the user terminal, detects human movement as an input device 34. The motion sensor may be capable of detecting the motion of the user or information provider. The device may be worn on the body and have a function capable of detecting the movement of each part of the body. Various locations may be included, such as the head, face, upper arms, forearms, hands, thighs, lower legs, feet, and trunk. is not limited to some or all of these and can be worn at a position where body movements can be detected and may be. The motion sensor may include an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. Similarly, this acquired information may be transmitted to the communication network 10 via the input / output interface device 23.

[0040] Also, the information source terminal, which is one aspect of the user terminal, may include an imaging device capable of capturing a moving image as the input device 34. The imaging device may image a part of the head, face, eyes, body, the whole body, etc. of the user or the information source provider. When the imaging device images the head or face, it can detect the direction of the face, and when imaging the face, it can detect the expression, and when imaging the eyes it can detect the line of sight, and when imaging a part of the body or the whole body, it may be possible to detect body movements together with or instead of the above-mentioned motion sensor. Based on this imaging information, the information obtained about the face direction, line of sight, expression, and body movements may similarly be transmitted to the communication network 10 via the input / output interface device 23.

[0041] The output device 36 of the user terminal may be a display device or a sound generating device. The display device may be various devices as long as it has a display function, for example, a liquid crystal display, a plasma display panel, an organic EL display, a micro LED display, a head-mounted display and so on, but is not limited to these and may be various devices. The sound generating device may be a speaker headphones, earphones, etc. The information output by the output device 36 may be the information generated in the arithmetic device 31 or the information received from the communication network 10 ​​​​

[0042] 3. Functions of Each Device Next, an example of the functions of each of the server device 20 and the terminal device 30 will be described with reference to FIG. 3. FIG. 3 is a block diagram schematically showing an example of the functions of the system shown in FIG. 1. As shown in FIG. 3, an example system may include an acquisition unit 41 that acquires various data, a synthesis unit 42 that synthesizes some or all of the acquired data, and a provision unit 44 that provides the synthesized data. Further, an example system may include a control information management unit 43 that manages control information. Note that FIG. 3 focuses on functions, and as is clear from each description in this application document, the hardware on which such functions are executed may be a server, an information source terminal, a user terminal, or a combination thereof. Therefore, the functions related to each example system may be executed on a server, on an information source terminal, on a user terminal, or may be distributed and executed on a plurality of these hardware. Note that FIG. 3 illustrates a mode in which the acquisition unit 41 acquires information from an information source terminal, but as will be described later, the acquisition unit 41 may acquire information from a user terminal. Similarly, the provision unit 44 may also provide information to a user terminal or to an information source terminal, which is a mode of a user terminal. The acquisition unit 41 has a function of acquiring information related to the system of this example. The information related to the system of this example acquired by the acquisition unit 41 may be of various data types. The acquisition unit 41 may acquire, as data, a stream (sometimes also called streaming).

[0043] 3-1. Acquisition Unit 41

[0044] ​​​​​​​​​​​​​ In this application document, it may be obtained by unifying with the notation of "stream". That is, the acquisition unit 41 may acquire a first stream of the first data type including information related to motion, sound, comment, and / or gifting, and may also acquire a second stream of the second data type including video. The content of the data carried by the stream may include motion, sound, comment, intermediate generated information, information related to gifting, video (including moving images and still images), etc. The video may be related to a game or captured by an imaging device. The video related to a game may be a video executed within the game or a game video in which a player is playing within the game. Also, as for what is captured by the imaging device, there is no limitation to the type, and it may be various things such as a video capturing nature, a video capturing a city, a video of an attraction or an event, etc. Also, the content of the data carried by the stream may, in addition to or instead of the above, be face direction information, gaze information, expression information, etc. In this application document, for convenience, the stream may be classified into a first data type and a second data type. The first data type may include motion, sound, comment, intermediate generated information, information related to gifting, etc., but is not limited to these, and may be the above-mentioned face direction information, gaze information, expression information, etc. The second data type may be data having a basic display background function, for example, video. Motion, sound, and comment may be as described above.

[0045]

[0046]

[0047] ​​​​​​​​​​​The intermediate generated information is information used for the synthesis described later. By using the intermediate generated information, there is an advantage that the overall transmitted capacity can be reduced and the burden on the communication band can be decreased.

[0048] Examples of the intermediate generated information include depth information, mask information, low-resolution rendering images, font rendering, UI (HP, MP, score, language-dependent information, personal information), and the like. .

[0049] Regarding the depth information, for at least two pieces of information such as a video and an avatar, when the depth information of each of the depths of one and the other is known, there may be a case where one does not need to be used, and there is an advantage that it is not necessary to send information regarding the one that cannot be used. For example, when the avatar moves to the back side of the game video being relayed live and the avatar becomes completely invisible due to the game video being relayed live, it is not necessary to send information regarding the invisible avatar, and the communication volume can be reduced. For this reason, the information regarding the depth information of the information to be displayed may be sent as the intermediate generated information. That is, the first data type may include the intermediate generated information related to the video, and the intermediate generated information may include the depth information related to the video.

[0050] The mask information may be information related to covering a predetermined location. For example, the mask information may be coordinate information for indicating a region to cover a predetermined region of the video. Also, the mask information may include information related to the covering time. By synthesizing the video information and the mask information, there is an advantage that it is possible to cover a predetermined location of the video.

[0051] Examples of the low-resolution rendering image include, for example, thumbnails. The resolution of the original image It may be an image with a lower resolution than. Such a resolution lower than the original image resolution Since it is an image of, it is possible to reduce the communication burden. Therefore, for example, when displaying thumbnails of a plurality of images on the user terminal, instead of communicating those original images by synthesizing the thumbnails with other data and communicating them, the communication capacity can be reduced and one or more thumbnails can be transmitted compared to directly communicating the original images, which has the advantage of being able to send them.

[0052] Font rendering may be information required when rendering a font. When communicating after rendering the font, since it becomes image information due to the rendering the communication volume may increase. Therefore, by communicating the setting information of the font regarding what kind of rendering to perform and using such setting information to perform font rendering at the stage of display, there is an advantage that the communication burden can be reduced. at the stage of display, there is an advantage that the communication burden can be reduced. There is an advantage that the communication burden can be reduced.

[0053] HP, MP, score, language-dependent information, personal information, etc. are information related to the user's profile of the user terminal to be displayed. Here, HP, MP, and score indicate information of the user, not information related to the information source provider. Language-dependent information and personal information are also information related to the user's profile. These are to be displayed on the user terminal but it is sufficient if they are synthesized at the stage of being displayed on the user terminal, and they may be different information for each user terminal to be displayed. Therefore, these information may be communicated separately from the original data. Note that the language-dependent information may be set according to each individual user terminal as described above, or for the same synthesized such as watching the same video or participating in the same game, it is sufficient if they are synthesized at the stage of being displayed on the user terminal, and they may be different information for each user terminal to be displayed. Therefore, these information may be communicated separately from the original data. Note that the language-dependent information may be set according to each individual user terminal as described above, or for the same synthesized content such as watching the same video or participating in the same game, it may be set according to each individual user terminal, or may be the same for the same synthesized content such as watching the same video or participating in the same game. Of the user terminals that view the obtained data, the language used in the largest number of user terminals may be synthesized in the server, user terminal, or information source terminal. The largest number of The language used in user terminals may be determined in the server or information source terminal, etc., and such determination may be made by obtaining the languages used in each user terminal, counting the number of user terminals using each language, and identifying the language used by the largest number of user terminals among the languages used. Also, the language environment in such language-dependent information may be set by referring to locale information that is the setting of the country or region where the user terminal is located, and the language environment may also be linked to coordinates and directions in real space such as the position of the user terminal measured by the GPS provided in the terminal and the direction the terminal is facing, or the seat information of the person holding the user terminal in an event venue or movie theater. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus.

[0054] As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus. As described above, the information processing apparatuses for synthesizing each data are not all the same, and may be synthesized on a suitable information processing apparatus according to the information that can be synthesized. For example, information related to font rendering, personal profiles such as HP, MP, score, language-dependent information, and personal information may be synthesized on the user terminal to be displayed, but information closely related to video such as background information may be synthesized on the server apparatus.

[0055] Information related to gifting is information related to providing gifts. Information related to gifting includes, for example, the type of gift provided, the quantity of the gift provided, the owner information of the gift provided, the recipient information of the gift provided, and the timing of providing the gift provided. Information related to gifting is information related to providing gifts. Information related to gifting includes, for example, the type of gift provided, the quantity of the gift provided, the owner information of the gift provided, the recipient information of the gift provided, and the timing of providing the gift provided. Information related to gifting is information related to providing gifts. Information related to gifting includes, for example, the type of gift provided, the quantity of the gift provided, the owner information of the gift provided, the recipient information of the gift provided, and the timing of providing the gift provided. The owner information and the other party information may be part or all of the information (e.g., the date and time of the provision). User name, ID, user terminal ID, or other information that can identify the owner or other party In addition, other information such as gift amount, gift level, and gift restrictions are included. This is also fine.

[0056] The acquisition unit 41 also acquires control information (to be described later) managed by the control information management unit 43. Examples of the control information include viewpoint control information and performance control information. The acquisition unit 41 may acquire the data as a stream.

[0057] The stream may or may not include a timestamp. In the case where the first and second streams are included, there is an advantage that time adjustment is possible. Each stream may include time information.

[0058] The protocol of the information acquired by the acquisition unit 41 may be any protocol. As a protocol, it is mainly a two-way interactive transmission of information. It may be a protocol that mainly transmits information in one direction. For example, RTMP (Real Time Messaging Protocol) is one example. In the latter case, HTTP (Hypertext Transfer Protocol l), HLS (HTTP Live Streaming), WebSocket (RF C6455), but is not limited to these protocols. The communication path (CDN) may be different, and the billing model may be different for each process. It may also be something.

[0059] The acquisition unit 41 may acquire information from within the same information processing apparatus as the program in which the acquisition unit 41 is executed, or from an information processing apparatus different from the program in which the acquisition unit 41 is executed. In the latter case, the acquisition unit 41 may acquire information from another information processing apparatus on the network. The acquisition unit 41 may acquire information directly or indirectly from another information processing apparatus. In particular, in recent years, since the network may be configured complexly, the indirect acquisition means that the information processing apparatus in which a program having an acquisition function is executed acquires information through one or more information processing apparatuses intervening between it and another information processing apparatus. Also, the acquisition unit 41 may acquire information from a user terminal or from an information source terminal. The acquisition unit 41 may store the information acquired by the acquisition unit 41 in the system.

[0060] The acquisition unit 41 may manage the information acquired by the acquisition unit 41. Note that the acquisition unit 41 may appropriately set the storage period according to the type of information acquired by the acquisition unit 41.

[0061] 3-2. Composition Unit 42 The synthesis unit 42 has a function of synthesizing the streams acquired by the acquisition unit 41. For example, it may have a function of synthesizing data related to a first stream and data related to a second stream.

[0062] The synthesis method may be various methods. The synthesis method itself may use various methods for synthesizing information of one stream and another stream. For example, one stream and another stream may be overlapped in their respective shapes. In this case, for example, one rectangular stream A team and another rectangular stream may be superimposed and combined. Also, the shape of one stream and / or the shape of the other stream may be deformed and then superimposed. For example, the shape of the other stream may be changed so that it can be fitted into a part within the shape of one stream, and after deforming the shapes to fit together like in an inset composition, they may be superimposed and combined. Also, after applying a certain effect to one stream and / or one or both of the other streams, they may be combined. In this case, examples of the effect include changing color, style, saturation, etc., and it may be made to fit into the space. Also, when appropriately supplementing missing information during composition, there is an advantage that the amount of information to be communicated is reduced, thus reducing the communication burden. For example, motion may be limited to information about movement, and by supplementing information about the avatar that is the object to be moved, the movement of the avatar can be realized. In this case, instead of communicating information including the avatar, only the movement of the avatar is communicated,

[0063] so there is an advantage of reducing the communication burden. For such composition, the combining unit 42 may have model data of the avatar related to the motion. The combining unit 42 may synthesize the model data of the avatar related to the motion and the motion to realize the movement of the avatar related to the motion. Also, the combining unit 42 may have object data. The object data may include data related to the background of the virtual space where the avatar exists and various objects drawn on such background. By using the object data, the combining unit 42 may synthesize various objects that can be drawn into an image.

[0064] ​​​​​​​​For example, when the motion of the avatar related to the first stream and the background video related to the second stream are synthesized, the synthesizing unit 42 may reproduce the motion of the avatar using the motion obtained by using the avatar stored in advance, and synthesize this avatar and the background video. Here, the synthesizing unit 42 may have information about the avatar in advance, and the synthesizing unit 42 may have a function of reproducing the motion of the avatar by acquiring the motion information associated with the avatar. In this case, compared with the case where the motion of the avatar itself is acquired from the information source terminal as a video, the motion is acquired from the information source terminal and synthesized with the information of the avatar that is the basis of the motion stored in the synthesizing unit 42, so that there is an advantage that the information of the motion of the avatar can be efficiently transmitted. Further, the synthesizing unit 42 may have information of one or more avatars, and one or more such avatars may be stored in association with the motion based on one or more information source terminals. Thereby, there is an advantage that the motion acquired from one information source terminal and the motion acquired from another information source terminal can be distinguished and the motion of the avatar can be transmitted using the avatars stored in the synthesizing unit 42 in association with each other. Also, instead of the background video, it may be a game. When the acquisition unit 41 acquires the motion of the avatar related to the first stream, the background video related to the second stream, and the game, the synthesizing unit 42 may reproduce the motion of the avatar using the motion obtained by using the avatar stored in advance, and synthesize this avatar and the game video. In this case, there is an advantage that a situation where the avatar is performing a live relay of the game can be reproduced.

[0065]

[0066] Also, for example, when information related to gifting and the video being distributed are combined , information related to gifting obtained from various information source terminals can be reflected in the video being distributed, and while efficiently transmitting information, the information source provider related to the information source terminal has the advantage of being able to obtain a feeling of participating in the video through gifting. That is, the first data type includes information related to gifting, the second data type includes the video being distributed , and the combination may include the information related to gifting being reflected in the video being distributed.

[0067] In addition, by adopting a method of combining one stream and another stream, there is an advantage that the communication quality of each stream can be set according to the content. For example, for the stream related to the second data type which is a background video, even if a part of the quality deteriorates, it can be filled in using the already communicated information, so the quality does not have to be high. On the other hand, since the stream related to the first data type is meaningful for each piece of information, it is preferable that the quality is high. By adopting such a method of combination, there is an advantage that appropriate combined information can be provided without demanding excessive communication quality.

[0068] The information processing device that executes the function of combining by the combining unit 42 may be referred to as a "combining device". The combining device may be any of a server device, an information source terminal, and a user terminal.

[0069] That is, the combination may be performed by the server. Also, the first stream is obtained from the first information source terminal , the second stream is obtained from the second information source terminal, and the combination is performed by the first information source terminal​​​​​​ Alternatively, it may be performed at the second information source terminal. Also, these synthesized data may be provided to one or a plurality of user terminals. Further, the synthesis may be performed at the user terminal, and the synthesized data may be provided at the user terminal.

[0070] Also, with respect to the first stream and the second stream obtained from the first information source terminal, they may be synthesized at the first information source terminal and provided to one or a plurality of user terminals, synthesized at the server and provided to one or a plurality of user terminals, or synthesized at the user terminal and provided to the user terminal.

[0071] FIGS. 5 to 7 illustrate the location of the information processing apparatus in which the synthesis unit 42 performs the function of stream synthesis according to the above-described pattern classification. FIG. 5 illustrates a situation in which information and streams obtained from information source terminals 51A and 51B are synthesized in the server device 50 and provided to user terminals 5 2A and 52B. FIG. 6 illustrates a situation in which information and streams obtained from information source terminals 51A and 5 1B are synthesized in the information source terminal 51B and provided to user terminals 52A and 52B. FIG. 7 illustrates a situation in which information and streams obtained from information source terminals 51A and 51B are synthesized in the user terminal 52A and provided to the user terminal 52A, and also synthesized in the user terminal 52B and provided to the user terminal 52B. Note that the synthesis function by the synthesis unit 42 can be classified according to the patterns as shown in these figures, but other functions performed by the synthesis unit 42 are not necessarily the synthesis functions classified in such patterns, and ...

[0072] ... ... It is not necessary to be executed within the same information processing apparatus and may be executed within other information processing apparatuses. . That is, the synthesizing unit 42 may be executed within one or a plurality of information processing apparatuses. For example, the first stream related to the first data type including motion, the second stream related to the second data type including video, and the third stream related to the first data type including sound are acquired, and within the first information processing apparatus, the first stream related to the first data type including motion and the second stream related to the second data type including video are synthesized, and the synthesized result of the first stream and the second stream and the third stream related to the first data type including sound are provided to the second information processing apparatus. Within the second information processing apparatus, the synthesized result of the first stream and the second stream and the third stream are synthesized, and such synthesized result may be provided to within the second information processing apparatus or the third information processing apparatus.

[0073] Also, as another aspect, after synthesizing the data related to the first stream of the first data type including the acquired motion, sound, comment, and / or information related to gifting, the data related to the second stream of the second data type including the acquired video, the third stream of the second data type may be acquired from the terminal that provided the first stream, the terminal that provided the second stream, or other terminals, and synthesized with the data related to the synthesized stream and the data related to the third stream. Alternatively, the fourth stream of the first data type may be acquired from the terminal that provided the first stream, the terminal that provided the second stream, or other terminals, and synthesized with the data related to the synthesized stream and the data related to the fourth stream.

[0074] Also, FIGS. 8 to 11 are classifications including the data to be transmitted regarding the synthesis mode. . In these figures and the corresponding descriptions, the parts illustrated as CDN (HLS) intend to show an example using CDN (Content Delivery Network) and then using HLS ( HTTP Live Streaming), but it is just an example. Examples where CDN is used but HLS is not used, examples where HLS is used without using CDN, or other examples may also be possible.

[0075] FIG. 8 shows a configuration where the information processing device 803 acquires the sound 801 and the motion 802, synthesizes them with the avatar video corresponding to the motion within the information processing device 803, and sends the synthesized sound and video to the RTMP server 805 at 804. The RTMP server 805 acquires it and the RTMP server 805 sends it to the CDN (HLS) 806, and the CDN (HLS) 806 distributes it to the viewer A80 7. Here, the information processing device 803 may be an information source terminal or a server device that acquires information from the information source terminal. Also, the information processing device 803 may be equipped with OBS (Open Broadcaster Software) and may acquire information from other information processing devices via HDMI, AirPlay, etc.

[0076] FIG. 9 shows that the host 803, which is an information processing device, acquires the sound 801A and the motion 802, synthesizes them with the avatar video corresponding to the motion within the information processing device 803, and sends the synthesized sound and video at 805. The RTMP server 807 acquires it and the RTMP server 807 sends it to the C Send it to DN(HLS)808, and CDN(HLS)808 sends the synthesized audio and video to game server A811 at 809A, to guest B812 at 809B, and to viewer 813 at 809C. Here, host 803, which is an information processing device, may be an information source terminal or a server device that acquires information from an information source terminal. Guest A811 and guest B812 may be information source terminals, and each information source provider can input and output audio 801B and audio 801C bidirectionally and can view video 810A and video 811B respectively. Also, viewer 813 may be a user terminal and can view video 810C. Host 803, guest A811, and guest B812, which are information processing devices, may be able to communicate bidirectionally about audio with the collaboration server 806 (804A, 804B, 804C). In this figure, two information processing devices, guest A810 and guest B811, are illustrated, but there may be one or two or more. Similarly, viewer 813, which is an information processing device, may be one or two or more. Send it to game server A811 at 809A, to guest B812 at 809B, and to viewer 813 at 809C. Here, host 803, which is an information processing device, may be an information source terminal or a server device that acquires information from an information source terminal. Guest A811 and guest B812 may be information source terminals, and each information source provider can input and output audio 801B and audio 801C bidirectionally and can view video 810A and video 811B respectively. Also, viewer 813 may be a user terminal and can view video 810C. Host 803, guest A811, and guest B812, which are information processing devices, may be able to communicate bidirectionally about audio with the collaboration server 806 (804A, 804B, 804C). In this figure, two information processing devices, guest A810 and guest B811, are illustrated, but there may be one or two or more. Similarly, viewer 813, which is an information processing device, may be one or two or more. Figure 10 shows that host 804, which is an information processing device, and guest A805, which is an information processing device, can acquire motion 801A and motion 801B respectively, can input and output audio 802A and audio 802B bidirectionally respectively, and can view video 803A and video 803B respectively. Host 804, which is an information processing device, may be configured to transmit audio bidirectionally to the collaboration server 809 at 806A, transmit motion bidirectionally at 807A, and transmit gifting information bidirectionally at 808A. Also, guest, which is an information processing device In this figure, two information processing devices, guest A810 and guest B811, are illustrated, but there may be one or two or more. Similarly, viewer 813, which is an information processing device, may be one or two or more. Figure 10 shows that host 804, which is an information processing device, and guest A805, which is an information processing device, can acquire motion 801A and motion 801B respectively, can input and output audio 802A and audio 802B bidirectionally respectively, and can view video 803A and video 803B respectively.

[0077] Host 804, which is an information processing device, may be configured to transmit audio bidirectionally to the collaboration server 809 at 806A, transmit motion bidirectionally at 807A, and transmit gifting information bidirectionally at 808A. Also, guest, which is an information processing device Figure 10 shows that host 804, which is an information processing device, and guest A805, which is an information processing device, can acquire motion 801A and motion 801B respectively, can input and output audio 802A and audio 802B bidirectionally respectively, and can view video 803A and video 803B respectively. Host 804, which is an information processing device, may be configured to transmit audio bidirectionally to the collaboration server 809 at 806A, transmit motion bidirectionally at 807A, and transmit gifting information bidirectionally at 808A. Also, guest, which is an information processing device Figure 10 shows that host 804, which is an information processing device, and guest A805, which is an information processing device, can acquire motion 801A and motion 801B respectively, can input and output audio 802A and audio 802B bidirectionally respectively, and can view video 803A and video 803B respectively. A805 can be configured to transmit audio bidirectionally to the collaboration server 809, motion bidirectionally to 807B, and gifting information bidirectionally to 808B. In particular, the sound, motion, and gifting information acquired from the guest A805 via the collaboration server 809 can be synthesized with the sound, motion, and gifting information acquired at the host 804 at the host 804 and transmitted as video to the RTMP server 810, then to the CDN (HLS) 811, and then to the viewer 812. Note that in this figure, the transmission path of information from the host 804 to the guest A805 via the collaboration server 809 is illustrated, but this is not limited to such an example. For example, information can be transmitted bidirectionally from the host 804 via the collaboration server 809 to a guest B, guest C, guest D, etc. (not shown) that have functions similar to those of the guest A. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. In this figure, the transmission path of information from the host 804 to the guest A805 via the collaboration server 809 is illustrated, but this is not limited to such an example. For example, information can be transmitted bidirectionally from the host 804 via the collaboration server 809 to a guest B, guest C, guest D, etc. (not shown) that have functions similar to those of the guest A. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. to a guest B, guest C, guest D, etc. (not shown) that have functions similar to those of the guest A. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. server 809 to a guest B, guest C, guest D, etc. (not shown) that have functions similar to those of the guest A. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. Similarly, information can be transmitted bidirectionally. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. Similarly, information can be transmitted bidirectionally. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812. Similarly, information can be transmitted bidirectionally. Also, in this figure, the path of information transmission from the host 804 to the viewer 812 via the RMP server 810 and the CDN (HLS) 811 is shown, but this is not limited to such an example. For example, information can be transmitted from the host 804 via the RTMAP 810 and the CDN (HLS) 811 to one or more other viewers (not shown) that have functions similar to those of the viewer 812.

[0078] Figure 11 shows that the host 803, which is an information processing device, acquires gifting information 801A and motion 802A, and the host 803, which is an information processing device, transmits sound 805A, gifting information 806A, and motion 807A to the client rendering server 808. The client rendering server 808 transmits these to the viewer A, which is an information processing device. which is an information processing device. Transmit the sound 805B, gifting information 806B, and motion 807B to 804, and synthesize sound and motion in the viewer A804, which is an information processing device, and provide the gifting information 8 01B and the video 802B. In this figure, the information transmission path from the host 803 to the viewer A80 4 via the client rendering server 808 is illustrated. However, it is not limited to such an example. For example, information may be transmitted bidirectionally or unidirectionally from the host 803 to viewers B, C, D, etc. that have functions similar to those of the viewer A and are not shown, via the client rendering server 80 8. In addition, as described in each of the above figures, the first stream and the second stream obtained from one or a plurality of information source terminals may be combined with various combinations of video and audio. For example,

[0079] consider an example where the first stream obtained from the first information source terminal is video, the second stream obtained from the first information source is audio, the third stream obtained from the second information source terminal is video, and the fourth stream obtained from the second information source terminal is audio. In this case, the combining unit 42 may combine all of the first to fourth streams, or may combine a part of the first to fourth streams. Specifically, the combining unit 42 may combine the first to third streams, or may combine the first, second, and fourth streams. Also, when the fourth stream is not obtained from the second information source terminal and only the third stream is obtained, the combining unit 42 may combine the first to third streams, or may combine the first and third streams, or may combine the second and third streams. Further, when the third stream is not obtained from the second information source terminal and only the fourth stream is obtained, the combining unit 42 may combine the first to third streams, or may combine the first and third streams, or may combine the second and third streams. Specifically, the combining unit 42 may combine the first to third streams, or may combine the first, second, and fourth streams. Also, when the fourth stream is not obtained from the second information source terminal and only the third stream is obtained, the combining unit 42 may combine the first to third streams, or may combine the first and third streams, or may combine the second and third streams. Further, when the third stream is not obtained from the second information source terminal and only the fourth stream is obtained, the combining unit 42 may combine the first to third streams, or may combine the first and third streams, or may combine the second and third streams. When only the fourth stream is acquired, the combining unit 42 may combine the first, second, and fourth streams, or may combine the first and fourth streams, or may combine the second and fourth streams. In the description of this paragraph, the stream has been described as video or audio, but the stream may include both audio and video. Of course, these are just examples, and it goes without saying that various other combining modes are possible. In addition, when the acquisition unit 41 acquires a plurality of streams, the combining unit 42 may perform temporal adjustment on such a plurality of streams. For example, when one or a plurality of streams acquired by the acquisition unit 41 each include temporal information such as a time stamp, the combining unit 42 may use the temporal information of each stream to combine the streams so that the difference in the temporal information of each stream is equal to or less than a predetermined value. Depending on the communication situation, when the difference between the first temporal information related to the first stream acquired from the first information source terminal and the second temporal information related to the second stream acquired from the second information source terminal is greater than a predetermined value, it may be configured such that one is not combined, one piece of information is not used, and only the other is provided by the providing unit 44. In this case, there is an advantage that a certain amount of information can be continuously transmitted even when the communication situation deteriorates. In this case, a method of selecting the information to be used and the information not to be used may use a method of compensating for the deterioration of communication information using the amount of information. That is, the first stream and the second stream each include temporal information, and the combining unit 42

[0080] Needless to say, these are just examples, and it goes without saying that various other combining modes are possible.

[0081] Also, when the acquisition unit 41 acquires a plurality of streams, the combining unit 42 may perform temporal adjustment on such a plurality of streams. For example, when one or a plurality of streams acquired by the acquisition unit 41 each include temporal information such as a time stamp, the combining unit 42 may use the temporal information of each stream to combine the streams so that the difference in the temporal information of each stream is equal to or less than a predetermined value. Depending on the communication situation, when the difference between the first temporal information related to the first stream acquired from the first information source terminal and the second temporal information related to the second stream acquired from the second information source terminal is greater than a predetermined value, it may be configured such that one is not combined, one piece of information is not used, and only the other is provided by the providing unit 44. In this case, there is an advantage that a certain amount of information can be continuously transmitted even when the communication situation deteriorates. In this case, a method of selecting the information to be used and the information not to be used may use a method of compensating for the deterioration of communication information using the amount of information. If one or a plurality of streams acquired by the acquisition unit 41 each contain temporal information such as a time stamp, the combining unit 42 may use the temporal information of each stream to combine the streams so that the difference in the temporal information of each stream is within a predetermined value. Depending on the communication situation, when the difference between the first temporal information related to the first stream obtained from the first information source terminal and the second temporal information related to the second stream obtained from the second information source terminal is greater than a predetermined value, one may not be combined, one piece of information may not be used, and only the other may be provided by the providing unit 44. In this case, even if the communication situation deteriorates, there is an advantage that a certain amount of information can be continuously transmitted. In this case, a method of selecting the information to be used and the information not to be used may use a method of compensating for the deterioration of communication information using the amount of information. In addition, depending on the communication situation, when the difference between the first time information related to the first stream obtained from the first information source terminal and the second time information related to the second stream obtained from the second information source terminal is greater than a predetermined value, one may not be combined, one piece of information may not be used, and only the other may be provided by the providing unit 44. That is, the first stream and the second stream each include time information, and the combining unit 42 When the difference between the first time information related to the first stream obtained from the first information source terminal and the second time information related to the second stream obtained from the second information source terminal is greater than a predetermined value, one may not be combined, one piece of information may not be used, and only the other may be provided by the providing unit 44. In this case, even if the communication situation deteriorates, there is an advantage that a certain amount of information can be continuously transmitted. In this case, there is an advantage that a certain amount of information can be continuously transmitted even when the communication situation deteriorates. In this case, there is an advantage that a certain amount of information can be continuously transmitted even when the communication situation deteriorates. In this case, a method of selecting the information to be used and the information not to be used may use a method of compensating for the deterioration of communication information using the amount of information.

[0082] That is, the first stream and the second stream each include temporal information, and the combining unit 42 ​adjusts the time information related to the first stream and the time information related to the second stream to a predetermined relationship so that the first stream and the second stream can be combined. Note that as the predetermined relationship, the time information related to the combined streams may be adjusted so as to be the same respectively and then combined, or may be adjusted so as to be within a predetermined time and then combined. It may be within the predetermined time because, depending on the type of information, even if a time error occurs, the user may not feel uncomfortable when viewing it.

[0083] Also, when the combining unit 42 acquires intermediate generated information, it may combine using such intermediate generated information. Also, when the combining unit 42 acquires intermediate generated information, it determines whether to combine using such intermediate generated information, and may combine only when it is determined to combine. In this case, since there is no need to combine when there is no need to combine, there is an advantage that processing can be performed efficiently . As a case where it is determined that there is no need to combine, for example, there is a case where it is determined that the background information is not displayed with the information on the back side using the background information. More specifically, for example for at least two pieces of information such as a video and an avatar, when the background information of one depth and the background information of the other depth are known respectively, the information on the front side that is not on the back side may be used without combining . Note that in this case, for such two pieces of information, the display area on the video may also be used for determination . For example, when the background information related to video 1 is A and the background information related to video 2 is B, and B has more background information than A (that is, there is a possibility that video 1 cannot be seen behind video 2 ), when video 2 does not cover video 1, video 1 is behind video 2 . Note that in this case, for such two pieces of information, the display area on the video may also be used for determination . For example, when the background information related to video 1 is A and the background information related to video 2 is B, and B has more background information than A (that is, there is a possibility that video 1 cannot be seen behind video 2 ), when video 2 does not cover video 1, video 1 is behind video 2 ), and in the case where video 2 does not cover video 1, video 1 is behind video 2 Therefore, the acquisition unit 41 acquires the image 1, and the synthesis unit 42 In this way, the display area of ​​the image that can be synthesized may also be used. The display area may be the position of the display.

[0084] In addition, when synthesizing, the synthesis unit 42 uses information from a control information management unit 43 (to be described later) to synthesize the signals. However, it may be possible to synthesize the signal without using the information of the control information management unit 43. The former is, for example, the information managed by the control information management unit 43, which is If the settings are configured to provide synthesized information to all users, synthesis may be performed. If the device is configured to provide information that is not to be combined, it is not necessary to combine the information. Provide synthesized information to the user terminal of the user and do not synthesize information to other user terminals. If the configuration provides The providing unit 44 provides the synthesized information to a user terminal that is set to provide the synthesized information. Provide information that is not to be synthesized to a user terminal that is set to provide information that is not to be synthesized. As described later, the control information may include information on each setting of each user terminal for each piece of information on the performance. Since the information may be determined by the user, the patterns of the information to be synthesized may be various. The pattern of the information to be synthesized may be set using information from the control information management unit 43. In this case, the pattern of information to be synthesized is determined according to the synthesized pattern requested by the user terminal. This has the advantage that synthesis processing can be performed efficiently.

[0085] In addition, the synthesizing unit 42 synthesizes one of the first and second streams to be synthesized. It may have a function of adjusting the frame rate of one or both streams. For example, the first situation The first stream delivered from the first information source terminal and the second stream delivered from the second information source terminal may be in different delivery environments. The difference in the delivery environment may be due to the functions of the information source terminals, or may be due to the network communication status from each information source terminal. By having the function of adjusting the frame rate in the synthesizing unit, there is an advantage that appropriate synthesized data can be generated according to the functions and network status of the information source terminals that deliver the streams. The adjusted frame rate may be higher or lower compared to a predetermined frame rate or a frame rate defined in the stream acquired by the synthesizing unit. Also, the frame rate may be lower than a predetermined value determined in advance. Also, the frame rate may be lower than one or more predetermined values associated with the amount of acquired information of the stream to be synthesized. The amount of acquired information of the stream may be the amount of information per unit or the cumulative acquired information amount that has not been synthesized. In particular, when a delay occurs in the delivery of one stream, by adjusting the frame rate of such one stream to be lower than a predetermined value and then synthesizing it, instead of a low-quality and unstable stream due to the delay, there is an advantage that a high-quality video can be provided even if the frame rate is low. In this case, the synthesizing unit 42 may synthesize one stream adjusted to a frame rate lower than a predetermined value and another stream.

[0086] Furthermore, the synthesizing unit 42 may adjust one or both of the first stream and the second stream to be synthesized. Of the data in the stream, according to the terminal, data not to be synthesized and data to be synthesized are selected, and only the data to be synthesized may be configured to be synthesized. Here, the data not to be synthesized may be, for example, data already acquired at the user terminal or unnecessary data. When the synthesizing unit 42 has such a configuration, there is an advantage in that the communication volume can be reduced and the waiting time can be shortened.

[0087] In addition to the various synthesis methods described above, various synthesis methods may be used. For example, the synthesis may use chroma key technology, and the masking used during synthesis may be a method that keys on a hue like chroma key. Also, the synthesis may use an image memory for masking like an alpha mask for calculation. Also, a method of saving transparent information in a file like a PNG file may be used. Here, the image memory serving as the mask may be saved in binary in terms of whether it is transparent, or intermediate tones from 4-bit to 256-bit may be used according to the calculation cost and image quality. Also, the synthesis may be simply an overlay of at least two objects, or an image synthesis algorithm such as hue or multiplication may be applied to one or both of such at least two objects. Note that when one or both of the objects to be synthesized are 3D videos, synthesis may be performed using depth information (depth). Note that the methods described individually are just examples, and other general synthesis methods may be used.

[0088] 3-3. Control Information Management Unit 43 The control information management unit 43 has a function of managing control information. The information managed by the information management unit may be various information, for example, viewpoint control information, production control information, etc.The control information management unit 43 may store various information as control information, as shown in FIG. .

[0089] <Viewpoint control information> The viewpoint control information is information for controlling the viewpoint of the user. This may be, for example, setting the initial value of the viewpoint of the user, setting the movement of the user's viewpoint, etc. In sports games, etc., there are spectators watching the game, and You can select one of the spectators to set the initial value of the user's viewpoint, or select one of the spectators. Select a spectator and move to that spectator's point of view, or In other words, the control information management unit 43 allows the user to move the control information to another location. From one user terminal, information relating to a viewpoint in the game space is acquired, and the information is transmitted to the one user terminal. The game device may have a function of changing the viewpoint within the game space.

[0090] The viewpoint control information may include information related to a viewpoint in the game space. The information includes information for identifying the user whose viewpoint is to be changed and information for identifying the location after the viewpoint is changed. The information for identifying the user whose viewpoint is to be changed may include an ID related to the user. The information may be information for identifying a user terminal related to the user. The updated part may be the viewpoint of a user participating in the game space, or may be the viewpoint of a user participating in the game space. It may be the viewpoint of a character appearing in the game, or it may be a location in the game space. The viewpoint of the user participating in the game space may be identified by ID or by a unique identifier that is valid within the game. The user name may be used to identify the user. Similarly, the user name may be used to identify the character that appears in the game space. - may be specified by a character name, and a location in the game space may be information that can identify a location in the game space in the game space. As characters appearing in the game space , for example, in a competition such as a sports game, spectators can be cited. Depending on the position of the spectators , the appearance of the game is different. Therefore, by enabling the movement of the viewpoint, there is an advantage that the user can obtain new interest in the game. Also, as other characters in the game who are not spectators , they may be people related to the game, such as a referee of the game, a commentator of the game , a player waiting to appear in the game, a player playing in the game field , etc., or they may be other than these, and may also be a point or area in the game space, such as the center of the game field, above the center of the game field, a ceiling camera, behind the goal, etc. In addition, in other games other than competitions such as sports games , for a player who appears or a specific point or area on the game space, an object to which the viewpoint can be moved may be set. If the viewpoint can be moved to the viewpoints of people related to the game, there is an advantage that the game can be enjoyed from the viewpoints of each person related to the game, and if it is a point or area on the game space , there is an advantage that the game can be enjoyed from a viewpoint that cannot usually be seen. That is, the control information management unit 43 may change the viewpoint in the game space related to one user terminal to the viewpoint in the game space related to any one of the predetermined user terminals , and may change the viewpoint in the game space related to the one user terminal to the viewpoint of a location on the game space other than the predetermined user terminal .

[0091] The viewpoint control information may be information that can identify people related to the game described above, or may be information that can identify a point or area on the game space.

[0092] In this case, the control information management unit 43 may have a function of storing in advance the position of the viewpoint of the audience related to the game and options for selecting positions on the game space where the viewpoint can be provided. Further, the control information management unit 43 may transmit such options to one or a plurality of user terminals. Also, in the user terminal, one of such options may be selectable, and the selected option may be provided to the control information management unit 43. Further, the control information management unit 43 may acquire, store, and set the selected option.

[0093] Also, the control information management unit 43 may set the viewpoint using the viewpoint control information and a predetermined condition. For example, when moving the viewpoint to the viewpoint of people related to a specific game, a point or area on the game space, or a predetermined point or area on the game space, it may be configured to be set only when a predetermined condition is satisfied. Such a predetermined condition may be set in advance, and the control information management unit 43 may store such a condition in advance. Also, when the acquisition unit 41 acquires the viewpoint control information from the user terminal, the control information management unit 43 may have a function of determining whether the viewpoint control information satisfies the condition or not. And, when the viewpoint control information satisfies such a condition, the control information management unit 43 may move and set the viewpoint, but when the viewpoint control information does not satisfy such a condition, it may not set the viewpoint.

[0094] The predetermined conditions may include, for example, that the user who wishes to change the viewpoint has a predetermined item that, has obtained a predetermined evaluation score, has a predetermined rank, has passed a predetermined viewing time, has provided a predetermined valuable item, and / or has provided a predetermined quantity of a predetermined valuable part and so on. When having a predetermined item is included as a condition, the user who wishes to change the viewpoint has an advantage that motivates the acquisition of such a predetermined item. Also, when having obtained a predetermined evaluation score or having a predetermined rank is included as a condition, the user who wishes to change the viewpoint has an advantage that motivates the acquisition of such an evaluation score or such a rank. Here, the evaluation score may be obtained by obtaining points in the game or the like, and the rank may be something like a status obtained by obtaining points in the game or the like. Also, the predetermined viewing time may be as described later. Also, the valuable item may be anything that is considered valuable within the game space. Being valuable means that there is a merit in holding it, and it may be an item, points within the game, or within the game space

[0095] and so on. When determining the predetermined conditions as described above, the control information management unit 43 may determine whether such conditions are met with respect to the information related to the above-mentioned predetermined conditions associated with the user terminal of the user who wishes to change the viewpoint. The information related to the predetermined conditions may be stored in the user terminal of the user who wishes to change the viewpoint, or may be stored in the control information management unit 43. In the former case, the control information management unit 43 makes an inquiry to the above-mentioned user terminal and so on. and so on. and so on. and so on. and so on. and so on. and so on. and so on.

[0096] When the control information management unit 43 defines the predetermined conditions as described above, it may determine whether such conditions are met with respect to the information related to the above-mentioned predetermined conditions associated with the user terminal of the user who wishes to change the viewpoint. The information related to the predetermined conditions may be stored in the user terminal of the user who wishes to change the viewpoint, or may be stored in the control information management unit 43. In the former case, the control information management unit 43 makes an inquiry to the above-mentioned user terminal and so on. and so on. and so on. and so on. Further, the control information management unit 43 may acquire information on which the success or failure of the condition is determined within the user terminal. Alternatively, the control information management unit 43 may inquire of the above-described user terminal, and after acquiring the information necessary to determine the success or failure of the condition obtained from the user terminal, determine the success or failure of the condition. Here, as the information necessary to determine the success or failure of the condition, there may be items, evaluation points, ranks, viewing times, valuable items, etc. that such a user has. Also, in association with a user who wishes to change the viewpoint, using the items, evaluation points, ranks, viewing times, valuable items, etc. stored in the control information management unit 43, the control information management unit 43 may determine the success or failure of such a condition. For example, the control information management unit 43 may be configured such that a change in the viewpoint to a certain location on a game space other than a predetermined user terminal is conditional on the provision of a predetermined valuable item.

[0097] <Presentation control information> Presentation control information is information for controlling presentations output by an output device in a user terminal. Since presentations include sounds, comments, giftings, etc., corresponding presentation control information includes sound control information, comment control information, gifting control information, etc. Note that in these documents, the information source provider may refer to a person who performs a presentation, and the user may refer to a person who views such a presentation being output, and the user terminal may also refer to a terminal equipped with an output device that outputs such a presentation.

[0098] Presentation control information may include grouping information. The grouping information may be information related to grouping when grouping users or information source providers. Information related to grouping indicates which group among a plurality of groups a single user or a single information provider belongs to. It may be information regarding belonging to a group. For example, when there are five users a to e and there are two groups, group A and group B, a, b, and e may be in group A and c and d may be in group B, etc. In this case, a may be associated with A , b may be associated with A, and e may be associated with A, and c may be associated with B and d may be associated with B. Here, an example of two groups A and B has been shown, but such examples include cases of games in which two teams such as soccer and baseball compete, and are not limited to these. In games that are divided into two parts, the situations of becoming players and spectators, etc. can be considered. Also, when divided into three or more groups other than these, it may be a racing game with three or more teams, etc., or in the game where the above two teams compete, it may be a referee or a spectator, etc. Note that the grouping information may be such that a plurality of users belong to one group, or one user belongs to one group. Also, although described as users above, similarly, it may be an information source provider. Also, there may be a group of users and a group of information source providers, or a group that mixes users and information source providers.

[0099] Hereinafter, for the sake of convenience of explanation, the terminal that provides the effect or the terminal that does not provide the effect will be described as a user terminal, but it may be one or more of the above-described grouped user terminals.

[0100] The effect control information is information regarding sound, comment, and / or gifting based on a specific one user terminal, and is provided or not provided to a specific other It may include information for setting whether or not. For settings not provided to a specific other user terminal In the case, a specific user has the advantage of being able to provide various performances without being known to the user related to the specific other user terminal, and there is an advantage that the range of the game is widened.

[0101] For example, the sound control information may include information for specifying a user terminal that does not provide sound, or information for specifying a user terminal that provides sound. In this case, since the user terminal that does not provide sound information or the user terminal that provides sound can be specified, there is an advantage that the range of the game is widened. More specifically, when the production control information is sound control information and the information for specifying the target user terminal of the sound information not provided is information for specifying user terminal A, when such production control information is set, the sound information may not be provided to user terminal A. Furthermore, the sound control information may include information for specifying an information source terminal that is a source of the sound. In this case, there is an advantage that the sound provided or not provided can be set according to the information source terminal. More specifically, when the production control information is sound control information and the information for specifying the target user terminal of the sound information not provided is information for specifying user terminal A, and the information source terminal of the sound not provided is information source terminal X, when such production control information is set, the sound information obtained from information source terminal Y may be provided to user terminal A, but the sound information obtained from information source terminal X may not be provided to user terminal A.

[0102] Similarly, instead of the above-mentioned sound control information, comment control information and gifting control information may also include corresponding information, and corresponding information processing may be executed.

[0103]

[0104] ​​​​​​​​​ Thus, information, sound, and / or comments related to gifting with a predetermined setting based on a specific one or more information source terminals may not be provided to other specific one or more user terminals. Also, the predetermined setting may be set according to control information based on one user terminal.

[0105] In addition, the above-described production control information may include information associated with grouping information. When the production control information includes information associated with the grouping information related to the user terminals to be provided or not provided, there is an advantage that the user terminals for which it is set whether sound, comments, and / or information related to gifting are to be provided or not can be specified in a group. Similarly, when the production control information includes information associated with the grouping information related to the information source terminal that is the information source, and the production control information is associated with such grouping information, there is an advantage that the information source terminal that is the information source can be specified in a group for sound, comments, and / or information related to gifting.

[0106] For example, when the production control information is sound control information and is information for specifying the group of user terminals A and B as the target user terminals of the sound information not to be provided, when such production control information is set, the sound information may not be provided to user terminals A and B. Also, when the production control information is sound control information and is information for specifying the group of user terminals A and B as the target user terminals of the sound information not to be provided, and when the information source terminal of such sound information not to be provided is information source terminal X, when such production control information is set, information source terminal Y ​​​​​​​​​​​​​​​​ The sound information obtained from [source] is provided to user terminals A and B, but the sound information obtained from [source] may not be provided to user terminals A and B.

[0107] Similarly, instead of the above-described sound control information, comment control information and gifting control information may also include corresponding information, and corresponding information processing may be executed for each.

[0108] In addition, the above-described production control information may be set based on the designation of the information source terminal that performs the production, or may be set based on the designation of the user terminal to which the production is provided. The former is, for example, a configuration in which information related to gifting performed by X is obtained from the information source terminal related to X that performs gifting, and is not provided to the terminal related to user Y. In this case, as described above, the information source provider X has the advantage of being able to gift without being known to user Y.

[0109] The latter is, for example, a configuration in which production control information indicating that sound based on information source provider X is not provided to the user terminal related to user Y is obtained from the user terminal related to user Y. In this case, since user Y will not hear the sound from information source provider X, it can be used, for example, when user Y wants to ignore the sound from information source provider X that user Y is not interested in.

[0110] In addition, the control information management unit 43 may store information for determining the degree of production in the case of the above-described production. For example, as the degree, if the production is sound, the volume of the sound may be stored, and if the production is a comment, the size, color, font, and typeface of the font related to the text of the comment may be stored. The display position of the mento etc. may be set and stored, and if the effect is information related to gifting, the method of the effect at the time of gifting may be stored.

[0111] Also, the control information management unit 43 may have one or more pieces of information regarding the degree of the above-described effect for each of sound, comment, and gift ting. Further, when the control information management unit 43 has a plurality of pieces of information regarding the degree of the effect for each of sound, comment, and gifting, the control information management unit 43 may determine and provide the degree of each effect when performing those effects based on a predetermined condition.

[0112] For example, the predetermined condition for determining the degree of each effect may be the viewing time, the positions of the information source provider and the user within the game space, the presence or absence of payment of a valuable item, the amount of the valuable item, or a combination thereof. These may also be referred to as "user attributes".

[0113] The viewing time may be the length of time that the information source provider related to the effect is viewing the target application. Such a viewing time may be the total time viewed since the start when starting to use the target application for the first time, or may be the total time viewed since the time of re-login when logging out of the target application. In the former case, there is an advantage that the effect is determined based on the total time for the application, and in the latter case, there is an advantage that the effect is determined based on the total time since logging in. Note that the viewing time may be the length of time that the user on the viewing side is viewing the target application in the same manner as described above.

[0114] ​​​​​​​​​Also, when the degree of each effect is based on the viewing time, it may depend on the length of time that a single user is viewing the target application session. For example, in the case where the effect is cheering, if user A has viewed for 5 hours and user B has viewed for 20 hours, the volume of the cheering for user B may be set to be greater than the volume of the cheering for user A. Also, in the case where the effect is a comment, similarly, if user A has viewed for 10 hours and user B has viewed for 100 hours, the font size of the comment for user B may be set to be greater than the font size of the comment for user A. Also, in the case where the effect is a gifting, similarly, if user A has viewed for 20 hours and user B has viewed for 200 hours, the splendor of the gifting effect for user B may be set to be more elaborate (for example, more displays are shown during user B's gifting than during user A's gifting) than the splendor of the gifting effect for user A. Also, the degree of each effect may depend on the relationship between the position of the information source provider who performs the act to be effected, such as sound, comment, gifting, etc. (in this document, such a position may also be referred to as the "information source position") within the game space, and the position of the user who views such an effect (in this document, such a position may also be referred to as the "viewing position") within the game space. Such a relationship may be the distance between the information source position and the viewing position, or may also be determined by the presence or absence of an obstacle between the information source position and the viewing position and the degree of shielding of the obstacle. For example, in the case where the effect is cheering, regarding the information source position related to the information source provider X who issues the cheering and the viewing position of the user who views the cheering, if the distance between the information source position and the viewing position is short, the volume of the cheering may be relatively large; if there is an obstacle between the information source position and the viewing position, the volume of the cheering may be relatively small; if the obstacle has a strong shielding effect, the volume of the cheering may be very small.

[0115] Also, the degree of each effect may depend on the relationship between the position of the information source provider who performs the act to be effected, such as sound, comment, gifting, etc. (in this document, such a position may also be referred to as the "information source position") within the game space, and the position of the user who views such an effect (in this document, such a position may also be referred to as the "viewing position") within the game space. Such a position may also be referred to as the "information source position" in this document. The position of the user who views such an effect (in this document, such a position may also be referred to as the "viewing position") within the game space. The relationship may depend on the distance between the information source position and the viewing position, or may also be determined by the presence or absence of an obstacle between the information source position and the viewing position and the degree of shielding of the obstacle. For example, in the case where the effect is cheering, regarding the information source position related to the information source provider X who issues the cheering and the viewing position of the user who views the cheering, if the distance between the information source position and the viewing position is short, the volume of the cheering may be relatively large; if there is an obstacle between the information source position and the viewing position, the volume of the cheering may be relatively small; if the obstacle has a strong shielding effect, the volume of the cheering may be very small.

[0116] For example, in the case where the effect is cheering, regarding the information source position related to the information source provider X who issues the cheering The distance between the position and the viewing position of user A who hears cheering is the first distance, and the information source position related to such information source provider X and the distance between the viewing position of user B who hears cheering is the second distance. When the second distance is longer than the first distance, the volume of cheering set on the user terminal related to user B may be set to be larger than the volume of cheering set on the user terminal related to user A.

[0117] Similarly, when the effect is cheering, the distance between the information source position related to the information source provider X who emits cheering and the viewing position of user A who hears cheering is the first distance, and the information source position related to the information source provider Y who emits cheering and the viewing position of user A who hears cheering is the second distance. When the second distance is longer than the first distance, on the user terminal related to user A the volume of cheering set as the cheering from information source provider X may be set to be larger than the volume of cheering set as the cheering from information source provider Y.

[0118] Also, when the above-mentioned effect is cheering, such volume may be determined according to the length of the distance between the information source position and the viewing position. For example, when the distance between the information source position and the viewing position is doubled, the degree of the volume of cheering may be calculated by an operation using a constant of 2 times.

[0119] Also, when the effect is a comment, similarly to the above, according to the relationship between the information source position and the viewing position, the font of such comment may be set on the viewing terminal related to the viewing position.

[0120] ​​​​​​​​​​​Also, similarly in gifting, according to the relationship between the information source position and the viewing position, the grandeur of the gifting production may be set in the viewing terminal related to the viewing position. Regarding the grandeur of the production, for example, a plurality of levels of grandeur of the production are predefined , and among such a plurality, one level of grandeur of the production may be set. .

[0121] Also, the degree of each production may depend on the presence or absence of payment of valuable items, the amount of payment of valuable items, etc. The valuable items may be anything that is considered valuable within the game space as described above. Being valuable means having merit in holding it, and may be items, points within the game, or within the game space, etc.

[0122] For example, by the user paying a predetermined valuable item, the degree of production regarding sound, comments, and gifting may be increased. In this case, the production control information may include information indicating a specific production such as sound, comments, and gifting, information indicating one user terminal, and information intending to pay a predetermined valuable item. Based on such information, for the specific production based on the one user terminal, by paying a predetermined value, the control information management unit 43 can obtain the intention to increase the degree of production, and may set based on such production control information.

[0123] Also, similarly, the production control information may include information indicating a specific production such as sound, comments, and gifting, information indicating one user terminal, information intending to pay a predetermined valuable item, and the amount of the valuable item to be paid. Based on such information, based on the one user terminal, Regarding the specific effect, by paying a predetermined value for the amount of the valuable item, the control information management unit 43 can obtain an increase in the degree of the effect, and may set it based on such effect control information. The control information management unit 43 can obtain an increase in the degree of the effect by paying a predetermined value for the amount of the valuable item, and may set it based on such effect control information. Based on such effect control information.

[0124] Note that the effect control information may be set for each type of effect, for example, sound, comment, gifting, or may be set for the entire effect. In the latter case, since each effect can be treated as one setting, there is an advantage that the setting is easy. For each type such as sound, comment, gifting, or may be set for the entire effect. In the latter case, since each effect can be treated as one setting, there is an advantage that the setting is easy. In the latter case, since each effect can be treated as one setting, there is an advantage that the setting is easy.

[0125] In this way, according to the user attributes associated with one user terminal, voice or comment may be provided. It may be such that voice or comment is provided to a certain extent according to the user attributes associated with one user terminal. The user attributes may include the viewing period, the user attributes may include the location in the game space, and the user attributes may include the presence or absence and / or the degree of provision of valuable items. According to the user attributes associated with one user terminal, voice or comment may be provided to a certain extent. The user attributes may include the viewing period. The user attributes may include the location in the game space. The user attributes may include the presence or absence and / or the degree of provision of valuable items. The user attributes may include the presence or absence and / or the degree of provision of valuable items.

[0126] Also, the effect control information may include content for setting the effect on one user terminal to be different from the effect on another user terminal. As an example of different effects, the target of gifting may be made larger than the size of that on another user terminal and displayed on one user terminal, or the sound generated as an effect may be made louder than the sound of that on another user terminal and generated. In this case, it may be set to be different only when one user terminal has a higher computing function and / or storage function than another user terminal. In this case, when information is not obtained from the user terminal as the effect control information. The effect control information may include content for setting the effect on one user terminal to be different from the effect on another user terminal. As an example of different effects, the target of gifting may be made larger than the size of that on another user terminal and displayed on one user terminal. Or the sound generated as an effect may be made louder than the sound of that on another user terminal and generated. In this case, it may be set to be different only when one user terminal has a higher computing function and / or storage function than another user terminal. In this case, it may be set to be different only when one user terminal has a higher computing function and / or storage function than another user terminal. In this case, when information is not obtained from the user terminal as the effect control information. Even when the computing function and / or memory function of the user terminal is higher than a predetermined value, different presentations may be set. Also, in this case, information on the computing function and / or memory function of such a user terminal may be acquired in advance by the system from the user terminal, and the system may be configured to provide information on different presentations to such a user terminal based on such information. That is, the presentation based on a piece of the composite data provided on one user terminal may be different from the presentation based on the one piece of the composite data provided on another user terminal.

[0127] 3-4. Provision Unit 44 The providing unit 44 has a function of providing what is generated by the synthesizing unit 42. Also, the providing unit 44 may provide what is generated by the synthesizing unit 42 or what is acquired by the acquiring unit 41 without being generated by the synthesizing unit 42 based on the information in the control information management unit 43.

[0128] The providing may be in a mode of displaying and / or providing on a display device directly connected to the information processing apparatus that executes the synthesizing process, or may be in a mode of displaying and / or providing on a display device not directly connected to the information processing apparatus that executes the synthesizing process. The former may be a mode of providing the synthesized information in the information processing apparatus including the synthesizing unit 42, and the latter may be a mode of providing the synthesized information in an information processing apparatus different from the information processing apparatus including the synthesizing unit 42.

[0129] 4. Flow of Information Processing According to the Embodiment 4-1. Embodiment 1 Next, the information processing flow according to Embodiment 1, which is one aspect of the present invention, will be described with reference to FIG. 13. ​​​​​In the system according to the embodiment described in the first embodiment, The first information source provider becomes the first avatar, and the second information source provider associated with the second information source terminal becomes the second avatar. The game image is sent from a third information source terminal, and the viewer sees the image synthesized. This is intended for situations in which a person is viewing something that has been broadcast.

[0130] Step 1 In one example system, a first source terminal transmits a first data type to a first avatar. A first stream including the first motion is obtained (step 001). The example system may generate a video of a specific character for the first avatar. The information may include:

[0131] Step 2 An example system includes a second avatar that receives, from a second source terminal, the first data type. A second stream including the second motion is obtained (step 002). The example system may also provide character specific information for such second avatars. The information may include video information.

[0132] Step 3 An example system receives, from a third source terminal, as a second data type, background game A third stream including video is acquired (step 003).

[0133] Step 4 In one example, the system combines a first stream, a second stream, and a third stream. At this time, the first stream is stored in the example system. A first avatar in a first stream and a motion related to the first avatar in the first stream. After synthesizing with Yon to obtain a moving first avatar, it may be synthesized with other streams That's fine. Similarly, the second stream synthesizes a second avatar stored in an example system with the motion related to the second avatar in the second stream to obtain a moving second avatar, and then it may be synthesized with other streams. Also, the order of these syntheses may be in any order.

[0134] Step 5 An example system provides the synthesized result to a user terminal related to the user (step 005).

[0135] 4-2. Embodiment 2 Next, the information processing flow according to Embodiment 2, which is an aspect of the present invention, will be described with reference to FIG. 14. In the system according to the form described in Embodiment 2, the first information source provider becomes the first avatar, and the first information source terminal acquires the motion and game video related to such a first information source provider, and the situation where the viewer views the synthesized result of these is targeted. Note that in an example system, the relationship between the game video and the location of such a first information source provider is to acquire depth information as intermediate generated information. The first information source terminal acquires, as the motion of the first avatar, the situation where the first information source provider is moving as the first avatar (step 001). The motion of the first information source provider as the movement can be acquired through a sensor provided on the body of the first information source provider, or can be acquired by analyzing an imaging device that has imaged the movement of the first information source provider.

[0136] Step 1 ​

[0137] Step 2 In one example system, a first source terminal transmits a first data type to a first avatar. A first stream including the first motion is acquired (step 002). The system provides video information regarding a specific character of the first avatar. may have information.

[0138] Step 3 An example system receives from the first source terminal a second data type including game footage. The second stream is acquired (step 003). For example, a situation may arise in which a live broadcast of a game is being done in front of the camera.

[0139] Step 4 In one example, the system receives from a first information source terminal, as a first data type, a game video as described above. The third stream includes depth information, which is the spatial relationship between the first avatar and the second avatar (step Here, the depth information is calculated by dividing the depth information of the first avatar and the depth information of the game image. , or a difference between the depth information of the game image and the depth information of the first avatar. In the latter case, information indicating which side is the front may also be added. good.

[0140] Step 5 In one example, the system combines the first stream and the second stream (step 00 5). Here, the synthesis unit 42 synthesizes the images by using the depth information included in the third stream. For example, a first source may be present behind the game footage during a live broadcast of the game. There may be cases where it moves forward. When the avatar played by the first information source provider circles behind the game video such an avatar will disappear from view in the game video, so the composition unit 42 may not synthesize the first stream and the second stream for those that become invisible as described above using the depth information included in the third stream. In this case, there is an advantage that the burden of the composition process can be reduced.

[0141] Step 6 An example system provides the synthesized result to the user terminal related to the user (step 006).

[0142] Alternatively, instead of the above configuration, in step 4, the first information source terminal may be configured not to send the content related to the video on the back side using the depth information to an example system. Specifically, when the avatar played by the first information source provider is behind the game video and invisible, etc., if it is determined that the first stream is a non-combined stream such as being hidden by the second stream using the third stream related to the depth information, the first information source terminal may be configured not to send the first stream to an example system including the composition unit 42. In this case, since the non-combined first stream is not transmitted in the first place, there is an advantage that the information on the communication path from the first information source terminal to an example system including the composition unit 42 can be reduced.

[0143] 4-3. Embodiment 3 Next, the information processing flow according to Embodiment 3 which is an aspect of the present invention will be described with reference to FIG. 15. In the system according to the form described in Embodiment 3, the first information source provider is the first ​​​​​​​​​​Play the avatar, and the first information source terminal acquires the motion related to such a first information source provider. Then, in the server, the video and the synthesized first avatar are targeted for the situation where the viewer (user ) views. Also, there are three users, A, B, and C, and the terminals used by each of them are designated as user terminal A, user terminal B, and user terminal C. Assume that A performs gifting without setting to publicly disclose it to C. Here, user terminal A may be an information source terminal. Also, assume that the viewpoint of user terminal A is changed to that of user terminal B.

[0144] Step 1 Regarding the situation where the first information source provider plays the first avatar, in an example system, the first information source terminal acquires a first stream including the motion related to the first avatar, and synthesizes it with the video (step 001).

[0145] Step 2 An example system sends the synthesized video and the motion related to the first avatar to user terminals A, B, and C respectively (step 002).

[0146] Step 3 An example system acquires gifting control information from user terminal A, which has the content that the gifting of user terminal A is not notified to user terminal C (step 003). Such gifting control information may include that it pertains to user A as non - notified gifting. Also, the gifting control information may include user terminal C as the recipient to whom the gifting is not notified, or may include a group including user terminal C. Yes. Also, the gifting that is not notified is the gifting based on the user terminal A may be specified by the gifting control information notified from the user terminal A as well.

[0147] Step 4 One example of the system uses such gifting control information to set that the gifting of the user terminal A is not notified to the user terminal B (step 004).

[0148] Step 5 One example of the system obtains a second stream including information related to the gifting indicating that gifting is to be performed from the user terminal A (step 005).

[0149] Step 6 One example of the system sends to the user terminal B the synthesized information including the information related to the gifting and the synthesized information described above, and sends to the user terminal C only the synthesized information described above in which the information related to the gifting is not synthesized (step 006).

[0150] Step 7

[0151] One example of the system obtains perspective control information including changing the perspective of the user terminal A to the perspective of the user terminal B from the user terminal A (step 007). Step 8

[0152] Step 9 ​​​​​​One system provides the video from the perspective of user terminal B to user terminal A (step 009). step 009

[0153] Also, the systems and information processing described in this application document may be implemented by software, hardware, or a combination of these. Further, part or all of such systems and information processing may be implemented as a computer program for processing procedures, and may be executed on various computers. These computer programs may be stored in a storage medium. Also, these programs may be stored in a non-transitory or temporary storage medium. What is described in this application document is not limited to what is described in this application document. Needless to say, it can be applied to various examples within the scope of various technical ideas having various technical advantages and configurations described in this application document.

[0154]

Description of Reference Numerals

[0155] 1 System 10 Communication Network 20 (20A - 20C) Server Device 30 (30A - 30C) Terminal Device 21 (31) Arithmetic Unit 22 (32) Main Memory Device 23 (33) Input / Output Interface 24 (34) Input Device 25 (35) Auxiliary Storage Device 26 (36) Output Device 41 Acquisition Unit 42 Composition Unit 43 Control Information Management Unit 44 Provision Unit 51A, 51B Information Source Terminals 52A, 52B User Terminals​​​​​​

Claims

1. One or more computers, A first means for acquiring motion information and performance information; A second means for acquiring game footage; A third means for acquiring performance control information; and a fourth means for synthesizing the motion, the effect, and the game image based on the effect control information.

2. The program according to claim 1 , wherein the effect is a sound, a comment, and / or a gifting.

3. The performance control information indicates whether or not a performance is to be provided, 3. The program according to claim 1, wherein the fourth means performs synthesis so as to include or not include the effect based on the effect control information.

4. the fourth means provides display data for displaying the synthesized image to a plurality of terminals; The program according to claim 3 , wherein the performance control information indicates which of the plurality of terminals a performance is to be provided to and which terminal a performance is not to be provided to.

5. The first means acquires the information on the performance from any one of a plurality of terminals, The program according to claim 3 , wherein the performance control information indicates which of the plurality of terminals provides a performance and which terminal does not provide a performance.

6. The first means acquires the motion information and the performance information from a terminal; The program according to claim 1 , wherein the third means acquires the performance control information from the certain terminal.

7. the fourth means provides display data for displaying the synthesized image to a plurality of terminals; The program according to claim 1 , wherein the third means acquires the performance control information from the plurality of terminals.

8. The program according to claim 1 , wherein the performance control information indicates a degree of performance.

9. the fourth means provides display data for displaying the synthesized image to a plurality of terminals; 9. The program according to claim 1, wherein the performance control information is for setting a performance in one terminal among the plurality of terminals so that the performance in another terminal is different.

10. A first means for acquiring motion information and performance information; A second means for acquiring game footage; A third means for acquiring performance control information; A system comprising a fourth means for synthesizing the motion, the performance, and the game image based on the performance control information.

11. A first means acquires motion information and performance information; A second means acquires a game image; A third means acquires performance control information; A method in which a fourth means combines the motion, the effect, and the game image based on the effect control information.

12. A program that causes the terminal to function as a means for transmitting motion information and performance information to be composited with game images, as well as performance control information to be taken into consideration when composited.

13. A terminal having a means for transmitting motion information and performance information to be composited with game images, and performance control information to be taken into consideration when composited.

14. A method including a step in which a terminal transmits motion information and performance information to be composited with game images, and performance control information to be taken into consideration when the composite is performed.

15. The device, A first means for transmitting performance control information; and a second means for receiving display data for displaying an image in which motion, effects, and game images are combined based on the effect control information.

16. A first means for transmitting performance control information; A terminal comprising: a second means for receiving display data for displaying an image in which motion, effects, and game images are combined based on the effect control information.

17. The device is A step of transmitting performance control information; and receiving display data for displaying an image in which motion, effects, and game images are combined based on the effect control information.

Images