Information processing system, control method, and server device

Abstract

To provide an information processing system, control method and server device that can improve a user interface for determining information to be provided to other users, without requiring the user to perform complicated operations.SOLUTION: An information processing system comprises one or more computer processors. The one or more computer processors are configured to: transmit, to a second terminal device of a second user, information regarding display of a video including a first object representing a first user; transmit, to the second terminal device, information regarding display of a related image selected by the first user from a group including multiple related images regarding the first object; acquire reference information regarding the first object; and include, in the group, an image including the first object and generated based on the reference information as a related image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The embodiments disclosed in this specification relate to an information processing system, a control method, and a server device.

Background Art

[0002] Conventionally, an instant messenger that realizes a function of transmitting input information such as text information or image information input by a user's operation of a terminal device to a terminal device of another user has been known. In recent instant messengers, image information using a character object such as an avatar corresponding to a user can be transmitted from the user's terminal device to the terminal device of another user as input information.

[0003] For example, Patent Document 1 describes an information processing system in which an avatar stamp is used as an example of image information that can be transmitted from a user's terminal device to a terminal device of another user. In the information processing system described in Patent Document 1, a user can use image information (such as a self-made avatar stamp) corresponding to an object such as an avatar created and registered by the user himself / herself. Thus, in the information processing system described in Patent Document 1, a user can appeal an object corresponding to himself / herself to another user by communicating with the other user using an image corresponding to the object indicating himself / herself.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The information processing system described in Patent Document 1 had the problem that, in order to enrich the range of information candidates corresponding to the user, the user had to perform the cumbersome operation of generating information themselves, which hindered the promotion of user use of the information processing system. Furthermore, in the information processing system described in Patent Document 1, when there were many candidates for information corresponding to the user, the operation of selecting the information that the user wanted to provide to other users became cumbersome, which hindered the promotion of user use of the information processing system.

[0006] The information processing system, control method, and server device disclosed herein enable an improved user interface for determining information to be provided to other users without requiring users to perform complicated operations. [Means for solving the problem]

[0007] The disclosed information processing system is an information processing system comprising one or more computer processors, wherein one or more computer processors perform the following: transmit information regarding the display of a video including a first object representing a first user to a second terminal device of a second user; transmit information regarding the display of an associated image selected by the first user from a group including multiple associated images relating to the first object to the second terminal device; and one or more computer processors perform the following: acquire reference information relating to the first object and include an image containing the first object, generated based on the reference information, as an associated image in the group.

[0008] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors include images that include a first object and are generated independently of reference information as associated images in a group.

[0009] Furthermore, in the disclosed information processing system, it is preferable that, in acquiring reference information, one or more computer processors perform the action of acquiring information related to the displayed video as reference information.

[0010] Furthermore, in the disclosed information processing system, it is preferable that the information regarding the video acquired as reference information includes motion data for displaying a related image that includes the first object in the same pose as the first object in the pose at a predetermined timing included in the displayed video.

[0011] Furthermore, in the disclosed information processing system, it is preferable that the information about the video acquired as reference information includes captured image information of the displayed video.

[0012] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors perform the action of including text information entered by a second user into the video, and that the information about the video obtained as reference information includes text information.

[0013] Furthermore, in the disclosed information processing system, it is preferable that the video information acquired as reference information includes information about a second object selected by the first user from among the second objects that can be selected to be displayed together with the first object in the video.

[0014] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors include an image representing a third object selected by a second user in a video, and that, in acquiring reference information, one or more computer processors acquire information about the image representing the third object as reference information.

[0015] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors store image information indicating owned objects held by the first user, and that in acquiring reference information, one or more computer processors acquire image information indicating owned objects as reference information.

[0016] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors store an image showing the profile of the first object, and that, in acquiring reference information, one or more computer processors acquire information from the image showing the profile as reference information.

[0017] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors perform the action of displaying a first object existing in the virtual space, and that, in acquiring reference information, one or more computer processors perform the action of acquiring predetermined image information indicating the virtual space as reference information.

[0018] Furthermore, in the disclosed information processing system, it is preferable that the rendering processing load for rendering the video onto the display screen is greater than the rendering and display load for rendering the related images onto the display screen.

[0019] Furthermore, in the disclosed information processing system, it is preferable that the video is a three-dimensional moving image in which a three-dimensional space including a first object is depicted, and the associated image is a two-dimensional image.

[0020] Furthermore, in the disclosed information processing system, it is preferable that the display size of the video on the display screen is larger than the display size of the related images on the display screen.

[0021] Furthermore, in the disclosed information processing system, it is preferable that the amount of data transmitted when sending display information for displaying a video is greater than the amount of data transmitted when sending information related to the display of associated images.

[0022] The disclosed control method is a control method for an information processing system including one or more computer processors, wherein the one or more computer processors transmit information regarding display of a video including a first object indicating a first user to a second terminal device of a second user, and transmit information regarding display of a related image selected by the first user from a group including a plurality of related images regarding the first object to the second terminal device, and the one or more computer processors obtain reference information regarding the first object, and include, as a related image, an image including the first object generated based on the reference information in the group.

[0023] The disclosed server device is a server device that communicates with a first terminal device of a first user and a second terminal device of a second user, and receives information regarding display of a video including a first object indicating the first user transmitted from the first terminal device, receives information regarding display of a related image selected by the first user from a group including a plurality of related images regarding the first object transmitted from the first terminal device, and transmits the information regarding display of the video received from the first terminal device and the information regarding display of the related image to the second terminal device, and includes, as a related image, an image including the first object generated based on the reference information regarding the first object in the group.

Advantages of the Invention

[0024] The disclosed information processing system, control method, and server device can improve the user interface for determining information provided to other users without causing the user to perform complicated operations.

Brief Description of the Drawings

[0025] [Figure 1] It is a schematic diagram for explaining an example of the outline of an information processing system. [Figure 2] It is a schematic diagram for explaining an example of the outline of an information processing system. [Figure 3] This figure shows an example of the schematic configuration of an information processing system. [Figure 4] This is a diagram showing an example of the general configuration of a terminal device. [Figure 5] This figure shows an example of the data structure of various tables. [Figure 6] This figure shows an example of a general configuration of a server device. [Figure 7] This figure shows examples of various screens displayed on a terminal device. [Figure 8] This figure shows examples of various screens displayed on a terminal device. [Figure 9] This figure shows examples of various screens displayed on a terminal device. [Figure 10] This figure shows examples of various screens displayed on a terminal device. [Figure 11] This figure shows examples of various screens displayed on a terminal device. [Figure 12] This figure shows examples of various screens displayed on a terminal device. [Figure 13] This figure shows an example of a screen displayed on a terminal device. [Figure 14] (a) is a diagram showing an example of a screen displayed on a terminal device, and (b) is a diagram showing an example of the data structure of a reference information table. [Figure 15] This figure shows an example of a screen displayed on a terminal device. [Figure 16] This figure shows an example of a screen displayed on a terminal device. [Figure 17] This figure shows an example of the operation sequence of an information processing system. [Figure 18] (a) is a diagram showing an example of the operation flow of the reference information acquisition process, and (b) is a diagram showing an example of the operation flow of the candidate information display process. [Figure 19] This figure shows an example of the operation flow of the additional generation process. [Figure 20] This figure shows examples of various screens displayed on a terminal device. [Modes for carrying out the invention]

[0026] Various embodiments of the present invention will be described below with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the invention described in the claims and its equivalents.

[0027] (Overview of the Information Processing System) Figures 1 and 2 are schematic diagrams illustrating an example of an information processing system. The information processing system comprises terminal devices and a server device, each operated by a different user. For example, a terminal device is an information processing device such as a multi-function mobile phone (a so-called "smartphone") owned by the user. For example, a server device is a computer that provides video distribution services and information exchange services to the terminal devices via a communication network. The server device may also provide other services (such as online games) to the terminal devices in addition to the video distribution and information exchange services.

[0028] A video streaming service is a service in which an image showing a user operating a specific terminal device (such as a streaming image including a character object representing the user, as described later) is streamed from that specific terminal device to one or more other users' terminal devices. An information exchange service is a service in which various types of information, such as text information and image information, are transmitted from one terminal device to another terminal device between multiple users' terminal devices running instant messengers, etc. An information exchange service may also be implemented through a so-called chat function.

[0029] In at least two of the following services—video streaming services, information exchange services, and other services—character objects representing each user may be used in conjunction with each other. In conjunction with each other means, for example, that a character object whose appearance is changed in the video streaming service is displayed with the same changed appearance in the information exchange service. Alternatively, a character object permitted for use in the information exchange service may also become available for use in the video streaming service. Furthermore, a character object that is no longer permitted for use in the information exchange service may also become unavailable for use in the video streaming service.

[0030] The terminal device stores control programs such as application programs and loads the control programs into memory in response to a user's start operation. The terminal device starts at least one of the video distribution service and the information exchange service based on user input and the execution of commands contained in the loaded control program. Once the video distribution service is started, the terminal device implements multiple functions related to the video distribution service by executing commands contained in the control program. Similarly, once the information exchange service is started, the terminal device implements multiple functions related to the information exchange service by executing commands contained in the control program. Furthermore, if a user wishes to use the various services provided by the server device, the user performs a known login operation related to the various services using a terminal device capable of communicating with the server device, thereby initiating a session between the server device and the terminal device related to the various services. This allows the user to participate in and utilize the various services.

[0031] When a video streaming service is launched, for example, the user's terminal device implements a generation function that generates character video data, including motion data, based on various input data entered by the user. For example, the input data consists of multiple image data acquired at predetermined sampling time intervals by an imaging device installed in the user's terminal device, in which case the user is captured in each image data. Character video data is an example of information regarding the display of a streamed image that includes a character object representing the user. Character video data is generated at predetermined time intervals during the video streaming service.

[0032] Furthermore, for example, a user's terminal device displays a streamed image based on character video data. Along with displaying the streamed image, the user's terminal device also transmits the generated character video data to other users' terminal devices via the server device, and therefore implements an output function that transmits the generated character video data to the server device at predetermined intervals. Furthermore, for example, a user's terminal device implements a function that displays a streamed image containing a specific object based on information indicating predetermined instructions from other users transmitted from the server device at predetermined intervals.

[0033] Hereinafter, users who use at least the function of a terminal device to transmit character video data to a server device may be referred to as "distribution users." Terminal devices operated by distribution users may be referred to as "distribution terminal devices." Furthermore, users who use at least the function of a terminal device to receive character video data from a distribution terminal device and display the distribution image of a distribution user may be referred to as "viewing users." Terminal devices operated by viewing users may be referred to as "viewing terminal devices." Note that a single terminal device may have the functions of both a distribution terminal device and a viewing terminal device.

[0034] When the video streaming service starts, as shown in Figure 1(a), the streaming terminal device displays a streaming screen that includes streaming images related to the streaming user's streaming event. One or more viewing terminal devices receive character video data from the streaming terminal device via the server device and display a streaming screen that includes streaming images related to the streaming user's streaming event based on the character video data. The streaming image of a streaming user includes a character object that represents the streaming user. This character object is displayed to move according to the motion data of the streaming user included in the character video data.

[0035] If the distribution terminal device is equipped with a microphone, it can acquire audio emitted by the distribution user and transmit the acquired audio data of the distribution user to one or more viewing terminal devices. In this case, the distribution terminal device transmits the distribution user's character video data and audio data to the server device in order to transmit the distribution user's audio data, along with the character video data, to one or more viewing terminal devices via the server device. In addition, one or more viewing terminal devices receive the distribution user's character video data and audio data transmitted to the server device by the distribution terminal device from the server device.

[0036] When audio generated by a streaming user is acquired by a streaming terminal device, the streaming user's motion data included in the character video data includes face motion data showing the streaming user's lip movements. In this case, the streaming terminal device displays a streaming image that includes a character object representing the streaming user whose lips move in approximately sync with the audio generated by the streaming user. When one or more viewing terminal devices receive the streaming user's audio data along with the streaming user's character video data, they output the streaming user's audio and display a streaming image that includes a character object representing the streaming user whose lips move in approximately sync with the outputted audio.

[0037] The distribution terminal device generates character-related images, including a character object representing the distribution user, in response to instructions from the distribution user. For example, the character-related images are so-called "stamp images" that can be transmitted by the distribution terminal device to one or more other users' terminal devices in the information exchange service. The character-related images are not limited to stamp images; any image is acceptable as long as it is usable in the service provided by the server device. Other users may be limited to viewing users, or they may be users other than the distribution user, including viewing users. Other users may be users other than the distribution user, excluding viewing users.

[0038] Character-related images may be used in services other than the information exchange services provided by the server device. For example, in a video distribution service, the distribution terminal device that generates character-related images may be configured to transmit the character-related images to one or more viewing terminal devices. Also, in an online game, the distribution terminal device that generates character-related images may be configured to transmit the character-related images to the terminal devices of one or more other users participating in the online game.

[0039] In the example shown in Figure 1(a), the distribution terminal device generates three types of character-related images. The character-related images generated by the distribution terminal device are not limited to the three types shown in Figure 1(a); two types of character-related images may be generated, or four or more types of character-related images may be generated.

[0040] Next, the distribution terminal device sets up a usable group that includes character-related images that can be used in at least one service provided by the server device. The character-related images that belong to (are included in) the usable group set up by the distribution terminal device are not limited to the two types shown in Figure 1(a). The usable group may be set up so that only one type of character-related image belongs to it, or it may be set up so that three or more types of character-related images belong to it.

[0041] Character-related images that do not belong to (are not included in) the usable group set by the distribution terminal device are not limited to the one type shown in Figure 1(a). There may be two or more types of character-related images that do not belong to the usable group. There may be no character-related images that do not belong to the usable group, in which case the usable group may be set so that each character-related image is generated belongs to the usable group each time a character-related image is generated.

[0042] The following describes an example of the use of character-related images in the information exchange service, with reference to Figure 1(b). Figure 1(b) is a diagram showing an example of an information exchange screen displayed by a distribution terminal device using the information exchange service. In the example shown in Figure 1(b), input information entered by the distribution user is displayed chronologically from top to bottom within the user display area of ​​the information exchange screen, starting with the oldest input information. For example, the user display area is the area near the right edge of the information exchange screen. The input information includes text information, still image information, moving image information, and stamp images. In addition, input information from one or more other users' terminal devices is displayed chronologically from top to bottom within the other user display area of ​​the information exchange screen, starting with the oldest input information. For example, the other user display area is the area near the left edge of the information exchange screen.

[0043] As shown in Figure 1(b), the distribution terminal device displays selectable candidate information for each of the character-related images belonging to the distribution user's available group, in response to instructions from the distribution user. The candidate information may be the same image as the character-related image, a thumbnail image of the character-related image, or a simplified image of the character-related image. The candidate information may also include text information, a mark image, etc., indicating the character-related image.

[0044] The distribution terminal device displays the character-related image corresponding to the candidate information selected by the distribution user from the displayed candidate information, below the latest input information in the user display area of ​​the information exchange screen (Figure 1(b)). Along with displaying the character-related image, the distribution terminal device transmits information regarding the display of the character-related image to one or more other users' terminal devices via the server device. In this case, one or more other users' terminal devices that receive the information regarding the display of the character-related image display display the character-related image below the latest input information in the other user display area of ​​the information exchange screen. The information regarding the display of the character-related image is identification information for identifying the character-related image. The information regarding the display of the character-related image may also be the character-related image itself.

[0045] The distribution terminal device performs an additional generation process that generates new character-related images based on reference information relating to a character object representing the distribution user. The reference information includes information relating to the displayed distribution image, response information entered by the viewing user, information relating to objects currently or previously worn by the character object, information relating to additional objects displayed within the distribution screen, information relating to additional objects selected by the viewing user, information relating to owned objects associated with the distribution user, a profile image of the distribution user or character object, and / or image information indicating the virtual space in which the character object exists. The reference information is not limited to the examples above and may include any information relating to the distribution user or a character object representing the distribution user. Details of the reference information will be described later.

[0046] The following outlines an example of the additional generation process, referring to Figure 2. Hereafter, the character object representing the distribution user may simply be referred to as a "character object."

[0047] First, the distribution terminal device acquires reference information in response to instructions from the distribution user or viewing user, or automatically. In the example shown in Figure 2 (1), at least some of the image information of the displayed distribution image is acquired as reference information. The image information acquired as reference information includes images that represent character objects. The image information acquired as reference information does not necessarily include images that represent character objects.

[0048] As reference information, motion data of the character object in the distributed image may be acquired at a predetermined timing. The predetermined timing is a timing corresponding to instructions from the distribution user or viewing user, or a timing that is automatically determined. The acquired motion data may be a single motion data or a series of consecutive motion data.

[0049] Next, the distribution terminal device generates a new character-related image based on the acquired reference information (Figure 2(2)). For example, if the reference information is at least a portion of the image information of the distributed image, the distribution terminal device uses that portion of the image information as the character-related image. The distribution terminal device may also use an image generated by superimposing a predetermined image onto that portion of the image information as the character-related image. If the reference information is motion data, the distribution terminal device generates a character-related image based on the motion data acquired as reference information and the model data of the character object.

[0050] Next, the distribution terminal device assigns (includes) the new character-related image to the usable group and displays candidate information indicating the new character-related image in a selectable format, as shown in (3) of Figure 2. Then, when a distribution user selects one of the displayed candidate pieces of information, the distribution terminal device displays the character-related image corresponding to that candidate piece of information below the latest input information in the user display area of ​​the information exchange screen.

[0051] The distribution terminal device may add (include) the new character-related image to the usable group, and at the same time delete other character-related images that are included in the usable group. The other character-related images that are deleted may be any character-related images other than the new character-related image that are included in the usable group. As a result, the character-related images that are already included in the usable group are changed to the new character-related image.

[0052] The terminal device that generates character-related images is not limited to distribution terminal devices. Any terminal device that has a character object representing a user available may generate character-related images that include that character object.

[0053] As explained above with reference to Figures 1 and 2, the information processing system, terminal device, and server device increase the types of character-related images that become available based on information about the distribution user or information about the character object representing the distribution user. In this way, the information processing system, terminal device, and server device make it possible to improve the user interface for determining information to be provided to other users without requiring the user to perform complicated operations.

[0054] Furthermore, the information processing system, terminal device, and server device change character-related images based on information about the distribution user or information about a character object representing the distribution user to at least one of the available character-related images. In this way, the information processing system, terminal device, and server device make it possible to change the information provided to other users without requiring the user to perform complicated operations, thereby keeping the user's choices interesting.

[0055] Furthermore, in conventional information processing systems, in order for a user to increase, decrease, or change candidate character-related images, the user had to interrupt or terminate the service that used the character-related images. Thus, in conventional information processing systems, when a user performed an operation or gave an instruction to increase, decrease, or change candidate character-related images, multiple communications occurred regarding the interruption or termination of the service and the resumption of the interrupted service, which sometimes led to problems such as an increased processing load on the terminal device and / or the communication load between the terminal device and the server device.

[0056] In contrast, the information processing system, terminal device, and server device disclosed in this embodiment can diversify the information provided to other users and / or change the information provided to other users to keep users from getting bored with their choices by generating new character-related images based on reference information through an additional generation process, thereby reducing the processing load on the terminal device and / or the communication load between the terminal device and the server device. Furthermore, the information processing system, terminal device, and server device disclosed in this embodiment can enable automatic changes to character-related images with a simple interface while the service continues to run.

[0057] Furthermore, in the examples shown in Figures 1 and 2, the user and other users using the information exchange service may be users with a predetermined relationship. For example, another user with a predetermined relationship with a user is a mutual follower of the user. Note that if a user follows another user and the other user also follows the user (the user and the other user are mutually following each other), the user and the other user become mutual followers. Also, another user with a predetermined relationship with a user may be another user who is mutually following a predetermined user who is a mutual follower of the user. Furthermore, another user with a predetermined relationship with a user may be another user who is friends with the user, or another user whose specific information (telephone number, email address, predetermined ID (identification), etc.) is stored on the user's terminal device. Also, if the user is a broadcasting user, another user with a predetermined relationship with the user may be a viewing user who has viewed the broadcasted image distributed by the user. In this case, when a user and other users with a predetermined relationship perform an information exchange service as shown in Figures 1 and 2, a predetermined information exchange group composed of the user and other users may be temporarily created by the user's or other user's terminal device or server device.

[0058] The above-described explanations of Figures 1 and 2 are merely for the purpose of deepening understanding of the present invention. The present invention is specifically implemented in the embodiments described below and may be implemented by various modifications without substantially exceeding the principles of the present invention. All such modifications are included within the scope of the present invention and the disclosure of this specification.

[0059] (Information Processing System 1) Figure 3 shows an example of the schematic configuration of information processing system 1. Information processing system 1 comprises terminal devices 2 operated by multiple users and a server device 3. Terminal devices 2 and server device 3 are interconnected via a communication network such as a base station 4, a mobile communication network 5, a gateway 6, and the Internet 7. Communication between terminal devices 2 and server device 3 is conducted based on a communication protocol such as the Hypertext Transfer Protocol (HTTP). Alternatively, a connection may be established between terminal devices 2 and server device 3 using HTTP communication first, and then communication may be conducted based on WebSocket, which is a bidirectional communication method with lower costs (lower communication load and processing load) than HTTP communication. Note that the communication method between terminal devices 2 and server device 3 is not limited to the above method, and any communication technology can be used between terminal devices 2 and server device 3 as long as it can realize this embodiment.

[0060] The communication network may be a Local Area Network (LAN), a Wide Area Network (WAN), or any combination of LAN and WAN. The communication network may include a Public Switched Telephone Network (PSTN), an Integrated Services Digital Network (ISDN), and / or subnetworks. The communication network may also be a wired communication network such as an Ethernet network cable or a Universal Serial Bus (USB) cable. The communication network may also be a wireless communication network such as a cellular network including EDGE, 3G, 4G, and / or 5G wireless cellular systems. The wireless communication network may include Wi-Fi®, wireless LAN, Bluetooth®, or other known wireless forms of communication.

[0061] Terminal device 2 is an information processing device such as a smartphone. Terminal device 2 may also be a mobile phone, notebook PC, tablet device, tablet PC, head-mounted display (HMD), e-book reader, or wearable computer. Terminal device 2 may also be a portable game console or game console. Terminal device 2 may be any information processing device that is capable of displaying and outputting character objects representing the user and other users, outputting the voice of other users, and processing information exchange with other users.

[0062] In the example shown in Figure 3, one server device 3 is depicted as a component of the information processing system 1, but the server device 3 may be a collection of multiple physically separate server devices 3. In this case, each of the multiple server devices 3 may have the same function, or they may have the functions of a single server device 3 distributed amongst themselves. For example, a server device 3a for realizing a video distribution service and a server device 3b for realizing an information exchange service may be components of the information processing system 1.

[0063] (Terminal device 2) Figure 4 shows an example of the schematic configuration of terminal device 2. Terminal device 2 connects to server device 3 via a communication network such as base station 4, mobile communication network 5, gateway 6, and the Internet 7, and communicates with server device 3. Terminal device 2 generates character video data including motion data according to various data (image data, etc.) input by the user, and transmits the generated character video data and / or the user's voice data to server device 3. Terminal device 2 also receives character video data and / or voice data of other users transmitted from server device 3, displays the other user's character object and / or outputs the other user's voice based on the received character video data and / or voice data. Terminal device 2 also displays input information such as text information, still image information, moving image information, and stamp images input by the user, and input information input by other users transmitted from other users' terminal devices 2, in chronological order. To realize these functions, the terminal device 2 includes a terminal communication interface 21, a terminal storage device 22, a display device 23, an input device 24, an imaging device 25, a microphone 26, and a terminal processing device 27.

[0064] The terminal communication interface 21 is implemented as hardware, firmware, or communication software such as a TCP / IP (Transmission Control Protocol / Internet Protocol) driver or a PPP (Point-to-Point Protocol) driver, or a combination thereof. Terminal device 2 can transmit data to and receive data from other devices such as server device 3 via the terminal communication interface 21. The terminal communication interface 21 may also include a network controller such as an Ethernet PRO network interface card for interfacing with the communication network. The network controller may also comply with other direct wireless communication standards such as Bluetooth®, Near Field Communication (NFC), or infrared.

[0065] The terminal storage device 22 is, for example, a semiconductor memory device such as ROM (Read Only Memory) or RAM (Random Access Memory). The terminal storage device 22 stores operating system programs, driver programs, control programs, and data used in processing by the terminal processing device 27. The driver programs stored in the terminal storage device 22 include output device driver programs that control the display device 23 and input device driver programs that control the input device 24. The control programs stored in the terminal storage device 22 are, for example, application programs for realizing various functions related to various services, including video distribution services and information exchange services. The control programs may also be transmitted from the server device 3 or other devices.

[0066] The terminal storage device 22 stores, as data, identification information (e.g., user ID) to uniquely identify the user operating the terminal device 2. The terminal storage device 22 also stores, as data, background data and model data. Furthermore, the terminal storage device 22 stores, as data, a user table T1, an object table T2, a group table T3, and a reference information table T4. The terminal storage device 22 may also temporarily store data related to predetermined processing.

[0067] Background data is asset data used to construct a virtual space in a streamed image where the user's character object resides. Background data includes data for rendering the background of the virtual space, data for rendering various objects included in the streamed image, and data for rendering other background objects displayed in the streamed image. Background data may also include object position information indicating the location of various background objects in the virtual space.

[0068] Furthermore, the terminal storage device 22 stores multiple types of motion data for character-related images as data. The motion data for character-related images includes body motion data, which is a digital representation of the position and orientation of each part (head, arms, hands (including fingers), chest, torso, legs, or other parts) that causes the character object to perform a predetermined action. The motion data for character-related images may also include face motion data that causes the character object to make a predetermined facial expression.

[0069] Motion data for character-related images may be a collection of multiple consecutive motion data to represent a character object moving for a predetermined time (e.g., 3 seconds). Motion data for character-related images may also be a single motion data. If the motion data for character-related images is a single motion data, the character object included in the character-related image generated by that motion data will not perform any movement.

[0070] Furthermore, the terminal storage device 22 may store, as data, object data for character-related images, such as various background objects, gift objects, and object data for representing text, along with motion data for character-related images. By using such object data for character-related images related to the design of character-related images, it becomes possible to diversify the expression of character-related images. In this case, the terminal storage device 22 may also store object IDs associated with the object data for character-related images.

[0071] The display device 23 is a liquid crystal display. Alternatively, the display device 23 may be an organic EL (Electro-Luminescence) display or the like. The display device 23 displays moving images and / or still images corresponding to moving image data supplied from the terminal processing device 27 on its display screen. Furthermore, the display device 23 does not have to be a component of the terminal device 2; in this case, the display device 23 may be a display for an HMD capable of communicating with the server device 3, a projection mapping device capable of communicating with the terminal device 2 via wired or wireless connection, a retinal projection type projection device, or the like. The terminal device 2 may also be equipped with a display controller such as a graphics card or graphics adapter for interfacing with the display device 23.

[0072] The input device 24 is a pointing device such as a touch panel. If the input device 24 is a touch panel, it can detect various touch operations by the user, such as taps, double taps, and drags. The touch panel may be equipped with a capacitive proximity sensor and configured to detect non-contact operations by the user. The input device 24 may also be an input key or the like. The user can use the input device 24 to input characters, numbers, symbols, or positions on the display screen of the display device 23. When the input device 24 is operated by the user, it generates a signal corresponding to that operation. The input device 24 then supplies the generated signal to the terminal processing device 27 as a user instruction. The terminal device 2 may be equipped with an I / O interface for interfacing with the input device 24. The I / O interface may be connected to various other peripheral devices of the input device 24.

[0073] The imaging device 25 is a camera having an imaging optical system, an image sensor, and an image processing unit. The imaging optical system is, for example, an optical lens, which forms an image of the light beam from the subject onto the imaging surface of the image sensor. The image sensor is a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), etc., and outputs an image of the subject image formed on the imaging surface. The image processing unit creates moving image data in a predetermined file format from the images continuously generated by the image sensor at predetermined intervals and outputs it as imaging data. Alternatively, the image processing unit creates still image data in a predetermined file format from the images generated by the image sensor and outputs it as imaging data.

[0074] Microphone 26 is a sound collection device configured to acquire voices emitted by the user and convert them into audio data. Microphone 26 is configured to acquire the user's voice input, converts the user's voice input acquired by Microphone 26 into audio data, and outputs the audio data to the terminal processing device 27.

[0075] The terminal processing unit 27 is a processor that loads operating system programs, driver programs, and control programs stored in the terminal storage device 22 into memory and executes instructions contained in the loaded programs. The terminal processing unit 27 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), GPU (Graphics Processing Unit), or a combination of various electronic circuits. The terminal processing unit 27 may also be implemented by integrated circuits such as ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), and MCUs (Micro Controller Units). In Figure 4, the terminal processing unit 27 is shown as a single component, but the terminal processing unit 27 may be a collection of multiple physically separate processors. For example, multiple processors that operate cooperatively in parallel to execute instructions may be implemented.

[0076] Furthermore, the functions realized by the components disclosed herein (for example, the terminal processing unit 27 and / or the server processing unit 33 (described later)) may be implemented in a “Circuitry” or “Processing Circuitry” which includes general-purpose processors, application-specific processors, integrated circuits, ASICs, CPUs, conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be a “Circuitry” or “Processing Circuitry” which includes transistors and other circuits. A processor may also be a “Programmed Processor” which executes a program stored in memory.

[0077] In this specification, “Circuitry,” “Unit,” and “Means” refer to hardware programmed to perform or executing the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0078] If the hardware is a processor of the type “Circuitry”, then “Circuitry”, “Unit”, or “Means” is a combination of the hardware and the software used to constitute the hardware and / or processor.

[0079] "Processing Circuitry" may be used to control any computer-based and cloud-based control process. In the operation sequences, operation flows, and flowcharts described later, descriptions or blocks may be understood as indicating a module, segment, or portion of code containing one or more executable instructions for implementing a particular logical function or step of the control process. Alternative implementations of these are also included within the scope of the exemplary embodiments of the present advancements. For example, the particular logical functions or steps of the control process may be executed in an order different from the disclosed order, for example, substantially simultaneously, or in the reverse order of the disclosed order.

[0080] "Processing Circuitry" may include a CPU that executes one or more of the control processes disclosed herein. Process data and instructions may be stored in memory. Process data and instructions may be stored on a storage medium disk such as a hard disk drive (HDD) or portable storage medium. Process data and instructions may be stored remotely.

[0081] The claimed advancements are not limited by the form of the computer-readable medium on which the instructions for the process of the present invention are stored. For example, the instructions may be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other non-volatile computer-readable medium in an information processing device (such as another server device or computer) with which "Processing Circuitry" communicates. The instructions may also be stored in network-based storage, cloud-based storage, or other mobile-accessible storage, as long as they are executable by "Processing Circuitry".

[0082] The claimed advancements may be provided as utility applications, background daemons, operating system components, or combinations thereof. They may also be executed in conjunction with a CPU and an operating system, including Microsoft Windows®, UNIX®, Solaris, LINUX®, Apple MAC-OS, Apple iOS, and other systems known to those skilled in the art.

[0083] The functions and features disclosed herein may be performed by various distributed components of the information processing system 1. For example, one or more processors may perform these system functions, and each processor may be distributed among multiple components that can communicate within a network. Distributed components may include various human interfaces, communication devices (such as display monitors, smartphones, tablets, and personal digital assistants (PDAs)), and one or more client and server devices that share processing. Input to the information processing system 1 may be received as direct input from a user, or as remote input in real time or through a batch process. Furthermore, some implementations may be performed on modules or hardware that are not identical to those disclosed herein, in which case other implementations are also included in the claims.

[0084] The terminal processing unit 27 functions as a generation unit 271, a transmission unit 272, a reception unit 273, a display processing unit 274, and an audio output unit 275 by executing various instructions included in the control program. The functions of the generation unit 271, the transmission unit 272, the reception unit 273, the display processing unit 274, and the audio output unit 275 will be described later.

[0085] (Various Tables) Figure 5 shows an example of the data structure of user table T1, object table T2, and group table T3 stored in terminal storage device 22. Note that at least one of user table T1, object table T2, and group table T3 may be stored in server storage device 32 of server device 3. In this case, terminal device 2 may receive the latest table stored in server storage device 32 at a predetermined timing and store it in terminal storage device 22.

[0086] (User Table T1) Figure 5(a) shows an example of the data structure of user table T1, which manages users who use the services provided by server device 3. The users managed in user table T1 are delivery users, but other users may also be managed. In user table T1, for each user, the user ID, user name, user character object, user owned objects, and used objects are stored in relation to each other. In user table T1, the user's owned coins (information indicating the total amount of virtual currency "coins" owned by the user) and / or user points may be stored in relation to the user's user ID. The user ID is an example of identification data to uniquely identify each user. The name is an example of data indicating the name of each user.

[0087] For example, a character object is a model ID used to identify model data for generating animations of the character object. The model data is stored in the terminal storage device 22 in association with the model ID. The model data may be 3D model data for generating 3D animations, or 2D model data for generating 2D model data. The model data may include, for example, rig data (so-called "skeleton data") that shows the skeleton of the character object's face and parts other than the face, and surface data that shows the shape and texture of the character object's surface. The model data may include multiple different model data sets. Each of the multiple model data sets may have different rig data, or they may have the same rig data. Each of the multiple model data sets may have different surface data, or they may have the same surface data. In addition, multiple object IDs associated with each of the multiple sub-objects that make up the character object may be associated with the model ID. In this case, for each object ID of the multiple sub-objects, 3D model data for generating 3D animations may be stored in the terminal storage device 22 and the server storage device 32. In this case, during the character object display process, the object IDs of multiple sub-objects associated with the model ID may be extracted, the 3D model data corresponding to each object ID (for each sub-object) may be identified, and the character object may be displayed based on the identified 3D model data (for each sub-object). Alternatively, image information corresponding to each 3D model data may be stored.

[0088] A possessed object is an object ID that indicates the object each user possesses. Possessed objects include attachable objects that can be associated with specific parts of a character object, and wallpaper objects that are placed behind the user's character object in the distributed image. Details of attachable objects and wallpaper objects will be described later. Possessed objects are not limited to the examples above; any object that can be displayed on the distributed screen is acceptable. Possessed objects may also be objects that a user comes to possess as a result of an event in which the user participated, or as a result of a selection game (e.g., a lottery game) played by the user. Furthermore, possessed objects may also be objects that a user comes to possess by spending virtual currency "coins" in the purchase function of Information Processing System 1.

[0089] The object used is the object ID that indicates the object being used in the streamed image displayed on the user's terminal device 2. For example, if a user inputs a change command during a video streaming service, the user's owned attachment object is attached to the user's character object included in the streamed image being displayed. In this case, the object ID of the attachment object attached to the user's character object is stored in user table T1 as the object used, associated with the user ID. Also, for example, if a user inputs a change command during a video streaming service, a wallpaper object owned by the user is placed behind the user's character object in the streamed image being displayed. In this case, the object ID of the wallpaper object placed in the streamed image is stored in user table T1 as the object used, associated with the user ID.

[0090] (Object Table T2) Figure 5(b) shows an example of the data structure of object table T2, which manages objects selected as additional objects. Additional objects are, for example, gift objects sent by viewing users to broadcasting users. In object table T2, for each additional object, the object ID, name, image information, rarity, and location are stored in relation to each other.

[0091] The object ID is an example of identification data used to uniquely identify each additional object. The image information consists of one or more still images corresponding to each additional object. The image information may also consist of one or more moving images corresponding to each additional object. The rarity is information indicating the rarity of each additional object. The placement location is information indicating where the image information of each additional object is displayed. If the display position and display range of a character object are defined, information indicating the relative position to that character object may be stored as the placement location. Instead of image information, model data of the additional object may be stored. The model data of the additional object includes rig data and surface data that show the 3D shape, and an image of the additional object viewed from a predetermined viewpoint is drawn based on this model data.

[0092] Additional objects are classified into multiple types (categories). For example, additional objects include effect objects used as effect gifts, normal objects used as normal gifts, wearable objects used as wearable gifts, and message objects used as message gifts. Additional objects may also include wallpaper objects placed behind character objects in the distributed image. Information indicating the type (category) of an additional object may be stored in object table T2 in association with the object ID of the additional object. Below, the image information and placement location will be described according to the type of additional object.

[0093] Effect objects are objects that influence the overall impression of a streamed image. For example, an effect object might be an object that mimics confetti. If the effect object is an object that mimics confetti, then multiple images representing different pieces of paper are stored as image information.

[0094] Information indicating the "space" where the effect object will be placed is stored. For example, an object resembling confetti will be displayed across the entire streamed image. The effect object may be displayed across the entire streamed screen, including the streamer's streamed image, according to the viewer's instructions. The effect object may be displayed overlapping with character objects included in the streamed image. Unlike the attachment objects described later, the effect object is displayed without being associated with a specific part of the character. In this way, the terminal device 2 of the information processing system 1 can change the overall impression of the streamer's streamed image compared to before the effect object was displayed by displaying the effect object on the streamer's streamed image according to the viewer's instructions.

[0095] A typical object is, for example, an object that imitates a stuffed animal, a bouquet of flowers, an accessory, or an item suitable for a gift or present. Information indicating the "space" where the typical object will be placed is stored. For example, a typical object may be associated with information about a predetermined movement route, and the typical object will be displayed moving along the predetermined movement route within the distributed image. The information indicating the "space" where the "typical object" will be placed may be classified into multiple types. These multiple types of "space" may include, for example, "center space" (the central area of ​​the image when the distributed image is divided into three horizontally), "left space" (the left-side area of ​​the image when the distributed image is divided into three horizontally), and "right space" (the right-side area of ​​the image when the distributed image is divided into three horizontally). Alternatively, these multiple types of "space" may also include "center space" (the central area of ​​the image when the distributed image is divided into three vertically), "upper space" (the upper area of ​​the image when the distributed image is divided into three vertically), and "lower space" (the lower area of ​​the image when the distributed image is divided into three vertically). In this case, the movement route associated with the typical object will be set within the type of "space" associated with the placement location of that typical object. For example, if a normal object stores information indicating "the space on the right," then information regarding a movement route within the right-hand region of the image when the distributed image is divided into three sections horizontally will be associated with that normal object. Furthermore, normal objects may be displayed moving according to predetermined movement rules from a position set for each normal object within the distributed image, or from a position automatically determined randomly. These predetermined movement rules could be, for example, rules set to make the normal object move as if it were in free fall. Note that normal objects may be displayed overlapping with character objects. In this case, unlike the wearable objects described later, normal objects are displayed without being associated with any specific part of the character object. Also, normal objects may be displayed to bounce off character objects when they come into contact with them.

[0096] If a regular object is displayed overlapping with a character object included in the streamed image, for example, the regular object may be displayed overlapping with the "part of the character object other than the 'head' including the 'face'." In this case, the regular object will not be displayed overlapping with the "head" of the character object. Also, the regular object may be displayed overlapping with the "part of the character object other than the 'upper body' including the 'face'." In this case, the regular object will not be displayed overlapping with the "upper body" of the character object.

[0097] An attached object is an object that appears in the distributed image associated with a specific part (attachment location) of a character object. For example, specific locations on the character object's head include the front left, front right, back left, back right, center front, center back, left eye, right eye, left ear, right ear, and the entire hair. Alternatively, specific locations may include the thumb, ring finger, wrist, elbow, shoulder, upper arm, entire hand, entire arm, torso, upper body, lower body, etc.

[0098] An attachment object associated with a specific part of a character object will be displayed in the distributed image so as to be in contact with that specific part of the character object. Alternatively, an attachment object associated with a specific part of a character object may be displayed in the distributed image so as to cover part or all of that specific part of the character object. The specific part may be identified by 3D position information indicating its location in 3D coordinate space, or it may be associated with position information in 3D coordinate space.

[0099] The image information for attachable objects includes, for example, images of accessories (headbands, necklaces, or earrings, etc.), clothing (T-shirts, dresses, etc.), costumes, and other items that can be attached to character objects.

[0100] The system stores information indicating the attachment location of an attachment object, showing which part of the character object the attachment object is associated with. For example, if the attachment object is a "headband," the system stores information indicating the "head" of the character object as the attachment location. Similarly, if the attachment object is a "T-shirt," the system stores information indicating the "torso" of the character object as the attachment location.

[0101] The system may store information indicating multiple attachment points in 3D coordinate space as the placement location for an attachment object. For example, if the attachment object is a "headband," the system may store information indicating two locations on the character object: the "rear left side of the head" and the "rear right side of the head." This ensures that the attachment object, which resembles a "headband," is displayed as being attached to both the "rear left side of the head" and the "rear right side of the head" of the character object.

[0102] When multiple types of attachments that share the same attachment location are attached to a character object, these attachments are attached to the character object at different times. As a result, multiple types of attachments that share the same attachment location cannot be attached to the character object simultaneously. For example, if the attachment location for both an attachment representing a "headband" and an attachment representing a "hat" is stored as "head," then the attachments representing the "headband" and the "hat" will not be displayed on the character object's head at the same time.

[0103] In object table T2, the display time corresponding to the type of added object may be stored in association with the object ID of that added object. For example, the display time of an attached object may be longer than the display time of an effect object and the display time of a normal object. For example, if "60 seconds" is stored as the display time of an attached object, "5 seconds" may be stored as the display time of an effect object and "10 seconds" as the display time of a normal object.

[0104] (Group Table T3) Figure 5(c) shows an example of the data structure of a group table T3 that manages the information exchange groups to which a user belongs. A group table T3 is provided individually for each user, and the group table T3 shown in Figure 5(c) is the group table T3 of the user operating the terminal device 2 that stores the group table T3.

[0105] In group table T3, for each information exchange group, the group ID, name, and members of the information exchange group are stored in a related manner. The group ID is an example of identification data used to uniquely identify each information exchange group. The name is an example of data indicating the name of each information exchange group. The members are the user IDs of each user belonging to the information exchange group.

[0106] Furthermore, if group table T3 is stored in server storage device 32, the user ID of the corresponding user (the user operating terminal device 2 that stores group table T3) is associated with and stored for each group table T3.

[0107] Returning to Figure 4, the functions of the generation unit 271, transmission unit 272, reception unit 273, display processing unit 274, and audio output unit 275 will be explained.

[0108] (Generation part 271) The generation unit 271 functions when the terminal device 2 is used as a distribution terminal device operated by the distribution user. The generation unit 271 acquires imaging data continuously output by the imaging device 25 from the imaging device 25. For example, the imaging device 25 of the terminal device 2 (distribution terminal device) held by the distribution user is installed on the terminal device 2 so that the face of the distribution user, who is looking at the display screen of the display device 23, is in the direction of the imaging. The imaging device 25 continuously images the face of the distribution user in the direction of the imaging, acquires imaging data of the distribution user's face, and outputs it to the generation unit 271 of the terminal processing device 27. The imaging device 25 may also image parts other than the face of the distribution user, such as the head, arms, hands (including fingers), chest, torso, legs, or other parts, and acquire imaging data. The imaging device 25 may also be a 3D camera capable of detecting the depth of a person's face.

[0109] The generation unit 271 generates face motion data, which is a digital representation of the facial movements of the streaming user, as time progresses, based on the continuously acquired imaging data. The face motion data may be generated at predetermined sampling time intervals. In this way, the face motion data generated by the generation unit 271 can digitally represent the facial movements (changes in facial expressions) of the streaming user in a time series.

[0110] The generation unit 271 may generate body motion data, which is a digital representation of the position and orientation of each part of the distribution user (head, arms, hands (including fingers), chest, torso, legs, or other parts), together with or separately from face motion data, based on continuously acquired imaging data.

[0111] Body motion data may be generated based on detection information from a known motion sensor attached to the distribution user. In this case, the terminal communication I / F 21 of the terminal device 2 (distribution terminal device) has a predetermined communication circuit for acquiring detection information from the motion sensor attached to the distribution user via wireless communication. The generation unit 271 then generates body motion data based on the detection information acquired by the terminal communication I / F 21. The body motion data may be generated at predetermined sampling time intervals. In this way, the body motion data generated by the generation unit 271 can digitally represent the movements of the distribution user's body in a time series.

[0112] The generation of body motion data based on detection information from motion sensors attached to the streaming user may be performed, for example, in a shooting studio. In this case, the shooting studio may be equipped with a base station, a tracking sensor, and a display. The base station is, for example, a multi-axis laser emitter. The motion sensor attached to the streaming user may be, for example, a Vive Tracker provided by HTC CORPORATION, and the base station may also be, for example, a base station provided by HTC CORPORATION.

[0113] Furthermore, a supporter computer may be installed in a separate room adjacent to the shooting studio. The display in the shooting studio may be configured to display information received from the supporter computer. Server device 3 may be installed in the same room as the supporter computer. The room where the supporter computer is installed and the shooting studio may be separated by a glass window. In this case, the operator of the supporter computer can see the distribution user. The supporter computer may be configured to change the settings of various devices provided in the shooting studio in response to the supporter's operation. For example, the supporter computer can change the scanning interval by the base station, the settings of the tracking sensor, and various settings of other devices. The operator may input a message into the supporter computer, and the input message may be displayed on the display in the shooting studio.

[0114] The generation unit 271 generates character video data including face motion data and / or body motion data as needed, and outputs the generated character video data to the display processing unit 274. Hereinafter, face motion data and body motion data may be collectively referred to as motion data. The generation unit 271 also outputs the generated character video data and the user ID of the distribution user stored in the terminal storage device 22 to the transmission unit 272. If the generation unit 271 acquires audio data of the distribution user output from the microphone 26, it outputs the generated character video data, the acquired audio data, and the user ID of the distribution user stored in the terminal storage device 22 to the transmission unit 272.

[0115] (Transmitter 272) The transmitting unit 272 of the distribution user's terminal device 2 (distribution terminal device) transmits information regarding the distribution start instruction to the server device 3 via the terminal communication I / F 21, along with the distribution user's user ID stored in the terminal storage device 22, in response to the distribution user's instruction to start distribution. Hereinafter, the distribution user's user ID may be referred to as the distribution user ID.

[0116] Furthermore, the transmission unit 272 of the distribution user's terminal device 2 (distribution terminal device) transmits the distribution user's character video data output from the generation unit 271 to the server device 3 via the terminal communication interface 21, along with the distribution user ID stored in the terminal storage device 22, as information for displaying the distribution image of the distribution user in the distribution event. If the distribution user's voice data is output from the generation unit 271 along with the distribution user's character video data, the transmission unit 272 transmits the distribution user's character video data and voice data, along with the distribution user ID, to the server device 3 via the terminal communication interface 21.

[0117] Furthermore, the transmission unit 272 of the distribution user's terminal device 2 (distribution terminal device) transmits information regarding the distribution termination instruction to the server device 3 via the terminal communication interface 21, along with the distribution user ID stored in the terminal storage device 22, in response to the distribution user's instruction to terminate the distribution.

[0118] Furthermore, the transmission unit 272 of the viewing user's terminal device 2 (viewing terminal device) transmits information regarding the instruction to start viewing a distributed image, along with information regarding the distributed image and the user ID of the viewing user stored in the terminal storage device 22, to the server device 3 via the terminal communication interface 21 in response to the viewing user's instruction to start viewing the distributed image. The information regarding the distributed image is information for identifying the distributed image that the distribution user has instructed to start viewing, and is, for example, the identification information of the distributed image (distributed image ID) or the distribution user ID of the distribution user distributing the distributed image. Hereinafter, the user ID of the viewing user may be referred to as the viewing user ID.

[0119] Furthermore, the transmission unit 272 transmits information indicating various instructions entered by the user through the operation of the input device 24 to the server device 3 via the terminal communication interface 21, along with the user ID stored in the terminal storage device 22. These user instructions include, for example, additional instructions, modification instructions, and generation instructions. Information indicating additional instructions is transmitted to the server device 3 along with the object ID of the additional object to be included in the distribution image of the distribution user, the distribution user ID, and the viewing user ID stored in the terminal storage device 22 of the viewing user's terminal device 2 (viewing terminal device).

[0120] When a distribution user selects a partial object and gives instructions for creating a character object composed of the selected partial object, the model ID for identifying the model data of the character object and the object IDs of the multiple partial objects that make up the character object may be transmitted to the server device 3 via the terminal communication I / F 21, along with the user ID, etc., stored in the terminal storage device 22 of the distribution user's terminal device 2 (distribution terminal device). In this case, the server device 3 may store the user ID, the model ID, and the object IDs of the multiple partial objects that make up the character object in the server storage device 32, associating them with each other.

[0121] Furthermore, the viewing user's terminal device 2 (viewing terminal device), which receives the model ID for identifying the distribution user's character object model data, the object IDs of multiple sub-objects constituting the user's character object, and the user's user ID via the server device 3, may store the received user ID, model ID, and object IDs of the multiple sub-objects constituting the character object in the terminal storage device 22, associating them with each other. This allows the viewing user's terminal device 2 (viewing terminal device) to extract the object IDs of the multiple sub-objects associated with the received user ID (or model ID) simply by having the distribution user's user ID (or model ID) sent to the viewing user's terminal device 2 (viewing terminal device) via the server device 3. The viewing user's terminal device 2 (viewing terminal device) can then display the distribution image, including the distribution user's character object, based on the 3D model data (of each sub-object) corresponding to each extracted object ID.

[0122] The transmission unit 272 of the user's terminal device 2 transmits information regarding the new start instruction via the terminal communication interface 21 to the server device 3 in response to the user's instruction to start a new information exchange. Along with the information regarding the new start instruction, the user IDs of other users specified by the user and the user IDs stored in the terminal storage device 22 of terminal device 2 are transmitted to the server device 3. The other users specified by the user are one or more users designated by the user as partners for information exchange. Hereafter, the group to which the user and one or more other users belong may be referred to as an information exchange group.

[0123] The transmission unit 272 of the terminal device 2 of a user belonging to an information exchange group transmits information regarding the information exchange start instruction to the server device 3 via the terminal communication I / F 21 in response to the user's instruction to start information exchange within the information exchange group (e.g., text chat). Along with the information regarding the information exchange start instruction, the group ID of the information exchange group to which the user belongs, stored in the terminal storage device 22 of the terminal device 2, is transmitted to the server device 3 as destination information. If the server device 3 does not store the user IDs of each user belonging to the information exchange group, destination information including the group ID indicating the information exchange group or the user IDs of other users belonging to the information exchange group may be transmitted to the server device 3 along with the information regarding the information exchange start instruction.

[0124] When user input information is received by the user while the information exchange screen is being displayed, the transmission unit 272 of the terminal device 2 belonging to the information exchange group transmits the input information to the server device 3 via the terminal communication interface 21. Along with the input information, the user ID and destination information stored in the terminal storage device 22 of the terminal device 2 belonging to the information exchange group are transmitted to the server device 3. The destination information is either the group ID indicating the information exchange group, or the user ID of another user belonging to the information exchange group.

[0125] (Receiving unit 273) The receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives the character video data and user ID of the streaming user transmitted from the server device 3 via the terminal communication interface 21. The receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives the character video data, audio data, and user ID of the streaming user transmitted from the server device 3 via the terminal communication interface 21. The receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives the audio data and user ID of the streaming user transmitted from the server device 3 via the terminal communication interface 21. In addition, the receiving unit 273 receives information indicating various instructions from other users transmitted from the server device 3 to the streaming user's terminal device 2 (streaming terminal device) and the receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) via the terminal communication interface 21. The receiving unit 273 passes the received information to the terminal processing device 27.

[0126] The character video data and user ID of the streaming user transmitted from server device 3 were transmitted to server device 3 by the transmission unit 272 of the streaming user's terminal device 2 (streaming terminal device). Furthermore, the character video data, audio data, and user ID of the streaming user transmitted from server device 3 were transmitted to server device 3 by the transmission unit 272 of the streaming user's terminal device 2 (streaming terminal device). Additionally, the audio data and user ID of the streaming user transmitted from server device 3 were transmitted to server device 3 by the transmission unit 272 of the streaming user's terminal device 2 (streaming terminal device).

[0127] The receiving unit 273 of the terminal device 2 of a user belonging to the information exchange group receives information regarding a new start instruction transmitted from the server device 3, along with the user ID of the user belonging to the information exchange group, in response to the information regarding the new start instruction transmitted from the terminal device 2 of another user belonging to the information exchange group.

[0128] The receiving unit 273 of the terminal device 2 of a user belonging to the information exchange group receives input information and user IDs of other users belonging to the information exchange group, transmitted from the server device 3, via the terminal communication interface 21.

[0129] (Display processing unit 274) The display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) displays an image drawn based on character video data including motion data generated by the generation unit 271 as the distribution image for the distribution user on the display device 23, in response to the instruction to start the distribution event. The motion data is transmitted as character video data to the viewing user's terminal device 2 (viewing terminal device). The display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) also displays an image drawn based on the distribution user's character video data (motion data) received by the receiving unit 273 as the distribution image for the distribution user on the display device 23.

[0130] The following describes the rendering process of distributed images based on motion data. In the following description, the rendering process of distributed images based on motion data in the distribution terminal device uses the user ID stored in the terminal storage device 22, and the rendering process of distributed images based on motion data in the viewing terminal device uses the user ID received from the server device 3.

[0131] First, the display processing unit 274 extracts the model ID of the character object associated with the user ID and the object ID of the object being used from the user table T1. Next, the display processing unit 274 reads the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the object being used from the terminal storage device 22. Next, the display processing unit 274 generates an animation of the character object with the object being used attached to the placement location based on the read model data, the image information and placement location of the object being used, and the motion data. If the object IDs of multiple sub-objects that make up the character object are associated with the model ID of the character object, the display processing unit 274 extracts the model data corresponding to the object IDs of the multiple sub-objects associated with the extracted model ID. Next, the display processing unit 274 generates an animation of the character object with the object being used attached to the placement location based on the model data of the multiple sub-objects that were read, the image information and placement location of the object being used, and the motion data. The object being used may be included in multiple sub-objects.

[0132] If the motion data includes face motion data, the display processing unit 274 generates an animation for the character object so that the character object's facial expression changes based on the face motion data. This allows the display processing unit 274 to generate an animation for the character object that moves in sync with the facial movements of the streaming user, based on the read model data and face motion data. If the motion data includes both face motion data and body motion data, the display processing unit 274 generates an animation for the character object that moves in sync with the facial and body movements of the streaming user, based on the read model data, face motion data, and body motion data. Next, the display processing unit 274 draws a streaming image that includes the generated character object animation (moving image) and a background image showing the background generated using the background data stored in the terminal storage device 22. Then, the display processing unit 274 outputs drawing data indicating the drawn streaming image, and terminates the drawing process of the streaming image based on the motion data. If three-dimensional model data is used in the drawing process of the streaming image, the streaming image drawn in the drawing process of the streaming image is a three-dimensional moving image that depicts a three-dimensional virtual space including a character object that moves in sync with the facial and body movements of the streaming user. The following primarily describes distributed images that include character object animations (moving images), but distributed images are not limited to those that include character object animations (moving images). Distributed images may also include still images of character objects. Furthermore, distributed images may display new still images of character objects at predetermined intervals.

[0133] Furthermore, the display processing unit 274 may perform a modification process to display on the display device 23 a character object to which an object owned by the distribution user or an object sent by the viewing user as an object gift has been applied, in response to a modification instruction from the distribution user. For example, if the object to be modified by the distribution user is a "headband", the display processing unit 274 extracts the object ID of the said object ("headband") from the objects owned by the distribution user associated with the user ID in the user table T1 and stores it as an object used by the distribution user. The display processing unit 274 then extracts the image information and placement location associated with the object ID stored as an object used by the distribution user from the object table T2. Next, the display processing unit 274 generates a distribution image that includes the distribution user's character object with the object ("headband") attached to its head (the placement location associated with the object ID of the "headband") based on the extracted image information and placement location. The display processing unit 274 may also generate a distribution image that includes a character object with an object attached that moves in conjunction with the movement of the attachment part (placement location) of the character object. For example, in the streamed image, the attached object "headband" is displayed to move in conjunction with the movement of the character object's head.

[0134] Furthermore, the display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) displays the distributed image including the additional object in response to the viewing user's instruction to add an additional object. Also, the display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) displays the distributed image including the additional object in response to information indicating the instruction to add an additional object received from the viewing user's terminal device 2.

[0135] The display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) refers to object table T2 and extracts a still image or video (image information) of the effect object associated with the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with information indicating the addition of an additional object, if the object ID received is the object ID of any effect object, and generates a distribution image including the extracted still image or video. For example, if an instruction to add an effect object representing confetti or fireworks is given, the display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) generates a distribution image including a video of an effect object (not shown) that imitates confetti or fireworks. Similarly, if the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with information indicating the addition of an additional object is the object ID of any effect object, the display processing unit 274 refers to object table T2 and extracts a still image or video (image information) of the effect object associated with the object ID received along with the information indicating the addition of an additional object, and generates a distribution image for the viewing user including the extracted still image or video.

[0136] The display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device), if the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with information indicating the instruction to add an additional object is the object ID of a normal object, refers to object table T2 and extracts the still image or moving image (image information) and placement location of the normal object associated with the object ID of the normal object. Next, the display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) generates a distribution image for the viewing user that includes the still image or moving image of the normal object placed at the location indicated by the placement location. Similarly, if the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with information indicating the instruction to add an additional object is the object ID of a normal object, the display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) refers to object table T2 and extracts the still image or moving image (image information) and placement location of the normal object associated with the object ID of the normal object, and generates a distribution image for the viewing user that includes the still image or moving image of the normal object placed at the location indicated by the placement location.

[0137] The display processing unit 274 may generate a distribution image that includes a video of a normal object moving within the display area of ​​the distribution image. For example, the display processing unit 274 may generate a distribution image that includes a video of a normal object that moves as if falling from the top to the bottom of the distribution image. In this case, the normal object may be displayed within the display area of ​​the distribution image from the start of the fall until it falls to the bottom edge of the output image, and may disappear from the distribution image after falling to the bottom edge of the distribution image. The movement route of the normal object may be from left to right, from right to left, from top left to bottom left, or in any other direction of the distribution image, and may also be a route along a straight line, circular orbit, elliptical orbit, spiral orbit, or any other trajectory.

[0138] The display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) displays notification information regarding the attachment instruction within the distribution image if the object ID received along with information indicating an instruction to add an additional object from the viewing user is the object ID of the attachment object. Next, the display processing unit 274 refers to object table T2 and extracts the image information and placement location of the attachment object associated with the object ID received along with the information indicating an instruction to add an additional object. Next, the display processing unit 274 generates a distribution image including the distribution user's character object with the attachment object attached to the attachment area indicated by the placement location, based on the extracted image information and placement location. Next, the display processing unit 274 generates a distribution image that includes the distribution user's character object attached to the attachment area indicated by the attachment location, based on the extracted image information and placement location.

[0139] Furthermore, the display processing unit 274 of the user's terminal device 2 displays an information exchange screen relating to a new information exchange group to which the user and all other users designated by the user belong, in response to a user's instruction to start a new information exchange. Also, when the receiving unit 273 receives information regarding a new start instruction from another user and the user IDs of each user belonging to the new information exchange group, the display processing unit 274 of the user's terminal device 2 displays an information exchange screen relating to that information exchange group.

[0140] Furthermore, the display processing unit 274 of the terminal device 2 of a user belonging to an information exchange group displays the information exchange screen related to that information exchange group in response to a user's instruction to start information exchange within the information exchange group (for example, text chat, etc.). Also, when the receiving unit 273 receives information regarding the instruction to start information exchange by other users belonging to the information exchange group and the user IDs of each user belonging to the information exchange group, the display processing unit 274 of the terminal device 2 of a user belonging to an information exchange group displays the information exchange screen related to that information exchange group.

[0141] Furthermore, when input information is entered by a user while the information exchange screen is being displayed, the display processing unit 274 of the terminal device 2 belonging to the information exchange group displays the entered input information below the latest input information in the user display area of ​​the information exchange screen. Also, when input information from another user is received by the receiving unit 273 while the information exchange screen is being displayed, the display processing unit 274 of the terminal device 2 belonging to the information exchange group displays the received input information below the latest input information in the other user display area of ​​the information exchange screen.

[0142] (Audio output unit 275) The audio output unit 275 of the viewing user's terminal device 2 (viewing terminal device) outputs the voice of the streaming user, generated based on the audio data, from a speaker (not shown) when the streaming user's audio data is received by the receiving unit 273. In addition, when the audio output unit 275 of the viewing terminal device receives the streaming user's character video data and audio data, it outputs the streaming user's voice based on the audio data from the speaker, along with the display processing unit 274 displaying the streaming user's streaming image. As a result, the streaming user's voice is output while the lips of the streaming user's character object included in the streaming image are moving, allowing the viewing user to view the streaming image which includes the streaming user's character object that appears to be emitting sound.

[0143] (Server device 3) Figure 6 shows an example of the schematic configuration of the server device 3. The server device 3, which provides various services such as video distribution services and information exchange services to each terminal device 2 of multiple users, transmits the character video data and information indicating various instructions of the distribution user, which are sent from the distribution user's terminal device 2, to the terminal devices 2 of one or more viewing users. The server device 3 also transmits input information (text information, stamp images, etc.) received from one user's terminal device to the terminal devices 2 of one or more other users belonging to the same information exchange group as that user. The server device 3 may also transmit information received from one user's terminal device 2 to the terminal device 2 of the user indicated by the user ID included in the destination information received at the same time as the information. To this end, the server device 3 is equipped with a server communication I / F 31, a server storage device 32, and a server processing device 33.

[0144] The server communication interface 31 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination thereof. Server device 3 can send information to and receive information from other devices via the server communication interface 31.

[0145] The server storage device 32 is, for example, a semiconductor memory device such as ROM or RAM. The server storage device 32 may also be, for example, a magnetic disk, an optical disk, or any other storage device capable of storing data. The server storage device 32 stores operating system programs, driver programs, application programs, and data used in processing in the server processing device 33. The server storage device 32 may also store user table T1, object table T2, group table T3, and reference information table T4 as data. The server storage device 32 also stores the distribution user ID of a distribution user currently distributing data, and stores the user ID (viewing user ID) of a viewing user who is viewing the distribution image of that distribution user in association with the distribution user ID of that distribution user. The distribution user ID is a type of information related to the distribution image. The information related to the distribution image may also be distribution image identification information (distribution image ID). For example, the distribution image ID, distribution user ID, and viewing user ID may be stored in association with each other.

[0146] The server processing unit 33 is a processor that loads the operating system program, driver program, and control program stored in the server storage device 32 into memory and executes the instructions contained in the loaded program. The server processing unit 33 is, for example, an electronic circuit such as a CPU, MPU, DSP, GPU, or a combination of various electronic circuits. The server processing unit 33 may also be implemented by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. In Figure 6, the server processing unit 33 is shown as a single component, but the server processing unit 33 may be a collection of multiple physically separate processors. For example, multiple processors that operate cooperatively in parallel to execute instructions may be implemented. The server processing unit 33 functions as a server receiving unit 331 and a server transmitting unit 332 by executing various instructions contained in the control program.

[0147] (Server receiving unit 331) The server receiving unit 331 receives information regarding the start of distribution instructions transmitted from the distribution user's terminal device 2 (distribution terminal device) along with the distribution user ID via the server communication interface 31. The server receiving unit 331 also receives information regarding the end of distribution instructions transmitted from the distribution user's terminal device 2 (distribution terminal device) along with the distribution user ID via the server communication interface 31. In addition, the server receiving unit 331 receives information for displaying the distribution image of the distribution user transmitted from the distribution user's terminal device 2 (distribution terminal device) along with the distribution user ID via the server communication interface 31.

[0148] Furthermore, the server receiving unit 331 receives information regarding the instruction to start viewing transmitted from the viewing user's terminal device 2 (viewing terminal device) via the server communication interface 31, along with information regarding the distributed image and the viewing user ID. The server receiving unit 331 also receives information indicating additional instructions transmitted from the viewing user's terminal device 2 (viewing terminal device) via the server communication interface 31, along with the object ID, the distribution user ID, and the viewing user ID.

[0149] Furthermore, the server receiving unit 331 receives information regarding the new start instruction, the user ID of the user in question, and the user IDs of other users selected as information exchange partners from the terminal device 2 of a user who wishes to start new information exchange, via the server communication interface 31. In this case, the server receiving unit 331 sets the group ID of the new information exchange group. The server receiving unit 331 then stores the received user ID of the user and the user IDs of the other users in the server storage device 32 as user IDs belonging to the new information exchange group, associating them with the group ID of the new information exchange group.

[0150] Furthermore, the server receiving unit 331 receives information regarding the information exchange start instruction, the user ID of the user, and destination information from the terminal device 2 of a user belonging to the information exchange group via the server communication I / F 31. If the received destination information is a group ID indicating an information exchange group, the server receiving unit 331 refers to the server storage device 32 and extracts the user ID of the user associated with the group ID received as destination information. The server receiving unit 331 then identifies the extracted user ID as the user ID of the user on the destination terminal device 2. If the received destination information is a user ID, the server receiving unit 331 also identifies the user ID received as destination information as the user ID of the user on the destination terminal device 2. The server receiving unit 331 then passes the received information regarding the information exchange start instruction and the user ID of the user on the destination terminal device 2 to the server transmitting unit 332.

[0151] Furthermore, the server receiving unit 331 receives input information, the user ID of the user, and destination information from the terminal device 2 of a user belonging to the information exchange group via the server communication I / F 31. Next, the server receiving unit 331 extracts the user ID of the user on the destination terminal device 2 based on the received destination information. The server receiving unit 331 then passes the received input information, the user ID of the user who entered the input information, and the user ID of the user on the destination terminal device 2 to the server transmitting unit 332.

[0152] (Server transmission unit 332) The server transmission unit 332 transmits the character video data and user ID of the distribution user received by the server reception unit 331 to the viewing user's terminal device 2 (viewing terminal device) indicated by the viewing user ID associated with the distribution user ID of the distribution user, which is stored in the server storage device 32, via the server communication interface 31. The server transmission unit 332 transmits the character video data, audio data, and user ID of the distribution user received by the server reception unit 331 to the viewing user's terminal device 2 (viewing terminal device) indicated by the viewing user ID associated with the distribution user ID of the distribution user, which is stored in the server storage device 32, via the server communication interface 31.

[0153] Furthermore, the server transmission unit 332 transmits information indicating various instructions from the distribution user received by the server reception unit 331 to the terminal device 2 (viewing terminal device) of the viewing user indicated by the viewing user ID associated with the distribution user ID of the distribution user, which is stored in the server storage device 32, via the server communication interface 31. In addition, the server transmission unit 332 transmits information indicating various instructions from the viewing user received by the server reception unit 331 to the terminal device 2 of the distribution user indicated by the distribution user ID associated with the user ID of the viewing user, and to the terminal device 2 of other viewing users indicated by the user IDs of other viewing users associated with the distribution user ID, via the server communication interface 31.

[0154] Furthermore, in response to the setting of a new group ID for the new information exchange group by the server receiving unit 331, the server transmitting unit 332 extracts the user IDs belonging to the new information exchange group, which are associated with the new group ID, from the server storage device 32. Then, the server transmitting unit 332 transmits information regarding the display instruction of the information exchange screen for the new information exchange group to the terminal device 2 of the user identified by the extracted user ID, via the server communication interface 31.

[0155] Furthermore, when the server transmission unit 332 receives information regarding the information exchange start instruction and destination information from the server reception unit 331, it transmits information regarding the display instruction of the information exchange screen related to the received information exchange start instruction to the user terminal device 2 identified by the user ID related to the destination information, via the server communication interface 31.

[0156] Furthermore, when the server transmission unit 332 receives input information, the user ID of the user who entered the input information, and the user ID of the user of the destination terminal device 2 from the server reception unit 331, it transmits the received input information via the server communication interface 31 to the terminal device 2 of the user identified by a user ID other than the user ID of the user who entered the input information.

[0157] (Examples of various screens) The following describes examples of various screens displayed on the display device 23 of the user's terminal device 2, with reference to Figures 7 to 14(a) and 15 to 16. In Figures 7 to 14(a) and 15 to 16, multiple components with the same reference numerals are components that have equivalent functions.

[0158] Figure 7(a) shows an example of a distribution screen 600 displayed on the display device 23 of the distribution user's terminal device 2 (distribution terminal device). The distribution screen 600 includes a distribution image 602 containing a character object 601, a response information display area 603, an acquisition button 604, and a distribution end button 605. Figure 7(b) shows an example of a distribution screen 610 displayed on the display device 23 of the viewing user's terminal device 2 (viewing terminal device). The distribution screen 610 includes a distribution image 612 containing a character object 611, an evaluation button 613, an add button 614, a comment button 615, a response information display area 616, and a viewing end button 617.

[0159] The terminal device 2 (distribution terminal device) of the distribution user performs the process of transmitting information regarding the distribution start instruction from the transmission unit 272 to the server device 3 in response to the start of a distribution event. A distribution event is started when the distribution user inputs a distribution event start instruction from the distribution user to the terminal processing device 27 from the input device 24 in response to the distribution user's operation on a predetermined screen (not shown), such as the home screen, displayed on the display device 23. Next, the terminal device 2 (distribution terminal device) simultaneously (in parallel) performs the transmission process of character video data etc. by the transmission unit 272 and the display processing process of the distribution screen 600 by the display processing device 274.

[0160] When the receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives information indicating one or more streaming users who are currently streaming, the display processing unit 274 of the terminal device 2 (viewing terminal device) displays a streaming event selection screen (not shown) on the display device 23. The streaming event selection screen includes selection objects for selecting one or more streaming users who are currently streaming. Next, the transmitting unit 272 of the terminal device 2 (viewing terminal device) obtains information from the input device 24 indicating the streaming user selected from among the one or more streaming users included in the streaming event selection screen in response to the viewer's operation of the input device 24. In this way, the transmitting unit 272 receives the viewer's instruction to start viewing the streamed image from the input device 24. Then, the transmitting unit 272 transmits information regarding the viewing start instruction, including information indicating the selected streaming user, information regarding the streamed image, and the viewer's user ID stored in the terminal storage device 22 to the server device 3 via the terminal communication interface 21.

[0161] The server receiving unit 331 of server device 3 receives information regarding the start of viewing instructions, information regarding the distributed image, and the user ID of the viewing user transmitted from terminal device 2 (viewing terminal device) via server communication I / F 31. Next, the server transmitting unit 332 of server device 3 identifies the terminal device 2 (viewing terminal device) of the viewing user identified by the user ID received along with the information regarding the start of viewing instructions. Next, the server transmitting unit 332 distributes information for displaying the distributed image identified by the information regarding the distributed image received along with the information regarding the start of viewing instructions to the identified terminal device 2 (viewing terminal device) via server communication I / F 31. The information for displaying the distributed image is either the character video data of the distribution user, or the character video data and audio data of the distribution user.

[0162] When character video data is received by the receiving unit 273 of terminal device 2 (viewing terminal device), the display processing unit 274 of terminal device 2 (viewing terminal device) displays the distribution screen 610 based on the received character video data. When character video data and audio data are received by the receiving unit 273 of terminal device 2 (viewing terminal device), the display processing unit 274 of terminal device 2 (viewing terminal device) displays the distribution screen 610 based on the received character video data, and the audio output unit 275 of terminal device 2 (viewing terminal device) outputs the voice of the distribution user generated based on the audio data from a speaker (not shown). The server receiving unit 331 of server device 3 registers (stores in server storage device 32) the viewing user ID of the viewing user of terminal device 2 (viewing terminal device) that sent information regarding the start of viewing as the viewing user ID of the viewing user currently viewing.

[0163] The following explains each display element included in the streaming screen 600.

[0164] The character object 601 included in the distribution screen 600 is an animation (moving image) of the distribution user's character object generated by the display processing unit 274. The display processing unit 274 generates the animation of the distribution user's character object based on the generated motion data of the distribution user and the character object associated with the distribution user's user ID stored in the user table T1. If the motion data includes face motion data, the display processing unit 274 changes the facial expression of the character object 601 in synchronization with the movement of the distribution user's facial expression. If the motion data includes body motion data, the display processing unit 274 moves the corresponding parts of the character object 601 in synchronization with the movement of the distribution user's body. The display processing unit 274 displays the distribution image 602, including the generated character object 601, on the display device 23.

[0165] The response information display area 603 included in the distribution screen 600 contains response information (comment information, evaluation information, and / or information about additional objects) transmitted from each terminal device 2 (viewing terminal device) displaying the distribution screen 610.

[0166] In the example of the response information display area 603 shown in Figure 7(a), it is shown that comment information indicating the comment "cool" was sent from terminal device 2 (viewing terminal device) of a viewing user named "User R". Next, it is shown that evaluation information was sent from terminal device 2 (viewing terminal device) of a viewing user named "User F". And finally, it is shown that evaluation information was most recently sent from terminal device 2 (viewing terminal device) of a viewing user named "User A".

[0167] The distribution end button 605 included in the distribution screen 600 is an object image such as a button object used to end the display of the distribution screen 600 and end the distribution of the distribution image 612. When the distribution user selects the distribution end button 605 by operating the input device 24, the display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) ends the display of the distribution screen 600. Then, the transmission unit 272 of the terminal device 2 (distribution terminal device) transmits information regarding the distribution end instruction, along with the distribution user ID stored in the terminal storage device 22, to the server device 3 via the terminal communication I / F 21. When the server receiving unit 331 of the server device 3 receives the information regarding the distribution end instruction transmitted from the distribution user's terminal device 2 (distribution terminal device), it deletes the distribution user ID received along with the distribution end instruction information from the server storage device 32. Furthermore, the server receiving unit 331 deletes the viewing user ID associated with the deleted distribution user ID from the server storage device 32. As a result, the distribution of the distribution image for the distribution user who selected the distribution end button 605 ends. Furthermore, the distribution user ID of a distribution user who has ended their distribution does not necessarily have to be deleted from the server storage device 32. In this case, the distribution user ID of a distribution user who has ended their distribution may be stored in the server storage device 32 as a distribution termination user ID. Also, the viewing user ID associated with the distribution user ID of a distribution user who has ended their distribution does not necessarily have to be deleted from the server storage device 32. In this case, the viewing user ID associated with the distribution user ID of a distribution user who has ended their distribution may be stored in the server storage device 32 as a viewing termination user ID. In addition, the server transmission unit 332 may transmit information regarding the distribution termination instruction to the terminal device 2 (viewing terminal device) of a viewing user identified by the viewing user ID associated with the distribution user ID of a distribution user who has ended their distribution. In this case, the display of the distribution screen 610 will also be terminated on the viewing user's terminal device 2 (viewing terminal device) that is displaying the distribution image of the distribution user, upon receiving the information regarding the distribution termination instruction.

[0168] The following explains each display element included in the streaming screen 610.

[0169] The character object 611 included in the distribution screen 610 is an animation (moving image) of the distribution user's character object generated by the display processing unit 274. The display processing unit 274 generates the animation of the distribution user's character object based on the distribution user's character video data (motion data) received by the receiving unit 273 and the character object associated with the distribution user's user ID stored in the user table T1. An example of the display process of the distribution image 612 including the character object 611 will be described below.

[0170] When server device 3 receives information from the viewing user's terminal device 2 (viewing terminal device) regarding the user's instruction to start viewing the streamed image, information regarding the streamed image (the streaming user ID of the streamed image), and the viewing user's user ID, it stores the received viewing user's user ID in association with the streaming user ID of the streamed image.

[0171] If the character video data (motion data) of the broadcasting user received by the receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) includes the broadcasting user's face motion data, a broadcast image 612 including a character object 611 whose facial expression changes in sync with the broadcasting user's facial movements is displayed. For example, if the broadcasting user speaks, a broadcast image 612 including a character object 611 whose lips are moving is displayed. Also, if the motion data of the broadcasting user received by the receiving unit 273 includes the broadcasting user's body motion data, a broadcast image 612 including a character object 611 whose corresponding body parts move in sync with the broadcasting user's body movements is displayed. When the broadcasting user's character video data and audio data are received, a broadcast image 612 based on the received character video data is displayed by the display processing unit 274, and the broadcasting user's voice, generated based on the received broadcasting user's audio data, is output by the audio output unit 275. As a result, user A can listen to the broadcasting user's voice while watching the broadcasting user's character object 611 which is moving as if it were speaking. With that, the display process for the streamed image 612 is completed. The display process for the streamed image 612 is executed each time the streamer's output information is received at predetermined time intervals (for example, every 2 seconds).

[0172] The evaluation button 613 included in the distribution screen 610 is an object image for instructing the transmission of evaluation information regarding the character object 611 included in the distribution screen 610, or the transmission of evaluation information regarding the distribution user related to the character object 611. When the evaluation button 613 is selected in response to the operation of the input device 24 by the viewing user, the transmission unit 272 of the terminal device 2 (viewing terminal device) transmits to the server device 3 via the terminal communication I / F 21 evaluation information indicating that a "positive evaluation of the character object 611 included in the distribution screen 610" has been given, or an evaluation indicating that a "positive evaluation of the distribution user related to the character object 611 included in the distribution screen 610" has been given, along with information regarding the instruction to display the evaluation information. The server receiving unit 331 of the server device 3 may store the received evaluation information in the server storage device 32, and the server transmitting unit 332 of the server device 3 may transmit the received evaluation information to the distribution user's terminal device 2 (distribution terminal device).

[0173] The evaluation button 613 may also be an object image that instructs the transmission of evaluation information indicating that a "negative evaluation of the character object 611 included in the distribution screen 600" has been made, or evaluation information indicating that a "negative evaluation of the distribution user related to the character object 611 included in the distribution screen 600" has been made. Furthermore, the distribution screen 610 may include multiple types of evaluation buttons 613 (for example, two types of evaluation buttons: an evaluation button for sending positive evaluation information and an evaluation button for sending negative evaluation information).

[0174] If the distribution screen 610 is displayed on multiple terminal devices 2 (viewing terminal devices), aggregated information obtained by aggregating the evaluation information transmitted from each terminal device 2 (viewing terminal device) may be included in the distribution screen 600 and / or distribution screen 610. For example, the server device 3 may count the number of evaluation information transmitted from each of the multiple terminal devices 2 (viewing terminal devices) and generate display information to include the number of evaluation information in the distribution screen 600 and / or distribution screen 610. The aggregated information is not limited to the number of evaluation information, but may also include information about the time period in which the evaluation information was transmitted. The aggregation period may be the period from the start of the distribution event to the present time, the period from the start of display of the distribution screen 600 to the present time, the period from the start of distribution of the distribution screen 610 to the present time, or a predetermined period (for example, the period from 21:00 to 24:00). Multiple aggregation periods may also be set. If the server device 3 can transmit the evaluation information to the terminal devices 2 (distribution terminal devices) of the distribution users, the aggregation process of the evaluation information may be performed by the terminal devices 2 (distribution terminal devices). In this case, the process of including the summary information in the distribution screen 600 may be performed by terminal device 2 (distribution terminal device), and the process of including the summary information in the distribution screen 610 may be performed by terminal device 2 (viewing terminal device) that is displaying the distribution screen 610.

[0175] The additional button 614 included in the distribution screen 610 is an object image that instructs the display of the distribution screen 600 and the selection screen 700 for selecting additional objects to be displayed on the distribution screen 610. When the additional button 614 is selected in response to the operation of the input device 24 by the viewing user, the selection screen 700 is displayed on the viewing user's terminal device 2 (viewing terminal device). Details of the selection screen 700 will be described later.

[0176] The comment button 615 included in the streaming screen 610 is an object image that instructs the display of a comment input window (not shown) for entering comments (strings) to be displayed on the streaming screen 600 and the streaming screen 610. When the comment button 615 is selected in response to the operation of the input device 24 by the viewing user, the comment input window (not shown) is superimposed on the streaming screen 610.

[0177] The comment input window is displayed only on the display device 23 of the terminal device 2 of the viewing user who selected the comment button 615, and the string being entered in the comment input window is not displayed on the display device 23 of other terminal devices 2. For example, the display processing unit 274 of the viewing user's terminal device 2 creates comment information indicating the comment (string) entered in the comment input window in response to the operation of the input device 24 by that viewing user. The comment information is a type of response information.

[0178] The response information display area 616 included in the distribution screen 610 contains response information (comment information, evaluation information, and / or information about additional objects (described later)) created in response to operations by the viewing user, and response information transmitted from the terminal device 2 of other viewing users who are viewing the distribution image 612 of the same distribution user.

[0179] In the example of the response information display area 616 shown in Figure 7(b), it is indicated that comment information (string) indicating the comment "cool" was sent from terminal device 2 of a viewing user named "User R". It is also indicated that evaluation information was sent from terminal device 2 of a viewing user named "User F". Furthermore, it is indicated that evaluation information created by terminal device 2 of user A, who is operating this terminal device 2, was sent.

[0180] When a user of terminal device 2, which is displaying the streaming screen 610, creates response information and sends a transmission instruction to the server device 3 via terminal communication I / F 21, the transmission unit 272 of terminal device 2 sends the created response information along with information regarding instructions for displaying the response information. In addition, the user ID of the user of the viewing information stored in terminal storage device 22 is transmitted along with the information regarding instructions for displaying the response information.

[0181] The server receiving unit 331 of the server device 3 receives information regarding the display instruction of response information transmitted from the terminal device 2 displaying the distribution screen 610, along with the response information and the viewing user ID. The server transmitting unit 332 of the server device 3 extracts the distribution user ID stored in association with the received viewing user ID from the server storage device 32. The server transmitting unit 332 then transmits the received response information and viewing user ID to the terminal device 2 of the distribution user indicated by the extracted distribution user ID and the terminal device 2 of the viewing user indicated by the extracted viewing user ID. The receiving unit 273 of each terminal device 2 displaying the distribution screen 610 receives the response information and viewing user ID transmitted from the server device 3 via the terminal communication I / F 21. The display processing unit 274 of each terminal device 2 displaying the distribution screen 610 displays the distribution screen 610, which includes the response information and the name of the viewing user who entered the response information in chronological order within the response information display area 616, based on the name associated with the received response information and the received viewing user ID. The name associated with the received user ID is information extracted by the display processing unit 274 by referencing the user table T1. In this way, the display processing unit 274 of each terminal device 2 displaying the distribution screen 610 displays the received response information and the name of the viewing user who entered the response information in chronological order in the response information display area 616 each time response information is received.

[0182] An upper limit may be set on the number of response information items that can be displayed in the response information display area 603 and / or the response information display area 616. In the example of the response information display area 603 shown in Figure 7(a) and the response information display area 616 shown in Figure 7(b), the upper limit on the number of response information items that can be displayed is "3". If the number of response information items displayed in each terminal device 2 exceeds the set upper limit, the display processing unit 274 of each terminal device 2 displays a distribution screen 600 that includes the three newly received response information items in the response information display area 603 in order, or a distribution screen 610 that includes them in the response information display area 616.

[0183] Furthermore, the response information within the response information display area 616 may be displayed in a scrollable manner. In this case, in response to a swipe operation by the user (for example, an operation in which a finger touches the screen and moves from the bottom to the top of the screen), response information that is not currently contained in the response information display area 616 will be scrolled and displayed within the response information display area 616. The response information within the response information display area 603 included in the distribution screen 600 may also be displayed in a scrollable manner, similar to the response information display area 616.

[0184] The "End Viewing" button 617 is an object image, such as a button object, used to end the display of the streaming screen 610 and end viewing of the streaming image 612. When the viewing user selects the "End Viewing" button 617 by operating the input device 24, the display processing unit 274 of the viewing user's terminal device 2 ends the display of the streaming screen 610. The transmission unit 272 of the terminal device 2 then transmits information regarding the "End Viewing" instruction, along with the viewing user ID and destination information stored in the terminal storage device 22, to the server device 3 via the terminal communication interface 21. When the server receiving unit 331 of the server device 3 receives the information regarding the "End Viewing" instruction and the viewing user ID transmitted from the user's terminal device 2, it stores the viewing user ID that is identical to the received viewing user ID and is associated with the streaming user ID of the streaming user related to the streaming screen 610 that was ended, in the server storage device 32 as the "End Viewing" user ID. The "End Viewing" user ID is stored in association with the streaming user ID of the streaming user related to the streaming screen 610 that was ended. As a result, viewing of the streaming image for the viewing user who selected the "End Viewing" button 617 ends.

[0185] Figure 8(a) shows an example of a selection screen 700 displayed on the display device 23 of the viewing user's terminal device 2 (viewing terminal device). As shown in Figure 8(a), the selection screen 700 is displayed superimposed on the distribution screen 610. The selection screen 700 may be displayed after the display of the distribution screen 610 has been temporarily interrupted, and the distribution screen 610 may be redisplayed when the display of the selection screen 700 ends.

[0186] In the example shown in Figure 8(a), images representing eight types of additional candidate objects are included within the display area of ​​the selection screen 700. The names of the additional candidate objects may be placed near the images representing the additional candidate objects. The number of additional candidate objects included within the display area of ​​the selection screen 700 is not limited to eight; it may be seven or fewer, or nine or more. If the display processing unit 274 of the terminal device 2 cannot include some of the images representing multiple additional candidate objects within the display area of ​​the selection screen 700, it may display the images representing the additional candidate objects in a scrollable manner. In this case, in response to a swipe operation by the viewing user (for example, an operation in which a finger touches the screen and moves from right to left on the screen), images representing additional candidate objects not currently included within the display area of ​​the selection screen 700 are scrolled and displayed within the display area of ​​the selection screen 700.

[0187] Each additional candidate object represents an additional object. The image representing an additional candidate object is, for example, an icon image or thumbnail image representing a gift object, which is a type of additional object. On the selection screen 700, the images of each additional candidate object may be displayed categorized by the type of additional object (free gifts, paid gifts, accessories, cheering goods, appeals, variety items, cooperative gifts that require the cooperation of users belonging to a group, etc.). Additional objects belonging to paid gifts are additional objects that can be purchased by the user by consuming "coins," which are virtual currency. Additional objects belonging to free gifts are additional objects that can be obtained without consuming monetary value, but instead by consuming points that can be obtained by watching the broadcast, etc. Additional objects may also be objects that can be sent to the broadcaster by the viewer user after playing a predetermined mission executed on terminal device 2 and clearing the predetermined mission. Additional objects may also be objects that can be sent to the broadcaster by the viewer user after playing a known lottery game executed on terminal device 2 and winning the lottery game. In addition, additional objects may be managed (stored) as objects owned by the viewer user.

[0188] When an additional candidate object is selected in response to the user's operation of the input device 24, the input device 24 inputs an add command to the terminal processing device 27, along with the object ID corresponding to the selected additional candidate object. An example of the add processing in response to the add command is described below.

[0189] First, the display processing unit 274 obtains the input additional instruction and object ID, then refers to the object table T2 and extracts the still or moving image (image information) of the additional object associated with the obtained object ID from the terminal storage device 22. Next, the display processing unit 274 generates a new distribution image 612 that includes the extracted still or moving image, in addition to the distribution image 612 which contains the distribution user's character object 611.

[0190] Next, when the transmitting unit 272 obtains the input additional instruction and object ID, it transmits information indicating the additional instruction, along with the obtained object ID, the viewing user ID of the viewing user stored in the terminal storage device 22, and information regarding the distributed image (distributing user ID of the distributed image) to the server device 3 via the terminal communication interface 21. Next, the server receiving unit 331 of the server device 3 receives the information indicating the additional instruction, the object ID, the viewing user ID, and information regarding the distributed image (distributing user ID of the distributed image) via the server communication interface 31. If information regarding the distributed image (distributing user ID of the distributed image) is not transmitted, the server transmitting unit 332 of the server device 3 extracts the distributed user ID stored in association with the received viewing user ID from the server storage device 32. The server transmitting unit 332 extracts viewing user IDs other than the received viewing user ID from among the viewing user IDs associated with the information regarding the distributed image (distributing user ID of the distributed image) or the extracted viewing user ID. The server transmission unit 332 then transmits information indicating the received additional instruction, the object ID, and the received user ID (the user ID of the viewing user who issued the additional instruction) to the terminal device 2 of the viewing user indicated by the extracted viewing user ID, via the server communication interface 31. The server transmission unit 332 also transmits information indicating the received additional instruction, the object ID, and the received viewing user ID to the terminal device 2 of the distribution user indicated by the distribution image information (distribution user ID of the distribution image) or the extracted distribution user ID, via the server communication interface 31.

[0191] Then, the receiving unit 273 of the terminal device 2 of the viewing user (excluding the viewing user who issued the additional instruction) and the distribution user receives information indicating the additional instruction, as well as the user ID and object ID of the viewing user who issued the additional instruction, via the terminal communication I / F 21. Next, the display processing unit 274 of the terminal device 2 refers to the object table T2 and extracts a still image or moving image (image information) of the additional object associated with the received object ID from the terminal storage device 22. Next, the display processing unit 274 generates a new distribution image 612 that includes the extracted still image or moving image in the distribution images 602 and 612, which include the distribution user's character objects 601 and 611. The new distribution images 602 and 612 may also display new response information including the name of the viewing user who issued the additional instruction and the name of the additional object associated with the received object ID. With this, the additional processing is completed.

[0192] Figure 8(b) shows another example of the distribution screen 710 displayed on the display device 23 of the terminal device 2. Similar to the distribution screen 610, the distribution screen 710 includes a distribution image 612 containing a character object 611, an evaluation button 613, an add button 614, a comment button 615, a response information display area 616, and a viewing end button 617.

[0193] Unlike the distribution screen 610, the distribution screen 710 includes additional objects 711 and 712. Both additional objects 711 and 712 are still or moving images representing additional objects. Below, we will explain an example of the display process for additional object 712, using the case where the viewer who instructed the addition of additional object 712 is user A.

[0194] When an additional candidate object (corresponding to the additional object 712) is selected in response to an operation of the input device 24 by user A, the input device 24 of user A's terminal device 2 inputs an additional instruction to the terminal processing device 27 along with the object ID corresponding to the selected additional candidate object. The display processing unit 274 of user A's terminal device 2 obtains the input additional instruction and object ID, and then refers to the object table T2 to extract a still image or moving image (image information) of the additional object associated with the obtained object ID from the terminal storage device 22. The display processing unit 274 of user A's terminal device 2 then generates a new distribution image 612 that includes the additional object 712 based on the extracted still image or moving image, in addition to the distribution image 612 which includes the distribution user's character object 611.

[0195] The transmission unit 272 of user A's terminal device 2 transmits information indicating the additional instruction related to the additional object 712, the acquired object ID, and user A's viewing user ID stored in the terminal storage device 22 to the terminal devices 2 of the viewing users (excluding the viewing user who issued the additional instruction) and the distribution users via the server device 3. When the information indicating the additional instruction related to the additional object 712, user A's viewing user ID, and the object ID are received by the receiving unit 273 of the terminal devices 2 of the viewing users (excluding user A) and the distribution users, the display processing unit 274 of the terminal devices 2 of the viewing users (excluding user A) and the distribution users refers to the object table T2 and extracts a still image or moving image (image information) of the additional object associated with the received object ID from the terminal storage device 22. The display processing unit 274 of the terminal device 2 then generates a new distribution image 612 based on the extracted still image or moving image, which includes the distribution image 611 of the distribution user. With this, the display processing of the additional object 712 is completed.

[0196] Additional objects added to the streaming screen may change shape, size, placement, color, brightness, and / or design while the streaming screen is being displayed, if predetermined change conditions are met. The predetermined change conditions may also be conditions related to information indicating the instruction to add an additional object. For example, the predetermined change conditions may be that the timing of an additional instruction by any of the users viewing the same streaming screen is met, and / or that at least two of the users viewing the same streaming screen have given additional instructions for multiple different types of additional objects. Meeting the timing conditions for additional instructions means, for example, that the difference in the timing of additional instructions by multiple users is within a predetermined period. In response to an additional instruction by a user, the server device 3 awards the user a predetermined number of points. If the predetermined change conditions are met, the user who gave the additional instruction that satisfies the predetermined change conditions may be awarded more points than the predetermined number. The predetermined change conditions may be that the conditions related to the timing of additional instructions by any of the multiple viewing users having a predetermined relationship are met, and / or that the multiple viewing users having a predetermined relationship have given additional instructions for multiple different types of additional objects. In this case, even if the conditions related to the timing of additional instructions by a viewing user who does not have a predetermined relationship are met, the predetermined change conditions will not be met.

[0197] A predetermined relationship among multiple viewing users is, for example, when one of the multiple viewing users is a mutual follower of another of the multiple viewing users. Furthermore, if one user follows another user and that user also follows the first user (i.e., the first user and the other user are mutually following each other), then the first user and the other user are mutual followers. Alternatively, a predetermined relationship among multiple viewing users may also be when one of the multiple viewing users is mutually following a predetermined viewing user who is mutually following another of the multiple viewing users.

[0198] Furthermore, a predetermined relationship among multiple viewing users may be that multiple viewing users are friends. Alternatively, a predetermined relationship among multiple viewing users may be that one of the viewing users' terminal devices stores specific information (such as a phone number, email address, or predetermined ID (identification)) of the other viewing users. Another predetermined relationship among multiple viewing users may be that the number of times multiple viewing users have used the same information exchange service to exchange information is predetermined or greater. Finally, a predetermined relationship among multiple viewing users may be that each of the multiple viewing users has participated in the same and / or different information exchange services a predetermined or greater number of times.

[0199] Furthermore, the predetermined relationship among multiple viewing users may also refer to the relationship concerning the actions of the other viewing users when one of the multiple viewing users distributes a distribution image as the distribution user. Here, the relationship concerning the actions of the other viewing users is the relationship in which the number of times the other viewing users have viewed the distribution image distributed by the one viewing user is equal to or greater than a predetermined number of views. Alternatively, the relationship concerning the actions of the other viewing users may also refer to the relationship in which the other viewing users have sent comment information on the distribution image distributed by the one viewing user is equal to or greater than a predetermined number of transmissions.

[0200] Furthermore, the predetermined relationship among multiple viewers may also be a relationship relating to a predetermined event in which each of the multiple viewers is participating. Here, the relationship relating to a predetermined event means that each of the multiple viewers is currently participating in the same event. Alternatively, the relationship relating to a predetermined event may also mean that each of the multiple viewers has participated in the same event in the past. Alternatively, the relationship relating to a predetermined event may also mean that one or more of the multiple viewers are participating in an event in which other viewers have participated in the past.

[0201] When the user selects a change button (not shown) for the distributed image 602 by operating the input device 24, the selection screen 800 is displayed. For example, the change button is included in the distributed screen 600. The change button may also be included in a predetermined screen such as the home screen displayed on the display device 23. Figure 9(a) shows an example of the selection screen 800 displayed on the display device 23 of the user's terminal device 2 (distributed terminal device). The selection screen 800 shown in Figure 9(a) is displayed superimposed on the distributed screen 600. The selection screen 800 may be displayed after the display of the distributed screen 600 has ended.

[0202] The selection screen 800 includes multiple placement candidate objects 801, each representing one of several placement locations, and attachment candidate objects 802, which are candidates for attachment objects to be newly attached to the character object 601. When the distribution user selects one of the multiple placement candidate objects 801 by operating the input device 24, the attachment candidate object 802 corresponding to the object ID of the object owned by the distribution user and associated with the placement candidate corresponding to the selected placement candidate object 801 is displayed on the selection screen 800. When the distribution user selects one of the multiple attachment candidate objects 802 by operating the input device 24, the input device 24 inputs a change instruction to the terminal processing device 27, and the display processing device 274 executes the change process described above. The display processing device 274 then displays the distribution screen 810 (Figure 9(b)), which includes a distribution image 812 that includes the character object 811 with the new attachment object attached.

[0203] Figure 9(b) shows another example of the distribution screen 810 displayed on the display device 23 of the distribution user's terminal device 2 (distribution terminal device). The distribution screen 810 includes at least a distribution image 812 that includes the distribution user's character object 811 wearing a new attachment object. In the example shown in Figure 9(b), the character object 811 is displayed in which the clothing worn by the distribution user's character object 601 has been changed.

[0204] Information regarding the change instructions entered by the streaming user who initiated the video streaming service is transmitted via server device 3 to terminal device 2 (viewing terminal device) of the viewing user who is viewing the streaming image from that streaming user. As a result, the change processing corresponding to the change instructions entered by the streaming user is also executed on terminal device 2 (viewing terminal device) of the viewing user. Consequently, the streaming image including the streaming user's character object with the newly attached object is displayed on terminal device 2 (viewing terminal device).

[0205] As explained with reference to Figures 8 and 9, in response to additional instructions from the viewing user, at least a portion of the distributed image, including the user-distributed user's character object, is modified, and in response to modification instructions from the distribution user, at least a portion of the distribution user's character object is modified. This makes it possible to prevent viewers from becoming bored during the distribution event and to increase the motivation of users to continue using the video distribution service.

[0206] Furthermore, in conventional information processing systems, a streaming user had to interrupt a streaming event to a viewing user via a character object in order to change the streaming image containing the character object. After the streaming event was interrupted, it was necessary to activate a function to change the appearance of the character object and / or the background image within the streaming image containing the character object. The streaming user then had to perform the cumbersome procedure of changing the character object and / or streaming image using the change function and then resuming the streaming event. In contrast, the information processing system 1 of this embodiment allows the streaming user to change their character object without interrupting the streaming event and resetting the streaming user's character object. As a result, the user interface for changing the streaming user's character object during a streaming event is not complicated in the information processing system 1 of this embodiment, making it possible to improve usability compared to conventional information processing systems. In addition, since the user's character object can be changed without interrupting the streaming event in the information processing system 1 of this embodiment, the number and amount of communication between the server device 3 and the terminal device 2 can be reduced. Thus, the information processing system 1 of this embodiment makes it possible to improve communication between the server device 3 and the terminal device 2.

[0207] Figure 10(a) shows an example of a group settings screen 900 displayed on the display device 23 of a user's terminal device 2. The group settings screen 900 is displayed in response to user operations on a predetermined screen, such as the home screen, displayed on the display device 23. Users who can operate a terminal device 2 capable of displaying the group settings screen 900 are either streaming users or viewing users. Users who can operate a terminal device 2 capable of displaying the group settings screen 900 are not limited to streaming users and viewing users; any user who can receive services provided by the server device 3 may operate the terminal device 2 capable of displaying the group settings screen 900. A new information exchange group is set up in response to user operations on the group settings screen 900.

[0208] As shown in Figure 10(a), the group settings screen 900 includes a candidate display area 901, a selected object 902, and a start button 903. The candidate display area 901 displays information indicating one or more candidate users who can belong to the information exchange group. For example, one or more candidate users who can belong to the information exchange group are users who are mutual followers of the user operating the terminal device 2 that displays the group settings screen 900. Alternatively, one or more candidate users who can belong to the information exchange group may be users who are friends with the user operating the terminal device 2 that displays the group settings screen 900. The one or more candidate users who can belong to the information exchange group are not limited to those described above; any user who has a predetermined relationship with the user operating the terminal device 2 that displays the group settings screen 900 is acceptable. In the example shown in Figure 10(a), the display processing unit 274 of the user's terminal device 2 displays the thumbnail image and name of each of the character objects of the multiple users who can belong to the information exchange group in the candidate display area 901 as information indicating the candidate user.

[0209] The selection object 902 is an operation object used to select each candidate user displayed in the candidate display area 901. In the example shown in Figure 10(a), the display processing unit 274 of the user's terminal device 2 displays a checkbox object corresponding to each candidate user as the selection object 902. For example, when a selection object 902 is selected by the user's input device 24, the selection object 902 will be displayed with a "check" mark. In this case, the candidate user corresponding to the selection object 902 that is displayed with a "check" mark is selected. Also, when a selection object 902 that is displayed with a "check" mark is selected by the user's input device 24, the selection object 902 will be displayed without a "check" mark. In this case, the candidate user corresponding to the selection object 902 that is displayed without a "check" mark is not selected. Each selection object 902 is associated with the user ID of the corresponding candidate user.

[0210] The start button 903 is a button object, etc., that allows a user to create a new information exchange group and start exchanging information with other users belonging to the newly created information exchange group. When the start button 903 is selected by the user operating the input device 24, a new information exchange group is created to which the user operating the terminal device 2 and the candidate users selected by the selection object 902 belong as members. For example, the display processing unit 274 of the user's terminal device 2 identifies the user ID of the user stored in the terminal storage device 22 and the user ID of the candidate user associated with the selection object 902 that has been selected by the selection object 902. The display processing unit 274 then associates the identified user ID and the user ID of the candidate user with the newly created group ID as members and stores them in the group table T3. At this time, the display processing unit 274 may associate the name of the information exchange group that was automatically created with the newly created group ID, or it may associate the name of the information exchange group entered by the user. When a new information exchange group is created in response to the selection of the start button 903, the display processing unit 274 displays the information exchange screen 1000 for the users belonging to the new information exchange group.

[0211] Figure 10(b) shows an example of an information exchange start screen 910 displayed on the display device 23 of the user's terminal device 2. The information exchange start screen 910 is displayed in response to user operations on a predetermined screen, such as the home screen, displayed on the display device 23. Users who can operate the terminal device 2 capable of displaying the information exchange start screen 910 are either streaming users or viewing users. Users who can operate the terminal device 2 capable of displaying the information exchange start screen 910 are not limited to streaming users and viewing users; any user who can receive services provided by the server device 3 is acceptable. Information exchange is initiated by the selected information exchange group in response to user operations on the information exchange start screen 910.

[0212] As shown in Figure 10(b), the information exchange start screen 910 includes a candidate group area 911. The candidate group area 911 displays information indicating the information exchange group to which the user operating the terminal device 2 displaying the information exchange start screen 910 belongs. The information indicating the information exchange group can be selected by the user, and the group ID of the corresponding information exchange group is associated with the information indicating the information exchange group.

[0213] When a user operates the input device 24 and selects information indicating one of the information exchange groups included in the information exchange start screen 910, the group ID associated with the selected information exchange group is identified. The display processing unit 274 displays the information exchange screen 1000 for each user identified by the user ID associated with the identified group ID (each user belonging to the information exchange group indicated by the identified group ID).

[0214] Figure 11(a) shows an example of an information exchange screen 1000 displayed on the display device 23 of the user's terminal device 2. On the information exchange screen 1000, information exchange takes place among multiple users belonging to a newly established information exchange group, which is created when the user selects the start button 903 included in the group setting screen 900. Alternatively, on the information exchange screen 1000, information exchange takes place among multiple users belonging to an information exchange group selected by the user on the information exchange start screen 910. Hereinafter, among the multiple users belonging to an information exchange group that performs information exchange, users other than the user operating the terminal device 2 displaying the information exchange screen 1000 may be referred to as "other users". The information exchange screen 1000 (and each information exchange screen described later) may be implemented by a so-called chat screen that implements text chat, etc.

[0215] The information exchange screen 1000 includes other user input information 1001, other user information 1002, a text input object 1003, a candidate display button 1004, and user input information 1005. Other user input information 1001 is input information entered by other users. Other user information 1002 is information indicating other users (such as a character object icon image) that is associated with each other user input information 1001 and displayed accordingly. The text input object 1003 is an operation object for the user to input text information as input information. The candidate display button 1004 is an operation object for displaying candidate information 1101 corresponding to character-related images so that the user can select it. User input information 1005 is input information entered by the user.

[0216] Each time input information from another user transmitted from another user's terminal device 2 is received via server device 3, the display processing unit 274 displays other user input information 1001, indicating the received input information from another user, in the other user display area of ​​the information exchange screen 1000. For example, the other user display area is the area near the left edge of the information exchange screen 1000, and the other user input information 1001 is displayed in the other user display area in chronological order from the oldest to the newest, from top to bottom of the screen. Each time user input information is entered through a user input operation on text input object 1003, the display processing unit 274 displays user input information 1005, indicating the entered user input information, in the user display area of ​​the information exchange screen 1000. For example, the user display area is the area near the right edge of the information exchange screen 1000, and the user input information 1005 is displayed in the user display area in chronological order from the oldest to the newest, from top to bottom of the screen.

[0217] When the information exchange screen 1000 relating to a newly configured information exchange group is displayed, the transmission unit 272 of the user's terminal device 2 transmits information about the new information exchange group to the server device 3 via the terminal communication interface 21. The information about the new information exchange group includes the group ID of the new information exchange group, the name of the new information exchange group, and the user IDs of the users belonging to the new information exchange group.

[0218] The server receiving unit 331 of the server device 3 receives information about information exchange groups via the server communication interface 31. The server receiving unit 331 stores the received information about the new information exchange group in the server storage device 32. For example, the server receiving unit 331 stores the name of the new information exchange group and the user IDs of the users belonging to the new information exchange group in the server storage device 32, associated with the group ID of the new information exchange group.

[0219] The server transmission unit 332 of the server device 3 transmits information about the new information exchange group to the terminal device 2 of other users, indicated by the user ID of a user belonging to the new information exchange group other than the user who transmitted the information about the new information exchange group, via the server communication interface 31.

[0220] When the receiving unit 273 of another user's terminal device 2 receives information about a new information exchange group transmitted from the user's terminal device 2 via the server device 3, it stores the name of the new information exchange group and the user IDs of the users belonging to the new information exchange group in the group table T3, associating them with the group ID of the new information exchange group. Alternatively, when the display processing unit 274 of another user's terminal device 2 receives information about a new information exchange group, it may automatically display the information exchange screen 1000 related to the new information exchange group.

[0221] In the information exchange screen 1000, when new text information is entered as input information into the text input object 1003 by user operation, the display processing unit 274 displays the newly entered text information as user input information 1005 in the user display area. In the information exchange screen 1000, when image information or video information, etc., is entered as input information by user operation, the display processing unit 274 displays the entered image information or video information, etc., as user input information 1005 in the user display area. The information entered as input information is not limited to text information, image information, and video information, but may also be information in a format usable by a predetermined application program.

[0222] When input information entered by another user through their operation on the information exchange screen 1000 displayed by another user's terminal device 2 is received via the server device 3 along with the other user's user ID, the received input information is displayed in the other user display area as other user input information. In this case, images and other information related to the received other user's user ID are displayed as other user information 1002.

[0223] The transmission unit 272 of the user's terminal device 2 transmits the newly entered text information and the user's user ID to other users' terminal devices 2 via the server device 3. As a result, the newly entered text information is displayed as other user input information 1001 on the information exchange screen 1000 displayed on the terminal devices 2 of other users who are members other than the user who entered the text information.

[0224] On the information exchange screen 1000, if the user selects the candidate display button 1004 by operating the input device 24, the input device 24 sends an instruction to the terminal processing unit 27 to display candidate information for selecting a character-related image. Next, the display processing unit 274 determines whether a character-related image including the user's character object has been previously generated and stored in the terminal storage device 22. If no character-related image including the user's character object is stored, the display processing unit 274 displays a generation instruction object 1006 to instruct the generation of a character-related image. If a character-related image including the user's character object is stored, for example, the display processing unit 274 displays the information exchange screen 1100 including the four types of stored candidate information 1101.

[0225] Figure 11(b) shows an example of an information exchange screen 1010 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 1010 is displayed when the user selects a candidate display button 1004 included in the information exchange screen 1000, and it is determined that character-related images, including the user's character object, are not stored in the terminal storage device 22.

[0226] The information exchange screen 1010, like the information exchange screen 1000, includes other user input information 1001, other user information 1002, a text input object 1003, a candidate display button 1004, and user input information 1005. Furthermore, the information exchange screen 1010 includes a generation instruction object 1006. In the example shown in Figure 11(b), the information exchange screen 1010 displays information prompting the generation of character-related images along with the generation instruction object 1006.

[0227] In the information exchange screen 1010, if the user selects a generation instruction object 1006 by operating the input device 24, the input device 24 sends a character-related image generation instruction to the terminal processing device 27. When the generation unit 271 receives the character-related image generation instruction from the input device 24, it executes the character-related image generation process. An example of the character-related image generation process is described below.

[0228] First, the generation unit 271 extracts the model ID of the character object associated with the user ID stored in the terminal storage device 22, and the object ID of the object being used, from the user table T1. Next, the generation unit 271 reads the model data associated with the extracted model ID, and the image information and placement location associated with the extracted object ID of the object being used, from the terminal storage device 22. Next, the generation unit 271 reads the motion data and object data for the character-related image from the terminal storage device 22. Next, the generation unit 271 generates a character-related image that includes a character object with a predetermined movement and expression, and object data such as a background and text, based on the read model data, the image information and placement location of the object being used, and the motion data and object data for the character-related image. Finally, the generation unit 271 associates the generated character-related image with an associated image ID to uniquely identify the character-related image and stores it in the terminal storage device 22. Furthermore, the generation unit 271 generates candidate information indicating the generated character-related images, associates the generated candidate information with an associated image ID for identifying the character-related images, and stores it in the terminal storage device 22. The generation unit 271 also associates the object ID of the object used during the generation of the character-related images with the associated image ID and stores it in the terminal storage device 22, and the character-related image generation process ends. Note that some or all of the generated character-related images may be set to be unavailable for use in the information exchange service. In this case, an available group consisting of character-related images that can be used in the information exchange service may be set. When a new character-related image that can be used in the information exchange service is generated, the new character-related image is set to be included in the available group.

[0229] The following describes an example of the character-related image transmission process. First, the transmission unit 272 of the terminal device 2 that generated the character-related image transmits the character-related image, candidate information, and object ID of the object being used, which are associated with the related image ID and stored in the terminal storage device 22, to the server device 3 via the terminal communication I / F 21. The user ID stored in the terminal storage device 22 is transmitted to the server device 3 along with the character-related image, candidate information, and object ID of the object being used, which are associated with the related image ID. When the server receiving unit 331 of the server device 3 receives the character-related image, candidate information, and object ID of the object being used, which are associated with the related image ID, and the user ID, it stores the related image ID in association with the user ID, and stores the character-related image, candidate information, and object ID of the object being used in association with the related image ID in the server storage device 32. The server transmission unit 332 then transmits the received related image ID associated with the user ID, the character-related image associated with the related image ID, the candidate information, and the object ID of the object used to the terminal devices 2 of other users belonging to the information exchange group to which the user of the terminal device 2 that transmitted the character-related image belongs, via the server communication interface 31. The terminal devices 2 of the other users store the transmitted related image ID associated with the user ID, the character-related image associated with the related image ID, the candidate information, and the object ID of the object used in the terminal storage device 22. With this, the character-related image transmission process is completed. Note that the transmission unit 272 of the terminal device 2 that generated the character-related image does not need to transmit the candidate information and the object ID of the object used associated with the related image ID stored in the terminal storage device 22 to the server device 3. Also, the candidate information and the object ID of the object used associated with the user ID of one user belonging to the same information exchange group do not need to be transmitted to the terminal devices 2 of other users belonging to the same information exchange group.Furthermore, the timing at which a character-related image associated with the user ID of a user belonging to the same information exchange group is transmitted to the terminal device 2 of another user belonging to the same information exchange group may be the timing at which the user selects the character-related image as input information. The character-related image transmission process may be executed each time the character-related image generation process and / or additional generation process are performed.

[0230] The display processing unit 274 reads the candidate information generated by the generation unit 271 from the terminal storage device 22 and displays it. Figure 12(a) shows an example of the information exchange screen 1100 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 1100 is the screen that is displayed when the user selects a generation instruction object 1006 included in the information exchange screen 1010.

[0231] The information exchange screen 1100, like the information exchange screen 1000, includes other user input information 1001, other user information 1002, a text input object 1003, a candidate display button 1004, and user input information 1005. Furthermore, the information exchange screen 1010 includes four types of generated candidate information 1101 and an additional generation button 1102. The number of types of candidate information 1101 included in the information exchange screen 1100 is not limited to four; it may be three or fewer types, or five or more types. If the display processing unit 274 of the terminal device 2 cannot include some of the multiple types of candidate information 1101 within the display area of ​​the information exchange screen 1100, it may display the candidate information 1101 in a scrollable manner. In this case, in response to a swipe operation by the user (for example, an operation in which a finger touches the screen and moves from right to left on the screen), the candidate information 1101 that is not currently included in the display area of ​​the information exchange screen 1100 is scrolled and displayed within the display area of ​​the information exchange screen 1100.

[0232] When a user selects one of four candidate information items 1101 in response to an operation of the input device 24 (Figure 12(b)), the input device 24 sends the associated image ID to the terminal processing device 27. Next, the display processing unit 274 extracts the character-related image identified by the associated image ID received from the input device 24 from the terminal storage device 22 and displays the extracted character-related image as user input information 1201 in the user display area of ​​the information exchange screen 1200 (Figure 13). The character-related image displayed as user input information 1201 is either a two-dimensional still image or a three-dimensional still image depicting a three-dimensional virtual space including a character object. If the character-related image displayed as user input information 1201 is generated based on multiple consecutive motion data, the character-related image may also be a two-dimensional moving image including a character object that moves for a predetermined time (e.g., 3 seconds). In this case, the character-related image may also be a three-dimensional moving image depicting a three-dimensional virtual space including a character object that moves for a predetermined time (e.g., 3 seconds). As shown in Figure 13, the display size of the character-related image displayed as user input information 1201 on the display screen of the display device 23 is smaller than the display size of the distributed image 602 on the display screen of the display device 23. Thus, the rendering processing load for drawing the character-related image on the display screen of the display device 23 is smaller than the rendering processing load for drawing the distributed image 602 on the display screen of the display device 23. Furthermore, since the character video data, which includes motion data for displaying the distributed image 602, is transmitted continuously, the amount of data transmitted when sending the character video data is greater than the amount of data transmitted when sending the related image ID for displaying the character-related image. In this way, by using character-related data in various services, it becomes possible to display a user's character object on another user's terminal device 2 while reducing the processing load of terminal device 2 and / or server device 3, as well as the transmission load between terminal device 2 and server device 3, compared to when distributed images are used.

[0233] Along with the display processing unit 274 displaying the user input information 1201, the transmission unit 272 transmits the associated image ID received from the input device 24 and the user ID stored in the terminal storage device 22 to the server device 3 via the terminal communication interface 21. The server transmission unit 332 of the server device 3 extracts the user IDs of other users associated with the same information exchange group as the user ID received by the server reception unit 331 from the server storage device 32. The server transmission unit 332 also transmits the associated image ID and user ID received by the server reception unit 331 to the terminal device 2 of the other user identified by the extracted user ID, via the server communication interface 31.

[0234] Then, the receiving unit 273 of the other user's terminal device 2 receives the associated image ID and user ID transmitted from the server device 3 via the terminal communication I / F 21. Next, the display processing unit 274 of the other user's terminal device 2 extracts the character-related image identified by the associated image ID received by the receiving unit 273 from the terminal storage device 22, and displays the extracted character-related image as other user input information in the other user display area of ​​the distribution screen. In this way, the same character-related image used by the user is also displayed on the other user's terminal device 2 associated with the information exchange group to which the user belongs. If the character-related image identified by the received associated image ID is not stored in the terminal storage device 22, the transmitting unit 272 of the other user's terminal device 2 sends an image transmission request to the server device 3. The image transmission request is sent along with the received associated image ID and / or user ID, as well as the other user's user ID. When the server transmission unit 332 of the server device 3 receives an image transmission request from the server reception unit 331, it extracts the character-related image associated with the associated image ID received along with the image transmission request from the server storage device 32. Next, the server transmission unit 332 transmits the extracted character-related image along with the received associated image ID and user ID to the other user's terminal device 2. The reception unit 273 of the other user's terminal device 2 receives the associated image ID, character-related image, and user ID transmitted from the server device 3 via the terminal communication interface 21, associates them with each other, and stores them in the terminal storage device 22. Then, the display processing unit 274 of the other user's terminal device 2 displays the stored character-related image as other user input information in the other user display area of ​​the distribution screen.

[0235] Returning to Figure 12(a), if the user selects the additional generation button 1102 in response to the operation of the input device 24, the display processing unit 274 and generation unit 271 of the user's terminal device 2 execute the additional generation process. The additional generation process is the process of generating a new character-related image based on reference information relating to a character object representing the user. The reference information is information obtained by the reference information acquisition process and includes, for example, information indicating the displayed distribution image.

[0236] Referring to Figure 14(a), an example of the reference information acquisition process performed by the distribution user's terminal device 2 will be described. Hereinafter, the distribution user's terminal device 2 may be referred to as distribution terminal device 2a, and the viewing user's terminal device 2 may be referred to as viewing terminal device 2b. Figure 14(a) is a diagram showing an example of a distribution screen 600 displayed on the display device 23 of the distribution terminal device 2a. The distribution screen 600 shown in Figure 14(a) has the same display elements as the distribution screen 600 shown in Figure 7(a).

[0237] As shown in Figure 14(a), when the acquisition button 604 is selected in response to the operation of the input device 24 of the distribution terminal device 2a by the distribution user, the input device 24 inputs an instruction to acquire reference information to the terminal processing device 27.

[0238] The reference information includes image information of at least a portion of the distributed image 602. For example, the display processing unit 274 of the distribution terminal device 2a acquires image information of at least a portion of the distributed image 602 as reference information in response to an instruction to acquire reference information. The image information acquired as reference information is either captured image information (screenshot) of the distributed screen 600 captured by the display processing unit 274 at the time the instruction to acquire reference information was given, or image information corresponding to a portion of the image area of ​​that screenshot. The image information acquired as reference information may also be multiple screenshots (or a portion thereof) acquired by the display processing unit 274 in chronological order within the acquisition period (for example, the period from the time the instruction to acquire reference information was given to 2 seconds later). The acquisition period may also be the duration of the touch of the acquisition button 604 in response to the operation of the input device 24 of the distribution terminal device 2a by the distribution user. For example, the input device 24 of the distribution terminal device 2a may output the period from the time the distribution user's finger touched the acquisition button 604 to the time the distribution user's finger left the acquisition button 604 as the acquisition period. Hereafter, the image information acquired as reference information may be referred to as reference image information. The display processing unit 274 then stores the reference image information in the terminal storage device 22, and the reference information acquisition process ends. At least a portion of the image information of the distributed image 602 is an example of information that indicates the distributed image.

[0239] The reference information may include motion data of the character object 601 of the distributed image 602. In the example shown in Figure 14(a), the display processing unit 274 may acquire the motion data of the character object 601 of the distributed image 602 as reference information in response to an instruction to acquire reference information. For example, each time the generation unit 271 generates motion data, it temporarily stores motion data from the current time up to a predetermined time before (for example, if the current time is 8:05:30, the motion data from 8:05:00 to 8:05:30) in the terminal storage device 22. The display processing unit 274 then acquires the motion data of the character object 601 of the distributed image 602 that was displayed at the time of the instruction to acquire reference information as reference information. The display processing unit 274 may also acquire the motion data of the character object 601 of the distributed image 602 that was displayed within the acquisition period (for example, the period from that time up to 2 seconds before) from the time the instruction to acquire reference information was received as reference information. Hereafter, the motion data acquired as reference information may be referred to as reference motion data. The display processing unit 274 then stores the reference motion data in the terminal storage device 22, and the reference information acquisition process ends. The motion data of the character object 601 in the distributed image 602 is an example of information that indicates the distributed image.

[0240] The process for acquiring reference information relating to reference image information and reference motion data is not limited to being performed in response to the selection of the acquisition button 604 by the distribution user. For example, the process for acquiring reference information relating to reference image information and reference motion data may be performed automatically in response to instructions from the viewing user. In this case, the distribution screen 610 displayed by the viewing terminal device 2b of the viewing user who is viewing the distribution image 612 of the distribution user includes an acquisition button (not shown). When the acquisition button is selected in response to the operation of the input device 24 of the viewing terminal device 2b by the viewing user, the input device 24 inputs an instruction to acquire reference information to the terminal processing device 27, along with acquisition date and time information indicating the date and time when the acquisition button was selected. When the transmission unit 272 of the viewing terminal device 2b acquires the instruction to acquire reference information and the acquisition date and time information, it transmits information indicating the instruction to acquire reference information and the acquisition date and time information to the server device 3 via the terminal communication I / F 21. The viewing user ID of the viewing user stored in the terminal storage device 22 may also be transmitted along with the information indicating the instruction to acquire reference information and the acquisition date and time information. Next, the server receiving unit 331 of the server device 3 receives information indicating an instruction to acquire reference information and acquisition date and time information via the server communication interface 31. The server transmitting unit 332 of the server device 3 transmits the received information indicating an instruction to acquire reference information and acquisition date and time information to the distribution terminal device 2a via the server communication interface 31. The receiving unit 273 of the distribution terminal device 2a receives the received information indicating an instruction to acquire reference information and acquisition date and time information via the terminal communication interface 21. Based on the information indicating an instruction to acquire reference information and acquisition date and time information, the receiving unit 273 executes reference information acquisition processing related to reference image information and reference motion data at the date and time indicated by the acquisition date and time information. Then, the receiving unit 273 acquires the reference image information and / or reference motion data at the date and time indicated by the acquisition date and time information, stores the reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition processing ends.

[0241] Furthermore, the reference information acquisition process related to reference image information and reference motion data may be executed automatically when predetermined acquisition conditions are met. The predetermined acquisition conditions are conditions related to the response information included in the response information display area 603 within the distribution screen 600. For example, the display processing unit 274 of the distribution terminal device 2a acquires reference image information and / or reference motion data corresponding to the period with the largest number of response information entries.

[0242] The following describes an example of the reference information acquisition process related to reference image information and reference motion data corresponding to the period with the largest number of response information entries. First, the display processing unit 274 sets up multiple periods for each predetermined unit of time and stores the number of response information entries displayed within each period in the terminal storage device 22. The predetermined unit of time is, for example, 10 seconds. The predetermined unit of time may also be 5 seconds, 30 seconds, etc. If the predetermined unit of time is 10 seconds, as an example, the number of response information entries for each period, such as the period from 8:05:00 to 8:05:10, the period from 8:05:10 to 8:05:20, the period from 8:05:20 to 8:05:30, ..., is stored. There may be overlapping periods within each period (the period from 8:05:00 to 8:05:10, the period from 8:05:05 to 8:05:15, the period from 8:05:10 to 8:05:20, ...). The display processing unit 274 acquires reference image information and / or reference motion data for specific dates and times within each of the top Ma-ranked periods with the highest number of displayed response information within the past Na days. Na days are days 1, 2, 5, 10, etc. If Ma is "1", reference image information and / or reference motion data for a specific date and time within the period with the highest number of displayed response information is acquired. If Ma is "5", five pieces of reference image information and / or reference motion data are acquired for specific dates and times within each of the 1st to 5th ranked periods with the highest number of displayed response information. The specific date and time within a period is the start date and time of the period, the end date and time of the period, or any date and time within the period. The display processing unit 274 then stores the acquired reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0243] The display processing unit 274 may store in the terminal storage device 22 the number of a predetermined type of response information among the response information displayed within each period. For example, the display processing unit 274 may store the number of comment information, or the display processing unit 274 may store the number of evaluation information. The display processing unit 274 may also store the total number of multiple types of response information.

[0244] The predetermined acquisition conditions may also be conditions relating to viewers who are viewing the distribution screen 610 or viewers who have viewed the distribution screen 610. For example, the display processing unit 274 of the distribution terminal device 2a acquires reference image information and / or reference motion data corresponding to a timing when many viewers are simultaneously viewing the distribution image of the distribution user.

[0245] The following describes an example of the process for acquiring reference information related to reference image information and reference motion data corresponding to timings when many viewers are simultaneously viewing the streamed image of the streaming user. First, the server transmission unit 332 of the server device 3 counts the number of user IDs (viewer user IDs) of viewers who are viewing the streamed image of the streaming user as the number of simultaneous viewers at predetermined time intervals. Each time the server transmission unit 332 counts the number of simultaneous viewers, it transmits the number of simultaneous viewers information to the streaming terminal device 2a via the server communication I / F 31. The display processing unit 274 refers to the simultaneous viewers information received at predetermined time intervals and acquires reference image information and / or reference motion data for the top Ma timings with the highest number of simultaneous viewers within the past Na days. Na days are days 1, 2, 5, 10, etc. If Ma is "1", reference image information and / or reference motion data for the timing with the highest number of simultaneous viewers is acquired. If Ma is "5", five pieces of reference image information and / or reference motion data are acquired for each of the 1st to 5th timings with the highest number of simultaneous viewers. The display processing unit 274 then stores the acquired reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0246] The predetermined acquisition conditions may also be conditions relating to additional objects displayed on the distribution screen 600. For example, the display processing unit 274 of the distribution terminal device 2a acquires reference image information and / or reference motion data corresponding to the period with the largest number of displayed additional objects.

[0247] The following describes an example of the reference information acquisition process related to reference image information and reference motion data corresponding to the period with the largest number of displayed additional objects. First, the display processing unit 274 sets up multiple periods for each predetermined unit time (5 seconds, 10 seconds, 30 seconds, etc.) and stores the number of displayed additional objects displayed within each period in the terminal storage device 22. Overlapping periods may exist within each period. The display processing unit 274 acquires reference image information and / or reference motion data for a specific date and time within each period up to the highest number of displayed additional objects within the past Na days (Na days are 1 day, 2 days, 5 days, 10 days, etc.), up to the highest number of displayed additional objects. The specific date and time within the period is the start date and time of the period, the end date and time of the period, or any date and time within the period. Then, the display processing unit 274 stores the acquired reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0248] The display processing unit 274 may store in the terminal storage device 22 the number of additional objects of a predetermined type among the additional objects displayed within each period. For example, the display processing unit 274 may store the number of "bear" gift objects, or it may store the number of "accessory" gift objects. The display processing unit 274 may also store the total number of additional objects of multiple types. Furthermore, the display processing unit 274 of the distribution terminal device 2a may acquire reference image information and / or reference motion data corresponding to the period in which the total value of the displayed additional objects is highest. In this case, the additional objects can be transmitted to the distribution terminal device 2a by the viewer paying a price (such as "coins," which are virtual currency held by the viewer). For example, the display processing unit 274 stores in the terminal storage device 22 the total value of the displayed additional objects displayed within each period. The display processing unit 274 acquires reference image information and / or reference motion data at a specific date and time within each period up to the highest Ma rank with the highest total value of displayed objects within the past Na days (Na days are days 1, 2, 5, 10, etc.). Furthermore, when reference motion data is acquired, the object ID of the additional object displayed in the distributed image at the time of acquisition may also be acquired. Then, by using the reference motion data and the object ID of the additional object, a character-related image similar to the distributed image at the time of acquisition can be generated. In the video distribution service of this embodiment, the higher the price, the more luxurious, difficult to obtain, and / or popular the additional object provided. As a result, distribution users can use character-related images that have a high total price for the additional objects, a luxurious appearance, are difficult to obtain, and / or are popular.

[0249] The reference information may include response information contained in the response information display area 603 within the distribution screen 600. For example, each time comment information is displayed in the response information display area 603, the display processing unit 274 stores the displayed comment information in the terminal storage device 22 as reference information. In this case, date and time information such as the display date and time information or reception date and time information of the comment information may also be stored in the terminal storage device 22 along with the comment information. Furthermore, each time evaluation information is displayed in the response information display area 603, the display processing unit 274 may store the displayed evaluation information in the terminal storage device 22 as reference information. In this case, date and time information such as the display date and time information or reception date and time information of the evaluation information may also be stored in the terminal storage device 22 along with the evaluation information.

[0250] The comment information stored as reference information is not limited to that contained in the response information display area 603. For example, it may also be comment information entered by the streaming user and / or the viewing user in a specific game that may be played during the video streaming service. The specific game is, for example, a word game (quiz, association game, word chain, word collection game, riddle, answer checking game, etc.). The specific game is a network game in which streaming users can participate, each operating multiple terminal devices 2 connected to the network. The specific game may also be a network game in which viewing users can participate, each operating multiple terminal devices 2 connected to the network. Furthermore, the specific game may be a network game in which both streaming users and viewing users can participate, each operating multiple terminal devices 2 connected to the network. The specific game is not limited to word games, but may also be a competitive game, a sports simulation game, a shooting game, etc.

[0251] For example, comment information entered into each terminal device 2 by multiple users participating in the word game (broadcasting users and / or viewing users) is sent to the server device 3. The server device 3 sends the received comment information to the terminal devices 2 of the multiple users participating in the word game. At the end of the word game, the server device 3 may also send the comment information received during the word game to the terminal devices 2 of multiple users who were not participating in the word game. Each time comment information is received, the receiving unit 273 of each terminal device 2 stores the received comment information as reference information in the terminal storage device 22. In this case, date and time information such as the date and time the comment information was received may also be stored in the terminal storage device 22 along with the comment information.

[0252] The comment information stored as reference information may be limited to those that meet predetermined selection criteria. These predetermined selection criteria may include, for example, comment information that has received high ratings from other users, or comment information up to the top Mb ranking that has received a large number of high ratings from other users.

[0253] The reference information may include information about additional objects selected by the viewing user. For example, the object ID of an additional object transmitted along with information indicating an additional instruction in response to an operation on the selection screen 700 of the viewing terminal device 2b by a viewing user who is viewing the distributed image 612 of the distribution user may be stored in the terminal storage device 22 as reference information. For example, each time the receiving unit 273 of the distribution terminal device 2a receives information indicating an additional instruction, the user ID of the viewing user, and the object ID transmitted from the viewing terminal device 2b via the server device 3, it stores the received object ID in the terminal storage device 22 as reference information. In this case, date and time information such as the display date and time information of the additional object or the reception date and time information of the object ID of the additional object may be stored in the terminal storage device 22 along with the object ID of the additional object. In this way, information about additional objects displayed in the distribution screen 600 is stored in the terminal storage device 22 as reference information.

[0254] The object ID of an additional object stored as reference information may be the object ID of an additional object that is not displayed on the distribution screen 600. In this case, a selection screen (not shown) for selecting an additional object, similar to the selection screen 700, is displayed on the viewing terminal device 2b of a viewing user who is viewing the distribution image 612 of the distribution user. When an additional object is selected on this selection screen in response to an operation of the input device 24 of the viewing terminal device 2b by the viewing user, the input device 24 inputs the instruction to acquire reference information and the object ID of the selected additional object to the terminal processing device 27. When the transmission unit 272 of the viewing terminal device 2b acquires the instruction to acquire reference information and the object ID of the additional object, it transmits information indicating the instruction to acquire reference information and the object ID of the additional object to the server device 3 via the terminal communication interface 21. The viewing user ID of the viewing user stored in the terminal storage device 22 may also be transmitted along with the information indicating the instruction to acquire reference information and the object ID of the additional object. Next, the server receiving unit 331 of the server device 3 receives the information indicating the instruction to acquire reference information and the object ID of the additional object via the server communication interface 31. The server transmission unit 332 of the server device 3 transmits information indicating the instruction to acquire the received reference information and the object ID of the additional object to the distribution terminal device 2a via the server communication interface 31. The receiving unit 273 of the distribution terminal device 2a receives the information indicating the instruction to acquire the received reference information and the object ID of the additional object via the terminal communication interface 21. Based on the information indicating the instruction to acquire the reference information and the object ID of the additional object, the receiving unit 273 executes a reference information acquisition process to store the received object ID of the additional object as reference information in the terminal storage device 22. In addition to storing the received object ID of the additional object as reference information, the date and time of receipt of the information indicating the instruction to acquire the reference information may also be stored.

[0255] The reference information may include information about owned objects stored in association with the distribution user. For example, the object IDs of owned objects stored in association with the distribution user's user ID may be stored in the terminal storage device as reference information.

[0256] The reference information may include a profile image of the distribution user's character object. For example, a profile image is a type of profile information and includes an image of the distribution user's character object. The profile image does not necessarily have to include an image of the distribution user's character object; for example, the profile image may be an image that includes an image representing the distribution user. In addition to the profile image, the profile information may also include the distribution user's name, various information entered by the distribution user (distribution category, distribution time, favorite additional objects, etc.), etc. The distribution user's profile information is stored in the terminal storage device 22 of the distribution user's distribution terminal device 2a. The server device may also store the profile information transmitted from the distribution terminal device 2a in the server storage device 32. When the distribution user changes the profile information by operating the input device 24 of the distribution terminal device 2a, the reference information may also be automatically changed according to the changed profile information. In this case, the user may change the reference information (profile image) by operating the input device 24 of the distribution terminal device 2a.

[0257] The reference information may include image information that shows the virtual space in which the character object exists. For example, the streaming user's character object can move freely within the metaverse (internet space) world, as described later, in response to the streaming user's actions. In this case, image information showing the world may be acquired as reference information, either in response to the streaming user's actions or automatically.

[0258] The reference information may include image information that shows the game space in which the character object appears. For example, the streamer's character object can move freely within the 2D or 3D game space played by the streamer, in response to the streamer's actions. In this case, image information showing the game space may be acquired as reference information, either in response to the streamer's actions or automatically.

[0259] The reference information may include any information related to the distribution user or a character object that represents the distribution user, not limited to the examples described above.

[0260] (Reference information table T4) Figure 14(b) shows an example of the data structure of the reference information table T4 that manages reference information. The reference information table T4 is provided individually for each user, and the reference information table T4 shown in Figure 14(b) is the reference information table T4 of the user operating the terminal device 2 that stores the reference information table T4.

[0261] In the reference information table T4, reference information IDs, reference information, date and time information, etc., are stored in relation to each other. The reference information ID is an example of identification data used to uniquely identify each piece of reference information. Each time the reference information acquisition process is executed, a new reference information ID is assigned to the acquired reference information, and the new reference information ID and the acquired reference information are stored in the reference information table T4 in relation to each other. In addition, the date and time information is the date and time information acquired along with the reference information during the reference information acquisition process, and is stored in the reference information table T4 in relation to the new reference information ID along with the acquired reference information. Note that if date and time information is not acquired along with the reference information during the reference information acquisition process, the new reference information ID and the date and time information associated with the acquired reference information are not stored.

[0262] (Reference Information Selection Screen) Figure 15 shows an example of a reference information selection screen 1300 displayed on the display device 23 of the user's terminal device 2. The reference information selection screen 1300 is displayed when the additional generation process is started in response to the user selecting the additional generation button 1102 included in the information exchange screen 1100. For example, the display processing unit 274 of the terminal device displays the reference information selection screen 1300 in response to the user's selection of the additional generation button 1102 by operating the input device 24. Note that if no reference information is stored in the reference information table T4, the reference information selection screen 1300 will not be displayed.

[0263] The reference information selection screen 1300 includes candidate reference information 1301a and 1301b that indicate the reference information, object data 1302a and 1302b for character-related images, an add / generate instruction button 1303, and an exit button 1304.

[0264] Candidate reference information 1301a and 1301b, which indicate reference information, are image information or text information that indicates reference information stored in the reference information table T4, and each is displayed as selectable by the user. Candidate reference information 1301a shown in Figure 15 is an example of candidate reference information displayed when the reference information is reference image information or reference motion data. For example, if the reference information is reference image information, candidate reference information 1301a is the reference image information. Alternatively, if the reference information is reference image information, candidate reference information 1301a may be an image generated based on the reference image information (for example, an image with a reduced number of pixels from the reference image information). If the reference information is reference motion data, candidate reference information 1301a is an image showing a character object generated based on the reference motion data and the character object model data. Candidate reference information 1301b shown in Figure 15 is an example of an image displayed when the reference information is an additional object. For example, if the reference information is an additional object, candidate reference information 1301b is an image showing the additional object. Furthermore, if the reference information is comment information, the candidate reference information is image information or text information that represents the comment information.

[0265] The candidate reference information displayed on the reference information selection screen 1300 is not limited to two; it may be one, three, or more. If the display processing unit 274 of the terminal device 2 cannot include some of the candidate reference information within the display area of ​​the reference information selection screen 1300, it may display the candidate reference information in a scrollable manner. In this case, in response to a swipe operation by the user, candidate reference information not currently included within the display area of ​​the reference information selection screen 1300 may be scrolled and displayed within the display area of ​​the reference information selection screen 1300.

[0266] The candidate reference information included in the reference information selection screen 1300 may be information indicating only specific types of reference information. For example, the candidate reference information included in the reference information selection screen 1300 may be image information indicating reference image information only. Also, the candidate reference information included in the reference information selection screen 1300 may be image information indicating reference motion data only. Also, the candidate reference information included in the reference information selection screen 1300 may be image information or text information indicating comment information only. Also, the candidate reference information included in the reference information selection screen 1300 may be image information or text information indicating attached objects only. The candidate reference information included in the reference information selection screen 1300 may be image information or text information indicating attached objects sent as additional objects only. The candidate reference information included in the reference information selection screen 1300 may be image information or text information indicating gift objects sent as additional objects only. Also, the candidate reference information included in the reference information selection screen 1300 may be image information or text information indicating owned objects only. Also, the candidate reference information included in the reference information selection screen 1300 may be image information indicating profile images only. Furthermore, the candidate reference information included in the reference information selection screen 1300 may consist only of image information that represents image information within the virtual space. Note that the reference information selection screen 1300 may contain two or more types of candidate reference information.

[0267] The candidate reference information included in the reference information selection screen 1300 may be only a portion of the candidate reference information of a particular type of reference information. For example, the candidate reference information included in the reference information selection screen 1300 may be an image showing comment information extracted by a predetermined extraction method from the comment information stored as reference information in the reference information table T4.

[0268] In this case, the predetermined extraction method may be to extract identical comment information from the comment information stored as reference information, aggregate the number of identical comment information, and extract the comment information up to the top Mc rank with the highest aggregated number. For example, if the Mc rank is "2" and the reference information table T4 stores 5 "cool" comment information, 3 "cute" comment information, and 2 "do your best" comment information, then the two types of comment information, "cool" and "cute," will be extracted. Alternatively, the predetermined extraction method may be to extract comment information associated with date and time information for a predetermined period (e.g., 5 seconds before and after the time) related to the time when a specific additional object was displayed, from the comment information stored as reference information. For example, if the candidate reference information includes information indicating a specific additional object and comment information associated with date and time information for a predetermined period related to the time when that specific additional object was displayed, the user can select the specific additional object and the comment information that matches that specific additional object. This enables the information processing system 1 to provide character-related images that include specific additional objects and comment information desired by the user.

[0269] Furthermore, the candidate reference information included in the reference information selection screen 1300 may also be an image showing an additional object (attached object) extracted by a predetermined extraction method from among the additional objects (attached objects) stored as reference information in the reference information table T4. In this case, the predetermined extraction method is a method for extracting additional objects (attached objects) whose placement does not overlap with the object data for character-related images (data for objects showing backgrounds, text, etc.) used for character-related images.

[0270] In the reference information selection screen 1300, candidate reference information 1301a and 1301b do not necessarily have to be displayed. If candidate reference information 1301a and 1301b are not displayed in the reference information selection screen 1300, reference information selected from the reference information stored in the reference information table T4 according to a predetermined selection method is determined, and a character-related image based on the determined reference information is generated. The predetermined selection method is a method of randomly selecting a predetermined number of reference information from the reference information, or a method of selecting a predetermined number of reference information from the reference information according to the order in which each reference information is previously associated. The predetermined number is "1" or any natural number greater than or equal to "2". In addition, reference information may be selected from a specific type of reference information stored in the reference information table T4 according to a predetermined selection method. For example, a predetermined number of comment information may be selected from the comment information stored as reference information according to a predetermined selection method. The predetermined selection method is a method of randomly selecting a predetermined number of comment information from the comment information stored as reference information. Alternatively, the selection method may involve extracting identical comment information from the comment information stored as reference information, counting the number of identical comment information entries, and selecting the comment information with the highest count. For example, if the reference information table T4 stores five "cool" comment entries, three "cute" comment entries, and two "good luck" comment entries, the comment information "cool" will be selected (determined).

[0271] The object data 1302a and 1302b for character-related images are image information indicating the object data for character-related images stored in the terminal storage device 22, and each is displayed so as to be selectable by the user. The object data for character-related images included in the reference information selection screen 1300 is not limited to two types, and may be one type or three or more types. When the display processing unit 274 of the terminal device 2 cannot include a part of the object data for a plurality of types of character-related images within the display area of the reference information selection screen 1300, the object data for character-related images may be displayed in a scrollable manner. In this case, in response to a swipe operation by the user, the object data for character-related images that is not currently included in the display area of the reference information selection screen 1300 may be scrolled and displayed within the display area of the reference information selection screen 1300.

[0272] In the reference information selection screen 1300, the object data 1302a and 1302b for character-related images may not be displayed. When the object data 1302a and 1302b for character-related images are not displayed in the reference information selection screen 1300, a character-related image in which the object data for character-related images is not used is generated. Or, when the object data 1302a and 1302b for character-related images are not displayed, a character-related image in which predetermined object data for character-related images is used may be generated.

[0273] The additional generation instruction button 1303 is an example of an operation object for inputting an additional generation instruction for generating a new character-related image. For example, when the additional generation instruction button 1303 is selected in response to an operation of the input device 24 by the user, the input device 24 inputs an additional generation instruction to the generation unit 271.

[0274] When the generation unit 271 acquires an additional generation instruction from the input device 24, it determines reference information to be used in the generation process of a new character-related image. When either of the candidate reference information 1301a or 1301b has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 determines the reference information corresponding to the selected image. When neither of the candidate reference information 1301a or 1301b has been selected when the additional generation instruction button 1303 is selected, the generation unit 271 determines the reference information selected according to a predetermined selection method from the reference information stored in the reference information table T4.

[0275] Next, the generation unit 271 executes a generation process for generating a new character-related image based on the determined reference information. Hereinafter, an example of the generation process for generating a new character-related image when the determined reference information is reference image information will be described.

[0276] When the determined reference information is reference image information, the generation unit 271 uses the reference information (reference image information) as a new character-related image, associates a related image ID for uniquely identifying the new character-related image, and stores it in the terminal storage device 22. Further, the generation unit 271 may cut out an image within a predetermined image area of the reference information (reference image information), use the image as a new character-related image, associate a related image ID for uniquely identifying the new character-related image, and store it in the terminal storage device 22. Then, the generation unit 271 generates candidate information indicating the generated new character-related image, associates the related image ID for identifying the new character-related image with the generated candidate information, stores it in the terminal storage device 22, and the generation process ends. As a result, the information processing system 1 can provide a new character-related image including a character object in the same pose (posture) as the character object in the distribution image at the timing of acquiring the reference information.

[0277] The following describes an example of a generation process for generating a new character-related image when the determined reference information is reference motion data. First, the generation unit 271 extracts the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the object being used from the user table T1. Next, the generation unit 271 reads the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the object being used from the terminal storage device 22. Next, the generation unit 271 reads the determined reference information (reference motion data) and object data for the character-related image from the terminal storage device 22. Next, the generation unit 271 generates a new character-related image including a character object with a predetermined movement and expression, and object data such as background and text, based on the read model data, the image information and placement location of the object being used, and the reference information (reference motion data) and object data for the character-related image. Furthermore, if either object data 1302a or 1302b for character-related images has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 generates a new character-related image using the selected object data. This makes it possible for the information processing system 1 to provide a new character-related image that includes a character object in the same pose (posture) as the character object in the distributed image at the time of acquiring the reference information.

[0278] An example of a generation process for generating a new character-related image when the reference information is response information is described below. First, the generation unit 271 extracts the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the object being used from the user table T1. Next, the generation unit 271 reads the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the object being used from the terminal storage device 22. Next, the generation unit 271 reads the motion data and object data for the character-related image from the terminal storage device 22. Next, the generation unit 271 generates a new character-related image that includes a character object with a predetermined movement and expression, object data such as background and text, and object data corresponding to the reference information, based on the read model data, the image information and placement location of the object being used, the motion data and object data for the character-related image, and the reference information. Furthermore, if either object data 1302a or 1302b for character-related images has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 will generate a new character-related image using the selected object data. This enables the information processing system 1 to provide character-related images using information about character objects within the distribution screen.

[0279] If the reference information is comment information, the object data corresponding to the reference information is the object containing the comment information (for example, a character board object displaying the text "cool"). In this case, the newly generated character-related image will include the object containing the comment information.

[0280] An example of a generation process for generating a new character-related image when the reference information is related to an additional object is described below. First, the generation unit 271 extracts the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the object being used from the user table T1. Next, the generation unit 271 reads the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the object being used from the terminal storage device 22. Next, the generation unit 271 reads the motion data and object data for the character-related image from the terminal storage device 22. Then, based on the read model data, the image information and placement location of the object being used, the motion data and object data for the character-related image, and the reference information, the generation unit 271 generates a new character-related image that includes a character object with a predetermined movement and expression, object data such as background and text, and an additional object corresponding to the reference information. Furthermore, if either object data 1302a or 1302b for character-related images has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 will generate a new character-related image using the selected object data.

[0281] If the reference information concerns an attachment object among the additional objects, the additional object corresponding to the reference information is the attachment object applied to the character object. For example, if the additional object corresponding to the reference information is the attachment object "headband," the newly generated character-related image will include a character object wearing the headband. In this case, if the extracted usage object (an attachment object already applied to the character object) is of the same type as the attachment object corresponding to the reference information, the attachment object corresponding to the reference information may be applied to the character object instead of the extracted usage object. This enables the information processing system 1 to provide a character-related image using information about the character object in the distribution screen.

[0282] If the reference information concerns a gift object among the additional objects, the additional object corresponding to the reference information is the gift object held by the character object. For example, if the additional object corresponding to the reference information is the gift object "bear," the newly generated character-related image will include the character object holding the "bear." In this case, motion data for the character-related image, such as a pose corresponding to the character object's attributes and / or the type of gift object, may be applied to the character object. This enables the information processing system 1 to provide character-related images using information from the viewing user regarding the character object in the distribution screen.

[0283] In this case, multiple types of motion data for character-related images may be stored in the terminal storage device 22, and any of the motion data may be automatically applied. For example, different types of motion data may be stored for each parameter of the character object as motion data for character-related images. Parameters include gender (female, male, etc.), size (height above a predetermined value, below a predetermined value, etc.), personality (calm, lively, etc.), age (30 years or older, under 30, etc.), etc. For example, if the character object is female, motion data MF for character-related images may be automatically applied, and if the character object is male, motion data MM for character-related images may be automatically applied. If the height of the character object is above a predetermined value, motion data MT for character-related images may be automatically applied, and if the height of the character object is below a predetermined value, motion data ML for character-related images may be automatically applied. Furthermore, if the additional object corresponding to the reference information is the gift object "bear", motion data MG for character-related images may be automatically applied, and if the additional object corresponding to the reference information is the gift object "cyalume", motion data MP for character-related images may be automatically applied. Furthermore, if the character object is female and the additional object corresponding to the reference information is the gift object "bear", motion data MF1 for character-related images may be automatically applied; if the character object is male and the additional object corresponding to the reference information is the gift object "bear", motion data MF2 for character-related images may be automatically applied; if the character object is female and the additional object corresponding to the reference information is the gift object "glow stick", motion data MP1 for character-related images may be automatically applied; and if the character object is male and the additional object corresponding to the reference information is the gift object "glow stick", motion data MP2 for character-related images may be automatically applied.

[0284] Furthermore, when the reference information concerns a gift object among the additional objects, motion data for character-related images is not necessarily automatically applied to the character object. For example, motion data selected by the user from among several types of motion data for character-related images may be applied to the character object.

[0285] An example of a generation process for generating a new character-related image when the reference information is information about owned objects will be described. First, the generation unit 271 extracts the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the used object from the user table T1. Next, the generation unit 271 reads the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the used object from the terminal storage device 22. Next, the generation unit 271 reads the motion data and object data for the character-related image from the terminal storage device 22. Next, the generation unit 271 generates a new character-related image that includes a character object with a predetermined movement and expression, object data such as background and text, and owned objects corresponding to the reference information, based on the read model data, the image information and placement location of the used object, the motion data and object data for the character-related image, and the reference information. Furthermore, if either object data 1302a or 1302b for character-related images has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 generates a new character-related image using the selected object data. This enables the information processing system 1 to provide character-related images using user information related to the character object.

[0286] In this case, the newly generated character-related image will include the character object holding the possessing object. In this case, motion data for the character-related image may be applied to the character object, such that the pose is appropriate to the character object's attributes and / or the type of the gift object, similar to how the reference information is about the gift object among the additional objects.

[0287] If the determined reference information is a profile image, the generation unit 271 uses the profile image as a new character-related image. If the determined reference information is image information representing a virtual space, the generation unit 271 uses the image information representing the virtual space as a new character-related image. Furthermore, if the determined reference information is image information representing a game space, the generation unit 271 uses the image information representing the game space as a new character-related image. This enables the information processing system 1 to provide character-related images using various information related to character objects.

[0288] The generation unit 271 then associates a related image ID to uniquely identify the newly generated character-related image and stores it in the terminal storage device 22. The generation unit 271 also generates candidate information indicating the newly generated character-related image, associates a related image ID to identify the new character-related image with the generated candidate information and stores it in the terminal storage device 22, thus ending the new character-related image generation process.

[0289] If an additional object (attached object) stored as reference information in the reference information table T4 is a specific attached object, the generation unit 271 may automatically apply that specific attached object in the generation process of all character-related images. The automatic application process of a specific attached object may be limited to the application period from the time the specific attached object is stored as reference information. The automatic application process of a specific attached object may also be limited to the number of times the character-related image generated with the specific attached object applied has been used, or less than a predetermined upper limit. For example, if a distribution user uses a character-related image generated with a specific attached object applied 10 times (an example of the upper limit), the automatic application process of the specific attached object will not be executed thereafter, and the generation unit 271 will perform a generation process that does not automatically apply that specific attached object to all character-related images.

[0290] When a specific attachment object is stored as reference information, the generation process that applies that specific attachment object to already generated character-related images may be automatically re-executed. Furthermore, character-related images to which a specific attachment object has been applied may be usable in the information exchange service only for a predetermined usage period. For example, after the predetermined usage period has elapsed, the generation process that does not apply that specific attachment object to already generated character-related images may be automatically re-executed. Alternatively, after the predetermined usage period has elapsed, the candidate information for character-related images to which a specific attachment object has been applied may be deleted from the candidate information 1403 described later, thereby rendering the character-related images to which the specific attachment object has been applied unusable.

[0291] If an additional object (attached object) stored as reference information is a specific attached object, the generation unit 271 does not need to automatically apply that specific attached object in the generation process of some character-related images. For example, if the application of a specific attached object would cause the display position of that specific attached object to overlap with the display position of the object data for the character-related image used in the character-related image (data for objects representing background, text, etc.), the specific attached object does not need to be applied to the character-related image. Also, if the application of a specific attached object would cause the display position of that specific attached object to overlap with the display position of other attached objects that have already been attached to the character-related image, the specific attached object does not need to be applied to the character-related image.

[0292] The exit button 1304 is an operation object used to terminate the display of the reference information selection screen 1300. When the user selects the exit button 1304 through an operation of the input device 24, the display processing unit 274 of the terminal device 2 terminates the display of the reference information selection screen 1300.

[0293] Figure 16 shows an example of an information exchange screen 1400 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 1400 is displayed when a new character-related image is generated by the additional generation process.

[0294] On the information exchange screen 1400, similar to the information exchange screen 1100, there are other user input information 1401, other user information 1402, text input object 1003, candidate display button 1004, user input information 1404, and additional generation button 1102. The candidate information 1403 on the information exchange screen 1400 includes the four types of candidate information 1101 on the information exchange screen 1100 and candidate information corresponding to two new character-related images. When the candidate information 1403 includes candidate information corresponding to two new character-related images, one or more types of the previously included four types of candidate information 1101 may be deleted from the candidate information 1403. For example, by deleting three or more types of candidate information, the options for candidate information will decrease, making it possible to facilitate the user's selection of information.

[0295] When, in response to the user's operation of the input device 24, one piece of candidate information is selected from the six types of candidate information 1403, the input device 24 sends the related image ID associated with the selected candidate information to the terminal processing device 27. Next, the display processing unit 274 extracts the character-related image identified by the related image ID received from the input device 24 from the terminal storage device 22, and displays the extracted character-related image in the user display area of the information exchange screen 1400 as the user input information 1404 (Figure 16).

[0296] Along with the display of the user input information 1404 by the display processing unit 274, the transmission unit 272 sends the related image ID received from the input device 24 and the user ID stored in the terminal storage device 22 to the server device 3 via the terminal communication I / F 21. The server transmission unit 332 of the server device 3 extracts the user IDs of other users associated with the same information exchange group as the user ID received by the server reception unit 331 from the server storage device 32. Also, the server transmission unit 332 sends the related image ID and the user ID received by the server reception unit 331 to the terminal device 2 of the other user identified by the extracted user ID via the server communication I / F 31.

[0297] Then, the receiving unit 273 of the other user's terminal device 2 receives the associated image ID and user ID transmitted from the server device 3 via the terminal communication I / F 21. Next, the display processing unit 274 of the other user's terminal device 2 extracts the character-related image identified by the associated image ID received by the receiving unit 273 from the terminal storage device 22, and displays the extracted character-related image as other user input information in the other user display area of ​​the distribution screen. In this way, the same character-related image used by the user is also displayed on the other user's terminal device 2 associated with the information exchange group to which the user belongs. If the character-related image identified by the received associated image ID is not stored in the terminal storage device 22, the transmitting unit 272 of the other user's terminal device 2 sends an image transmission request to the server device 3. The image transmission request is sent along with the received associated image ID and / or user ID, as well as the other user's user ID. When the server transmission unit 332 of the server device 3 receives an image transmission request from the server reception unit 331, it extracts the character-related image associated with the associated image ID received along with the image transmission request from the server storage device 32. Next, the server transmission unit 332 transmits the extracted character-related image along with the received associated image ID and user ID to the other user's terminal device 2. The reception unit 273 of the other user's terminal device 2 receives the associated image ID, character-related image, and user ID transmitted from the server device 3 via the terminal communication interface 21, associates them with each other, and stores them in the terminal storage device 22. Then, the display processing unit 274 of the other user's terminal device 2 displays the stored character-related image as other user input information in the other user display area of ​​the distribution screen.

[0298] (Operation sequence of Information Processing System 1) Figure 17 shows an example of the operation sequence of the information processing system 1. This operation sequence is executed mainly by the terminal processing unit 27 and the server processing unit 33 in cooperation with each element of the two terminal devices 2 and the server device 3, based on control programs that are pre-stored in the terminal storage device 22 and the server storage device 32, respectively. Hereinafter, of the two terminal devices 2, the distribution terminal device 2a will be referred to as the terminal device 2 operated by the distribution user, and the viewing terminal device 2b will be referred to as the terminal device 2 operated by the viewing user.

[0299] First, the transmission unit 272 of the distribution terminal device 2a transmits the character video data generated by the generation unit 271 and the distribution user ID of the distribution user to the server device 3 via the terminal communication interface 21 (step S101). The transmission unit 272 may also transmit the distribution user's voice data to the server device 3 together with the character video data. From the start to the end of the video distribution service, the process in step S101 is executed continuously at predetermined time intervals (for example, every 2 seconds). Then, in response to the execution of the process in step S101, the processes related to steps S102 to S104 are executed intermittently.

[0300] The server transmission unit 332 of the server device 3 distributes the received character video data to the viewing terminal device 2b via the server communication interface 31 each time the character video data is received by the server reception unit 331 (step S102). The display processing unit 274 of the distribution terminal device 2a generates drawing data for displaying the distributed image based on the character video data generated by the generation unit 271, and displays the distributed screen on the display device 23 based on the generated drawing data (step S103). The display processing unit 274 of the viewing terminal device 2b generates drawing data for displaying the distributed image based on the character video data received by the reception unit 273, and displays the distributed screen on the display device 23 based on the generated drawing data (step S104).

[0301] Next, the transmission unit 272 of the viewing terminal device 2b transmits information regarding display instructions for displaying response information on the distribution screen, and / or information indicating additional instructions for displaying additional objects on the distribution screen, via the terminal communication interface 21 to the server device 3 in response to the operation of the input device 24 by the viewing user (step S105). Whenever the server receiving unit 331 receives information regarding various instructions (display instructions, additional instructions, etc.), the server transmission unit 332 of the server device 3 distributes the received information regarding various instructions to the distribution terminal device 2a via the server communication interface 31 (step S106). The display processing unit 274 of the viewing terminal device 2b generates drawing data for displaying response information and / or additional objects based on the various instructions input in response to the operation of the input device 24 by the viewing user, and displays the distribution screen on the display device 23 based on the generated drawing data (step S107). The display processing unit 274 of the distribution terminal device 2a generates drawing data for displaying response information and / or additional objects based on information regarding various instructions received by the receiving unit 273, and displays the distribution screen on the display device 23 based on the generated drawing data (step S108).

[0302] Next, the display processing unit 274 of the distribution terminal device 2a executes the reference information acquisition process (step S109). Details of the reference information acquisition process will be described later. When acquiring reference information transmitted from the viewing terminal device 2b, the reference information acquisition process may also be executed by the receiving unit 273 of the distribution terminal device 2a. The reference information acquisition process is not limited to processes that can only be executed by the distribution terminal device 2a. For example, the display processing unit 274 or the receiving unit 273 of the viewing terminal device 2b, which has a character object representing a user available, may execute the candidate information display process.

[0303] Next, the display processing unit 274 of the distribution terminal device 2a executes the candidate information display process (step S110). Details of the candidate information display process will be described later. Note that the candidate information display process is not limited to processes that can only be executed by the distribution terminal device 2a. For example, the display processing unit 274 of the viewing terminal device 2b, which has a character object representing a user available, may also execute the candidate information display process.

[0304] Next, the generation unit 271 and display processing unit 274 of the distribution terminal device 2a execute additional generation processing (step S111). Details of the additional generation processing will be described later. Additional generation processing is not limited to processing that can only be executed by the distribution terminal device 2a. For example, the generation unit 271 and display processing unit 274 of the viewing terminal device 2b, which can use character objects representing users, may execute candidate information display processing.

[0305] Next, the transmission unit 272 of the distribution terminal device 2a transmits the input information received in response to the operation of the input device 24 by the distribution user to the server device 3 via the terminal communication interface 21 (step S112). Next, the server transmission unit 332 of the server device 3 distributes the received input information to the viewing terminal device 2b via the server communication interface 31 each time input information is received by the server reception unit 331 (step S113). The display processing unit 274 of the distribution terminal device 2a generates drawing data for displaying the input information received in response to the operation of the input device 24 by the distribution user, and displays an information exchange screen on the display device 23 based on the generated drawing data (step S114). The display processing unit 274 of the viewing terminal device 2b generates drawing data for displaying the input information received by the reception unit 273, and displays an information exchange screen on the display device 23 based on the generated drawing data (step S115).

[0306] Figure 18(a) shows an example of the operation flow of the reference information acquisition process. The candidate information display process shown in Figure 18(a) is executed in step S109 of Figure 17.

[0307] First, the display processing unit 274 or the receiving unit 273 determines whether or not an instruction to acquire reference information has been received (step S201). If an instruction to acquire reference information has not been received (step S201-No), the display processing unit 274 or the receiving unit 273 terminates the reference information acquisition process. If an instruction to acquire reference information has been received (step S201-Yes), the display processing unit 274 or the receiving unit 273 acquires the reference information (step S202), stores the acquired reference information in the terminal storage device 22 (step S203), and terminates the reference information acquisition process.

[0308] Figure 18(b) shows an example of the operation flow of the candidate information display process. The candidate information display process shown in Figure 18(b) is executed in step S110 of Figure 17.

[0309] First, the display processing unit 274 determines whether or not an instruction to display candidate information has been received (step S301). If an instruction to display candidate information has not been received (step S301-No), the process returns to step S301 (waits until an instruction to display candidate information is received).

[0310] If a display instruction for candidate information is received (step S301-Yes), the display processing unit 274 determines whether or not a character-related image has been previously generated and stored in the terminal storage device 22 (step S302). If no character-related image is stored (step S302-No), the generation unit 271 generates a character-related image (step S303) and proceeds to step S304.

[0311] If a character-related image has been generated in the past and stored in the terminal storage device 22 (step S302-Yes), the display processing unit 274 generates candidate information corresponding to the character-related image, the generation unit 271 generates candidate information, the display processing unit 274 displays the generated candidate information (step S304), and the candidate information display process ends.

[0312] Figure 19 shows an example of the operation flow of the additional generation process. The additional generation process shown in Figure 19 is executed in step S111 of Figure 17.

[0313] First, the display processing unit 274 determines whether or not reference information is stored in the reference information table T4 (step S401). If reference information is not stored (step S401-No), the display processing unit 274 terminates the addition generation process. If reference information is stored (step S401-Yes), the display processing unit 274 displays the reference information selection screen 1300 (step S402).

[0314] Next, the display processing unit 274 determines whether or not an additional generation instruction has been input by the input device 24 (step S403). If no additional generation instruction has been input (step S403-No), the display processing unit 274 terminates the additional generation process. If an additional generation instruction has been input (step S403-Yes), the generation unit 271 determines the reference information (step S404).

[0315] Next, the generation unit 271 generates a new character-related image based on the determined reference information (step S405). Then, the display processing unit 274 displays the newly generated character-related image (step S406), and the additional generation process ends.

[0316] As detailed above, the information processing system 1 of this embodiment can increase the types of character-related images that become available based on information about the distribution user or information about a character object representing the distribution user. In this way, the information processing system 1 of this embodiment makes it possible to improve the user interface for determining information to be provided to other users without requiring the user to perform complicated operations. Furthermore, the information processing system 1 of this embodiment can change the types of character-related images based on information about the distribution user or information about a character object representing the distribution user. In this way, the information processing system 1 of this embodiment makes it possible to change the information provided to other users by the user without requiring the user to perform complicated operations, thereby keeping the user's choices interesting.

[0317] (Variation 1) It should be noted that the present invention is not limited to this embodiment. For example, in the information exchange screen, stamp images may be classified and displayed by type. For example, stamp images may include stamp images for conveying "OK," stamp images for conveying "NO," stamp images with a "smiling" expression, etc. Furthermore, stamp images may include stamp images generated as character-related images, stamp images generated as new character-related images, etc.

[0318] Figure 20 shows an example of information exchange screens 1500 and 1510, which include stamp images generated as character-related images. Information exchange screens 1500 and 1510 include type selection buttons 1501 for selecting the type of stamp image. When candidate information corresponding to any of the type selection buttons 1501 is selected in response to the user's operation of the input device 24, the stamp images of the type corresponding to the selected type selection button 1501 are displayed in the display area below the type selection button 1501. In Figure 20, buttons are shown as an example widget for selecting the type of stamp image, but tabs may also be used as the widget for selecting the type of stamp image. In this way, in information exchange screens 1500 and 1510, multiple stamp images are displayed by switching between their types.

[0319] Figure 20(a) shows an example of the information exchange screen 1500 when the type selection button 1501 corresponding to the stamp image generated as a character-related image is selected. On the information exchange screen 1500, candidate information 1101 corresponding to the stamp image generated as a character-related image is displayed. If the type selection button 1501 corresponding to the stamp image for conveying "OK" is selected, candidate information corresponding to the stamp image for conveying "OK" is displayed on the information exchange screen 1500.

[0320] Figure 20(b) shows an example of the information exchange screen 1510 when the type selection button 1501 corresponding to the stamp image generated as a new character-related image is selected. In the information exchange screen 1510, the type selection button 1501 corresponding to the stamp image generated as a new character-related image is selected, and candidate information 1511 corresponding to the stamp image generated as a new character-related image is displayed according to this selection. In the example shown in Figure 20(b), the type selection button 1501 corresponding to the stamp image generated as a new character-related image is a button object labeled "Add".

[0321] In this way, by configuring the information exchange screen to display stamp images categorized by type, it is possible to improve the user interface. With such an information exchange screen, even if the size of the display screen of the terminal device 2 is limited, it is possible to improve the ability to view multiple stamp images at a glance. Furthermore, with such information exchange screens 1500 and 1510, users can intuitively and / or easily find the stamp image they wish to use. In addition, with such information exchange screens 1500 and 1510, users can quickly grasp whether or not there are stamp images generated as new character-related images.

[0322] (Modification 2) At least a portion of the functions implemented by the terminal processing unit 27 of terminal device 2 may be implemented by a processor other than the terminal processing unit 27 of terminal device 2. At least a portion of the functions implemented by the terminal processing unit 27 may be implemented, for example, by the server processing unit 33 of server device 3. Specifically, some of the functions of the generation unit 271 and the display processing unit 274 may be executed in server device 3. For example, terminal device 2 transmits continuously acquired image data and audio data to server device 3 along with the user ID of the user operating terminal device 2. Server device 3 then uses the functions of the generation unit 271 and the display processing unit 274 to generate character video data and generate display information for displaying a communication screen including the distributed images of all users on the display device 23. Server device 3 then transmits the display information along with audio information (audio data) to each terminal device 2 of all users. Each terminal device 2 outputs the information received from server device 3 (so-called "server rendering").

[0323] (Variation 3) Terminal device 2 has the function of displaying communication screens, etc., by executing various commands contained in the control program stored in terminal device 2. In order to realize information exchange services and / or video distribution services, terminal device 2 may call a browser function embedded in a web application, and using this browser function, receive a web page (a document written in a markup language such as HTML (Hyper Text Markup Language)) from the server device and execute the control program embedded in this web page (so-called "browser rendering"). For example, by using HTML 5 as the markup language, terminal device 2 can easily execute new information processing. By adopting such a web application to realize information exchange services and / or video distribution services in terminal device 2, the program creator can provide new information processing to the client side (terminal device side) simply by embedding the new program in the web page transmitted by the server device, and the man-hours required to create a new program can be greatly reduced. Furthermore, on the client side, new services can be provided simply by receiving a web page without downloading a new control program, which reduces the load on the communication network, communication costs and / or communication time compared to downloading a control program, and also simplifies the user interface.

[0324] (Modification 4) The generation unit 271 of User A's terminal device 2 may generate output information including face motion data without using image data. For example, the terminal storage device 22 of the terminal device 2 stores face motion data corresponding to voice data, and the generation unit 271 may extract the face motion data corresponding to the acquired voice data of User A from the terminal storage device 22 and generate output information including the extracted face motion data, the voice data, and User A's user ID. Alternatively, the generation unit 271 of User A's terminal device 2 may generate output information including User A's voice data output from the microphone 26 and the user ID stored in the terminal storage device 22, without including face motion data. In this case, the display processing unit 274 of User B's terminal device 2 may extract the face motion data corresponding to User A's voice data included in the output information of User A received via the server device 3 and generate an animation for User A's character object. Note that the process of generating face motion data corresponding to the user's voice data may use a known lip-sync algorithm.

[0325] For example, if user A's terminal device 2 uses an HMD as a display device 23, it becomes possible to communicate with user B via a character object without acquiring user A's image data.

[0326] (Variation 5) The distributed images 602, 612, and 812 displayed on each terminal device 2 are not limited to moving or still images containing animations of three-dimensional or two-dimensional character objects generated based on the actions of the distribution user. Distributed images 602, 612, and 812 may include animations of character objects generated in response to the actions of the distribution user, and may also include moving or still images of the distribution user themselves. However, during the video distribution service, each terminal device 2 does not have to display these character object animations or moving or still images of the distribution user themselves. In this case, each terminal device 2 may output only the voice of the distribution user from its speaker.

[0327] Thus, the "distribution user" disclosed in this embodiment is not limited to a user who uses at least the function of a terminal device to transmit character video data to a server device, but may also be a user who transmits information and / or audio data for displaying various distribution images. The information for displaying various distribution images may include, for example, character video data, video data of the distribution user themselves, etc.

[0328] (Experimental variation 6) Furthermore, the “distributing user” disclosed in this embodiment is not limited to a user who hosts or organizes a video distribution service on their own. The “distributing user” may also be a user who hosts or organizes a collaborative distribution service that multiple people can participate in, or a video chat service or voice chat service that multiple people can participate in and / or view. Alternatively, the “distributing user” may also be a user who hosts or organizes an event (such as a party) service in a virtual space that multiple people can participate in and / or view. Thus, the “distributing user” may also be a user who primarily runs these services, and may be referred to as a host user, organizer user, or event organizer user.

[0329] The "viewing user" disclosed in this embodiment is not limited to a user who uses at least the function of a terminal device that receives character video data from a distribution terminal device and displays the distribution image of the distribution user. The "viewing user" may also be a user who receives information and / or audio data for displaying various distribution images.

[0330] A "viewing user" may be not only a user who receives information and / or audio data for displaying various streamed images, but also a user who can react. For example, a "viewing user" may be a user who watches a video streaming service or a collaborative streaming service, or a user who participates in a video chat service, a voice chat service, or an event service, and / or a user who watches these services. Therefore, "viewing user" in this disclosure may be rephrased as a guest user, a participating user, a listener, a viewing user, or a supporter user.

[0331] (Example 7) Furthermore, the information processing system 1 in the embodiments of this disclosure may provide the following internet space (metaverse), which is a digital world in which multiple people can simultaneously participate in activities such as interaction, work, and / or play at a level close to that of real society, through character objects (e.g., avatars, etc.), enabling social activities that transcend the gap between reality and virtuality. In such a space, users' avatars can freely walk around the world and communicate with each other.

[0332] Information processing system 1 may distribute a distributed image that includes one avatar (character object) from among multiple avatars in such a space as the character object of the distribution user. In other words, one-to-many video distribution may be performed in a many-to-many metaverse space. In such a space, there may be no particular distinction between the distribution user and the viewing user.

[0333] Although this embodiment and its variations have been described in detail above, the present invention is not limited to any particular embodiment. Furthermore, various changes, substitutions, and modifications can be made to the present invention without departing from its scope. [Additional Notes] [1] An information processing system comprising one or more computer processors, The one or more computer processors described above Information regarding the display of a video containing a first object representing the first user is transmitted to the second terminal device of the second user. To transmit to the second terminal device information regarding the display of the associated image selected by the first user from a group containing multiple associated images relating to the first object, Execute, The one or more computer processors described above Obtain reference information relating to the first object, To include the image containing the first object, generated based on the aforementioned reference information, as the associated image in the group. An information processing system characterized by performing the following. [2] The one or more computer processors To include an image that includes the first object and is generated without regard to the reference information as the associated image in the group. An information processing system according to claim 1, which performs the following: [3] In obtaining the reference information, one or more computer processors To obtain the information related to the displayed video as the reference information, An information processing system according to claim 1 or 2, which performs the following: [4] The information processing system according to claim 3, wherein the information relating to the video obtained as reference information includes motion data for displaying the associated image which includes the first object in the same pose as the first object in the pose at a predetermined timing included in the displayed video. [5] The information processing system according to claim 3, wherein the information relating to the video obtained as reference information includes captured image information of the displayed video. [6] The one or more computer processors To include the text information entered by the second user in the video, Execute, The information processing system according to claim 3, wherein the information relating to the video obtained as reference information includes the text information. [7] The information processing system according to claim 3, wherein the information relating to the video obtained as reference information includes information relating to the second object selected by the first user from among the second objects which can be selected to be displayed together with the first object in the video. [8] The one or more computer processors The video includes an image showing the third object selected by the second user. Execute, In obtaining the aforementioned reference information, one or more computer processors To obtain the image information representing the third object as the reference information, An information processing system according to claim 1, which performs the following: [9] The one or more computer processors To store information of an image representing an object owned by the first user, Execute, In obtaining the aforementioned reference information, one or more computer processors To obtain the image information representing the aforementioned owned object as the reference information, An information processing system according to claim 1, which performs the following:

[10] The one or more computer processors To store an image showing the profile of the first object, Execute, In obtaining the aforementioned reference information, one or more computer processors To obtain the image information representing the profile as the reference information, An information processing system according to claim 1, which performs the following:

[11] The one or more computer processors Displaying the first object that exists in the virtual space, Execute, In obtaining the aforementioned reference information, one or more computer processors To acquire predetermined image information representing the virtual space as the reference information, An information processing system according to claim 1, which performs the following:

[12] The information processing system according to claim 1, wherein the rendering processing load for rendering the video onto the display screen is greater than the rendering and display load for rendering the associated image onto the display screen.

[13] The information processing system according to claim 1, wherein the video is a three-dimensional moving image in which a three-dimensional space including the first object is drawn, and the associated image is a two-dimensional image.

[14] The information processing system according to claim 1, wherein the display size of the video on the display screen is larger than the display size of the associated image on the display screen.

[15] The information processing system according to claim 1, wherein the amount of data transmitted when transmitting display information for displaying the video is greater than the amount of data transmitted when transmitting information relating to the display of the associated image.

[16] A control method for an information processing system comprising one or more computer processors, The one or more computer processors described above Information regarding the display of a video containing a first object representing the first user is transmitted to the second terminal device of the second user. To transmit to the second terminal device information regarding the display of the associated image selected by the first user from a group containing multiple associated images relating to the first object, Includes, The one or more computer processors described above Obtain reference information relating to the first object, To include the image containing the first object, generated based on the aforementioned reference information, as the associated image in the group. A control method including

[17] A server device that communicates with the first terminal device of the first user and the second terminal device of the second user, The system receives information regarding the display of a video, which includes a first object representing the first user, transmitted from the first terminal device. The first terminal device receives information regarding the display of a related image selected by the first user from a group containing multiple related images relating to the first object, A processing unit transmits to the second terminal device information regarding the display of the video and the display of the related images received from the first terminal device. Equipped with, An image containing the first object, generated based on reference information relating to the first object, is included in the group as the associated image. A server device characterized by the following features. [Explanation of Symbols]

[0334] 1. Information Processing System 2 Terminal devices 21 Terminal Communication Interface 22 Terminal Storage Devices 23 Display device 24 Input devices 25 Imaging device 26 Mike 27 Terminal Processing Unit 271 Generation part 272 Transmitter 273 Receiving Unit 274 Display Processing Unit 275 Audio output section 3 Server equipment 31 Server Communication Interface 32 Server Storage Devices 33 Server Processing Units 331 Server receiving unit 332 Server Transmission Unit 4 base station 5. Mobile communication network 6 Gateways 7. Internet

Claims

1. An information processing system comprising one or more computer processors, The one or more computer processors described above Store an image showing the profile of the first object representing the first user. Information regarding the display of the video including the first object is transmitted to the second terminal device of the second user. The video includes text information entered by the second user or an image representing a third object selected by the second user. Information regarding the display of the associated image selected by the first user from a group containing multiple associated images relating to the first object is transmitted to the second terminal device and / or the third terminal device of the third user. Information relating to the aforementioned video, wherein information of the image showing the profile is obtained as reference information, To include the image containing the first object, generated based on the aforementioned reference information, as the associated image in the group. Execute An information processing system characterized by the following:

2. The one or more computer processors described above The information processing system according to claim 1, characterized in that it acquires reference information relating to the first object when the response information in the video satisfies predetermined conditions.

3. The information processing system according to claim 2, characterized in that the predetermined condition is a condition relating to the number of response information.

4. If an image showing the third object is included in the video, the video will display an image showing the third object associated with a predetermined part of the first object. The one or more computer processors described above If predetermined conditions are met, an image including the first object, generated based on the reference information including an image showing the third object, is included in the group as the associated image. If the predetermined conditions are not met, an image including the first object, generated based on the reference information which does not include an image showing the third object, is included in the group as the associated image. The information processing system according to claim 1, which performs the following.

5. The information processing system according to claim 1, wherein the information relating to the video obtained as reference information includes motion data for displaying the associated image which includes the first object in the same pose as the first object in the pose at a predetermined timing included in the displayed video.

6. The information processing system according to claim 1, wherein the information relating to the video obtained as reference information includes captured image information of the displayed video.

7. The one or more computer processors described above Displaying the first object that exists in the virtual space, Execute, The information processing system according to claim 1, wherein the reference information includes predetermined image information indicating the virtual space.

8. The information processing system according to claim 1, wherein the rendering processing load for rendering the aforementioned video onto the display screen is greater than the rendering and display load for rendering the aforementioned related image onto the display screen.

9. The information processing system according to claim 1, wherein the video is a three-dimensional moving image in which a three-dimensional space including the first object is depicted, and the associated image is a two-dimensional image.

10. The information processing system according to claim 1, wherein the display size of the video on the display screen is larger than the display size of the related image on the display screen.

11. The information processing system according to claim 1, wherein the amount of data transmitted when transmitting display information for displaying the aforementioned video is greater than the amount of data transmitted when transmitting information regarding the display of the aforementioned related image.

12. A control method for an information processing system comprising one or more computer processors, The one or more computer processors described above Store an image showing the profile of the first object representing the first user. Information regarding the display of the video including the first object is transmitted to the second terminal device of the second user. The video includes text information entered by the second user or an image representing a third object selected by the second user. Information regarding the display of the associated image selected by the first user from a group containing multiple associated images relating to the first object is transmitted to the second terminal device and / or the third terminal device of the third user. Information relating to the aforementioned video, wherein information of the image showing the profile is obtained as reference information, To include the image containing the first object, generated based on the aforementioned reference information, as the associated image in the group. including A control method characterized by the following.

13. A server device that communicates with the first terminal device of the first user and the second terminal device of the second user, The first terminal device receives information regarding the display of a video including a first object representing the first user, Store an image showing the profile of the first object, The first terminal device receives information regarding the display of the associated image selected by the first user from a group containing multiple associated images relating to the first object, A processing unit transmits information regarding the display of the video and the display of the related images received from the first terminal device to the second terminal device and / or to the third terminal device of the third user. Equipped with, The video includes text information entered by the second user or an image showing a third object selected by the second user. The group includes, as the associated image, an image containing the first object, which is generated based on reference information obtained as information of the image representing the profile, and which is related to the video and is included in the group as the associated image. A server device characterized by the following features.

Images