Information processing system, control method, and server device

Abstract

To provide an information processing system, control method and server device that can improve a user interface for determining information to be provided to other users, without requiring the user to perform complicated operations.SOLUTION: An information processing system comprises one or more computer processors. The one or more computer processors are configured to: transmit, to a second terminal device of a second user, information regarding display of a video including a first object representing a first user; transmit, to the second terminal device, information regarding display of a related image selected by the first user from a group including multiple related images regarding the first object; acquire reference information regarding the first object; and include, in the group, an image including the first object and generated based on the reference information as a related image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The embodiments disclosed in this specification relate to an information processing system, a control method, and a server device. [Background technology]

[0002] Conventionally, instant messengers have been known that realize a function of transmitting input information, such as text information or image information, input by a user operating a terminal device to a terminal device of another user. In recent instant messengers, image information using a character object, such as an avatar, corresponding to the user can be transmitted as input information from the terminal device of the user to the terminal device of the other user.

[0003] For example, Patent Document 1 describes an information processing system in which avatar stamps are used as an example of image information that can be transmitted from a user's terminal device to another user's terminal device. In the information processing system described in Patent Document 1, a user can use image information (such as a self-made avatar stamp) corresponding to an object such as an avatar that the user has created and registered. In this way, in the information processing system described in Patent Document 1, a user can communicate with other users using an image corresponding to an object that represents the user, thereby making it possible for the user to appeal to other users about the object that corresponds to the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-91363 Summary of the Invention [Problem to be solved by the invention]

[0005] In the information processing system described in Patent Document 1, in order to enrich the number of information candidates corresponding to the user, the user himself / herself needs to perform a cumbersome operation of generating information, which causes a problem that it is not possible to encourage users to use the information processing system. Also, in the information processing system described in Patent Document 1, when there are many information candidates corresponding to the user, the operation of selecting information that the user wants to provide to other users becomes cumbersome for the user, which causes a problem that it is not possible to encourage users to use the information processing system.

[0006] The information processing system, control method, and server device disclosed in this specification enable an improved user interface for determining information to be provided to other users without requiring the user to perform complicated operations. [Means for solving the problem]

[0007] The disclosed information processing system is an information processing system having one or more computer processors, wherein the one or more computer processors perform the following: transmitting information regarding the display of a video including a first object representing a first user to a second terminal device of a second user; transmitting information regarding the display of a related image selected by the first user from a group including multiple related images related to the first object to the second terminal device; and the one or more computer processors obtain reference information regarding the first object; and including an image including the first object, generated based on the reference information, in the group as a related image.

[0008] In addition, in the disclosed information processing system, it is preferable that one or more computer processors execute the following: include, in a group as related images, images that include the first object and that are generated without reference information.

[0009] In addition, in the disclosed information processing system, it is preferable that, in acquiring the reference information, one or more computer processors acquire information relating to the displayed moving image as the reference information.

[0010] In addition, in the disclosed information processing system, it is preferable that the information regarding the video acquired as reference information includes motion data for displaying a related image including a first object in a pose identical to the pose of the first object included in the displayed video at a predetermined timing.

[0011] In addition, in the disclosed information processing system, the information relating to the moving image acquired as the reference information preferably includes captured image information of the displayed moving image.

[0012] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors include text information input by a second user in the video, and that the information regarding the video obtained as reference information includes the text information.

[0013] In addition, in the disclosed information processing system, it is preferable that the information regarding the video acquired as reference information includes information regarding a second object selected by the first user from among second objects that can be selected to be displayed together with the first object in the video.

[0014] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors include an image showing a third object selected by the second user in the video, and that in obtaining the reference information, one or more computer processors obtain information about the image showing the third object as the reference information.

[0015] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors store image information indicating owned objects held by the first user, and that in acquiring the reference information, one or more computer processors acquire image information indicating the owned objects as reference information.

[0016] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors store an image showing a profile of the first object, and that in acquiring the reference information, one or more computer processors acquire information of the image showing the profile as the reference information.

[0017] Furthermore, in the disclosed information processing system, it is preferable that one or more computer processors execute the process of displaying a first object existing in the virtual space, and that in acquiring the reference information, one or more computer processors execute the process of acquiring predetermined image information showing the inside of the virtual space as the reference information.

[0018] In the disclosed information processing system, it is preferable that the rendering processing load for rendering the video on the display screen is greater than the rendering display load for rendering the related image on the display screen.

[0019] In the disclosed information processing system, the video is preferably a three-dimensional video image in which a three-dimensional space including the first object is depicted, and the related image is preferably a two-dimensional image.

[0020] In the disclosed information processing system, the display size of the moving image on the display screen is preferably larger than the display size of the related image on the display screen.

[0021] In the disclosed information processing system, the transmission amount of the display information for displaying the moving image is preferably larger than the transmission amount of the information related to the display of the related image.

[0022] The disclosed control method is a control method for an information processing system having one or more computer processors, and includes the one or more computer processors transmitting information regarding the display of a video including a first object representing a first user to a second terminal device of a second user, and transmitting information regarding the display of a related image selected by the first user from a group including multiple related images related to the first object to the second terminal device, and the one or more computer processors obtaining reference information regarding the first object, and including an image including the first object generated based on the reference information in the group as a related image.

[0023] The disclosed server device is a server device that communicates with a first terminal device of a first user and a second terminal device of a second user, and is equipped with a processing unit that receives information related to the display of a video including a first object representing the first user, transmitted from the first terminal device, receives information related to the display of a related image selected by the first user from a group including a plurality of related images related to the first object, transmitted from the first terminal device, and transmits the information related to the display of the video and the information related to the display of the related images received from the first terminal device to the second terminal device, and an image including the first object generated based on reference information related to the first object is included in the group as a related image. [Effects of the Invention]

[0024] The disclosed information processing system, control method, and server device make it possible to improve the user interface for determining information to be provided to other users without requiring the user to perform complicated operations. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an overview of an information processing system. [Figure 2] FIG. 1 is a schematic diagram illustrating an example of an overview of an information processing system. [Figure 3] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing system. [Figure 4] FIG. 2 is a diagram illustrating an example of a schematic configuration of a terminal device. [Figure 5] 3A to 3C are diagrams illustrating examples of data structures of various tables. [Figure 6] FIG. 2 illustrates an example of a schematic configuration of a server device. [Figure 7] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. [Figure 8] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. [Figure 9] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. [Figure 10] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. [Figure 11] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. [Figure 12] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. [Figure 13] FIG. 10 is a diagram illustrating an example of a screen displayed on a terminal device. [Figure 14] FIG. 1A is a diagram showing an example of a screen displayed on a terminal device, and FIG. 1B is a diagram showing an example of the data structure of a reference information table. [Figure 15] FIG. 10 is a diagram illustrating an example of a screen displayed on a terminal device. [Figure 16] FIG. 10 is a diagram illustrating an example of a screen displayed on a terminal device. [Figure 17] FIG. 10 is a diagram illustrating an example of an operation sequence of the information processing system. [Figure 18] FIG. 10A is a diagram showing an example of the operation flow of a reference information acquisition process, and FIG. 10B is a diagram showing an example of the operation flow of a candidate information display process. [Figure 19] FIG. 10 is a diagram illustrating an example of an operation flow of an additional generation process. [Figure 20] 3A to 3C are diagrams illustrating examples of various screens displayed on a terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0026] Various embodiments of the present invention will be described below with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions set forth in the claims and their equivalents.

[0027] (Outline of information processing system) 1 and 2 are schematic diagrams illustrating an example of an overview of an information processing system. The information processing system includes a terminal device operated by each of a plurality of users and a server device. For example, the terminal device is an information processing device such as a multi-function mobile phone (a so-called "smartphone") owned by the user. For example, the server device is a computer for providing a video distribution service and an information exchange service to the terminal device via a communication network. The server device may provide services (online games, etc.) other than the video distribution service and the information exchange service to the terminal device.

[0028] A video distribution service is a service in which an image showing a user operating a specific terminal device (such as a distribution image including a character object showing the user, as described below) is distributed from the specific terminal device to one or more other users' terminal devices. An information exchange service is a service in which various information such as text information and image information is transmitted from one terminal device to another terminal device between the terminal devices of multiple users running instant messengers, etc. The information exchange service may be a service realized by a so-called chat function.

[0029] A character object representing each user may be used in conjunction with at least two or more of the video distribution service, the information exchange service, and other services. The use of a character object in conjunction with the other services means, for example, that a character object whose appearance has been changed in the video distribution service is also displayed with the changed appearance in the information exchange service. The use of a character object in conjunction with the other services may also mean that a character object whose use is permitted in the information exchange service can also be used in the video distribution service. The use of a character object in conjunction with the other services may also mean that a character object whose use is no longer permitted in the information exchange service can no longer be used in the video distribution service.

[0030] The terminal device stores a control program such as an application program and loads the control program into memory in response to a start operation from a user. The terminal device starts at least one of a video distribution service and an information exchange service in response to a user's input of an instruction and execution of instructions contained in the loaded control program. When the video distribution service is started, the terminal device executes instructions contained in the control program to realize multiple functions related to the video distribution service. When the information exchange service is started, the terminal device executes instructions contained in the control program to realize multiple functions related to the information exchange service. When a user wishes to use various services provided by a server device, the user uses a terminal device capable of communicating with the server device to perform a known login operation related to the various services, thereby starting a session between the server device and the terminal device related to the various services. This allows the user to participate in and use various services.

[0031] When a video distribution service is started, for example, the user's terminal device realizes a generation function that generates character video data including motion data based on various input data entered by the user. For example, the input data is a plurality of image data acquired at predetermined sampling time intervals by an imaging device provided in the user's terminal device, and in this case, each image data shows the user. The character video data is an example of information related to the display of a distribution image including a character object representing the user. The character video data is generated at predetermined time intervals during the video distribution service.

[0032] Furthermore, for example, the user's terminal device displays a distributed image based on the character video data. In addition to displaying the distributed image, the user's terminal device transmits the generated character video data to the terminal device of another user via the server device, thereby realizing an output function of transmitting the generated character video data to the server device at predetermined time intervals. Furthermore, for example, the user's terminal device realizes a function of displaying a distributed image including a specific object based on information indicating predetermined instructions from another user transmitted from the server device at predetermined time intervals.

[0033] Hereinafter, a user who uses at least the function of a terminal device to transmit character video data to a server device may be referred to as a broadcasting user. A terminal device operated by a broadcasting user may be referred to as a broadcasting terminal device. Furthermore, a user who uses at least the function of a terminal device to receive character video data from a broadcasting terminal device and display a broadcast image of the broadcasting user may be referred to as a viewing user. A terminal device operated by a viewing user may be referred to as a viewing terminal device. Note that one terminal device may have the functions of both a broadcasting terminal device and a viewing terminal device.

[0034] When the video distribution service starts, as shown in FIG. 1(a), the distribution terminal device displays a distribution screen including a distribution image related to the broadcast user's distribution event. One or more viewing terminal devices receive character video data from the distribution terminal device via the server device, and display a distribution screen including a distribution image related to the broadcast user's distribution event based on the character video data. The broadcast user's distribution image includes a character object representing the broadcast user. The character object is displayed to move in accordance with the broadcast user's motion data included in the character video data.

[0035] If the broadcast terminal device is equipped with a microphone, the broadcast terminal device can capture voice uttered by the broadcast user and transmit the captured voice data of the broadcast user to one or more viewing terminal devices. In this case, the broadcast terminal device transmits the broadcast user's character video data and voice data to the server device in order to transmit the broadcast user's voice data together with the character video data to one or more viewing terminal devices via the server device. Furthermore, the one or more viewing terminal devices receive the broadcast user's character video data and voice data transmitted to the server device by the broadcast terminal device from the server device.

[0036] When the voice produced by the broadcast user is acquired by the broadcast terminal device, the broadcast user's motion data included in the character video data includes face motion data indicating the movement of the broadcast user's lips. In this case, the broadcast terminal device displays a broadcast image including a character object representing the broadcast user whose lips move in approximate synchronization with the voice produced by the broadcast user. When one or more viewing terminal devices receive the broadcast user's voice data along with the broadcast user's character video data, they output the broadcast user's voice and display a broadcast image including a character object representing the broadcast user whose lips move in approximate synchronization with the output voice.

[0037] The broadcasting terminal device generates a character-related image including a character object representing the broadcasting user in response to an instruction from the broadcasting user. For example, the character-related image is a so-called "stamp image" that can be sent by the broadcasting terminal device to one or more other users' terminal devices in an information exchange service. The character-related image is not limited to a stamp image, and can be any image that can be used in the service provided by the server device. The other users may be limited to viewing users, or may be users other than the broadcasting user, including the viewing user. The other users may be users other than the broadcasting user, but not including the viewing user.

[0038] The character-related image may be used in services other than the information exchange service provided by the server device. For example, in a video distribution service, a distribution terminal device that generates a character-related image may be configured to transmit the character-related image to one or more viewing terminal devices. Also, in an online game, a distribution terminal device that generates a character-related image may be configured to transmit the character-related image to one or more terminal devices of other users participating in the online game.

[0039] In the example shown in Fig. 1(a), the distribution terminal device generates three types of character-related images. The character-related images generated by the distribution terminal device are not limited to the three types shown in Fig. 1(a), and two types of character-related images or four or more types of character-related images may be generated.

[0040] Next, the distribution terminal device sets an available group including character-related images that can be used in at least one service provided by the server device. The character-related images that belong to (are included in) the available group set by the distribution terminal device are not limited to the two types shown in Fig. 1(a). The available group may be set so that one type of character-related image belongs to the available group, or the available group may be set so that three or more types of character-related images belong to the available group.

[0041] The character-related images that do not belong to (are not included in) the available group set by the distribution terminal device are not limited to the one type shown in Fig. 1(a). There may be two or more types of character-related images that do not belong to the available group. There may be no character-related images that do not belong to the available group. In this case, the available group may be set so that each time a character-related image is generated, the generated character-related image belongs to the available group.

[0042] An example of the use of character-related images in an information exchange service will be described below with reference to FIG. 1(b). FIG. 1(b) is a diagram showing an example of an information exchange screen displayed by a broadcasting terminal device using the information exchange service. In the example shown in FIG. 1(b), input information entered by a broadcasting user is displayed in chronological order from top to bottom of the screen in the user display area of the information exchange screen, starting with the oldest input information. For example, the user display area is an area near the right side of the information exchange screen. The input information includes text information, still image information, moving image information, stamp images, etc. Furthermore, input information of other users transmitted from one or more terminal devices of other users is displayed in the other user display area of the information exchange screen in chronological order from top to bottom of the screen, starting with the oldest input information. For example, the other user display area is an area near the left side of the information exchange screen.

[0043] As shown in FIG. 1(b), in response to an instruction from the broadcaster user, the broadcast terminal device selectably displays candidate information indicating all or some of the character-related images belonging to the broadcaster user's available group. The candidate information may be an image identical to the character-related image, a thumbnail image that is a reduced version of the character-related image, or a simplified image that is a deformed version of the character-related image. The candidate information may also include text information, mark images, etc. indicating the character-related image.

[0044] The broadcasting terminal device displays a character-related image corresponding to candidate information selected by the broadcasting user from the displayed candidate information, below the most recently input information, in the user display area of the information exchange screen (Figure 1(b)). Along with displaying the character-related image, the broadcasting terminal device transmits information relating to the display of the character-related image to one or more terminal devices of other users via the server device. In this case, the terminal devices of one or more other users that receive the information relating to the display of the character-related image display the character-related image below the most recently input information in the other user display area of the information exchange screen. The information relating to the display of the character-related image is identification information for identifying the character-related image. The information relating to the display of the character-related image may be the character-related image itself.

[0045] The broadcasting terminal device executes an additional generation process to generate a new character-related image based on reference information related to a character object representing the broadcast user. The reference information is information indicating the displayed broadcast image, response information input by the viewing user, information related to an attachment object currently worn by the character object or previously worn by the character object, information related to an additional object displayed within the broadcast screen, information related to an additional object selected by the viewing user, information related to an owned object associated with the broadcast user, a profile image of the broadcast user or character object, and / or image information showing the virtual space in which the character object exists. The reference information is not limited to the above examples, and may be any information related to the broadcast user or a character object representing the broadcast user. Details of the reference information will be described later.

[0046] An example of the additional generation process will be outlined below with reference to Fig. 2. Hereinafter, a character object representing a broadcast user may be simply referred to as a character object.

[0047] First, the distribution terminal device acquires reference information in response to an instruction from a distribution user or a viewing user, or automatically. In the example shown in (1) of FIG. 2, at least a portion of image information of a displayed distribution image is acquired as reference information. The image information acquired as reference information includes an image showing a character object. The image information acquired as reference information does not have to include an image showing a character object.

[0048] As the reference information, motion data of the character object of the distribution image may be acquired at a predetermined timing. The predetermined timing may be a timing corresponding to an instruction from the distribution user or the viewing user, or an automatically determined timing. The acquired motion data may be a single piece of motion data or multiple consecutive pieces of motion data.

[0049] Next, the broadcasting terminal device generates a new character-related image based on the acquired reference information ((2) in FIG. 2). For example, if the reference information is image information of at least a portion of the broadcast image, the broadcasting terminal device sets the portion of the image information as the character-related image. The broadcasting terminal device may also set the character-related image as an image generated by superimposing a predetermined image on the portion of the image information. If the reference information is motion data, the broadcasting terminal device generates the character-related image based on the motion data acquired as reference information and model data of the character object.

[0050] Next, the broadcasting terminal device assigns (includes) the new character-related image to the available group, and displays selectable candidate information indicating the new character-related image, as shown in (3) of Fig. 2. Then, when a broadcasting user selects one piece of candidate information from the displayed candidate information, the broadcasting terminal device displays the character-related image corresponding to that one piece of candidate information below the most recently entered information in the user display area of the information exchange screen.

[0051] The distribution terminal device may assign (include) the new character-related image to the available group, and may delete other character-related images included in the available group. The deleted other character-related images may be all character-related images included in the available group other than the new character-related image. As a result, the character-related images already included in the available group are changed to the new character-related images.

[0052] The terminal device that generates the character-related image is not limited to the distribution terminal device, and any terminal device that can use a character object representing a user may generate a character-related image including the character object.

[0053] As explained above with reference to Figures 1 and 2, the information processing system, terminal device, and server device increase the types of character-related images that can be used based on information about the broadcasting user or information about a character object representing the broadcasting user. In this way, the information processing system, terminal device, and server device make it possible to improve the user interface that determines the information to be provided to other users without requiring the user to perform complicated operations.

[0054] Furthermore, the information processing system, terminal device, and server device change the character-related image based on information about the broadcasting user or information about a character object representing the broadcasting user to at least one of the available character-related images. In this way, the information processing system, terminal device, and server device can change the information provided to other users without requiring the user to perform complicated operations, preventing the user from getting bored of making choices.

[0055] Furthermore, in conventional information processing systems, in order for a user to perform an operation or give an instruction to increase, decrease, or change the candidates for character-related images, the user had to interrupt or terminate a service that uses the character-related image. As described above, in conventional information processing systems, when a user performs an operation or gives an instruction to increase, decrease, or change the candidates for character-related images, multiple communications are generated regarding the interruption or termination of the service and the resumption of the interrupted service, which can cause a problem of an increase in the processing load of the terminal device and / or the communication load between the terminal device and the server device.

[0056] In contrast, the information processing system, terminal device, and server device disclosed in this embodiment perform an additional generation process that generates new character-related images based on reference information, thereby diversifying the information provided to other users and / or changing the information provided to other users to prevent users from getting bored of their choices, thereby reducing the processing load on the terminal device and / or the communication load between the terminal device and the server device.Furthermore, the information processing system, terminal device, and server device disclosed in this embodiment make it possible to automatically change character-related images using a simple interface while continuing to execute a service.

[0057] In the examples shown in FIGS. 1 and 2, the user and other users using the information exchange service may be users who have a predetermined relationship. For example, other users who have a predetermined relationship with a user are mutual followers of the user. Note that if a user follows another user and the other user also follows the user (the user and the other user mutually follow each other), the user and the other users become mutual followers. Furthermore, other users who have a predetermined relationship with a user may also be other users who have a mutual follow relationship with a predetermined user who is a mutual follower of the user. Furthermore, other users who have a predetermined relationship with a user may also be other users who have a friend relationship with the user, or other users whose specific information (such as a phone number, email address, or a predetermined ID (identification)) is stored in the user's terminal device. Furthermore, if the user is a broadcasting user, other users who have a predetermined relationship with the user may also be viewing users who have viewed a broadcast image broadcast by the user. In this case, when the user and other users who have a predetermined relationship use the information exchange service shown in FIGS. 1 and 2, a predetermined information exchange group consisting of the user and the other users may be temporarily created by the user's or the other users' terminal device or a server device.

[0058] 1 and 2 are merely provided to deepen understanding of the present invention. The present invention is specifically embodied in the following embodiments, and may be embodied in various modifications without substantially departing from the principles of the present invention. All such modifications are included within the scope of the present invention and the disclosure of this specification.

[0059] (Information Processing System 1) FIG. 3 is a diagram illustrating an example of a schematic configuration of an information processing system 1. The information processing system 1 includes a terminal device 2 operated by each of multiple users, and a server device 3. The terminal device 2 and the server device 3 are connected to each other via a communication network, such as a base station 4, a mobile communication network 5, a gateway 6, and the Internet 7. Communication between the terminal device 2 and the server device 3 is performed based on a communication protocol such as Hypertext Transfer Protocol (HTTP). Alternatively, a connection may be established between the terminal device 2 and the server device 3 initially using HTTP communication, and thereafter communication may be performed based on WebSocket, which provides bidirectional communication with lower costs (lower communication loads and processing loads) than HTTP communication. Note that the communication method between the terminal device 2 and the server device 3 is not limited to the above-described method, and any communication method technology may be used between the terminal device 2 and the server device 3 as long as it can realize this embodiment.

[0060] The communication network may be a local area network (LAN) or a wide area network (WAN), or any combination of a LAN and a WAN. The communication network may include a public switched telephone network (PSTN), an integrated services digital network (ISDN), and / or a subnetwork. The communication network may be a wired communication network, such as an Ethernet network cable or a Universal Serial Bus (USB) cable. The communication network may be a wireless communication network, such as a cellular network, including EDGE, 3G, 4G, and / or 5G wireless cellular systems. The wireless communication network may include Wi-Fi, wireless LAN, Bluetooth, or other known wireless forms of communication.

[0061] The terminal device 2 is an information processing device such as a smartphone. The terminal device 2 may be a mobile phone, a notebook PC, a tablet terminal, a tablet PC, a head mounted display (HMD), an e-book reader, a wearable computer, or the like. The terminal device 2 may be a portable game device, a game console, or the like. The terminal device 2 may be any information processing device as long as it is capable of displaying and outputting character objects representing the user and other users, outputting the voices of other users, and exchanging information with other users.

[0062] 3, one server device 3 is illustrated as a component of the information processing system 1, but the server device 3 may be a collection of multiple physically separate server devices 3. In this case, each of the multiple server devices 3 may have the same function, or may have the functions of a single server device 3 in a distributed manner. For example, a server device 3a for realizing a video distribution service and a server device 3b for realizing an information exchange service may be components of the information processing system 1.

[0063] (Terminal device 2) 4 is a diagram illustrating an example of a schematic configuration of a terminal device 2. The terminal device 2 is connected to a server device 3 via a communication network such as a base station 4, a mobile communication network 5, a gateway 6, and the Internet 7, and communicates with the server device 3. The terminal device 2 generates character video data including motion data according to various data (such as imaging data) input by a user, and transmits the generated character video data and / or the user's voice data to the server device 3. The terminal device 2 also receives character video data and / or voice data of other users transmitted from the server device 3, and displays character objects of other users and / or outputs voices of other users based on the received character video data and / or voice data. The terminal device 2 also chronologically displays input information such as text information, still image information, video information, and stamp images input by the user, as well as input information input by other users transmitted from their terminal devices 2. To realize such functions, the terminal device 2 includes a terminal communication I / F 21, a terminal storage device 22, a display device 23, an input device 24, an imaging device 25, a microphone 26, and a terminal processing device 27.

[0064] The terminal communication I / F 21 is implemented as hardware, firmware, or communication software such as a TCP / IP (Transmission Control Protocol / Internet Protocol) driver or a PPP (Point-to-Point Protocol) driver, or a combination of these. The terminal device 2 can transmit data to and receive data from other devices, such as a server device 3, via the terminal communication I / F 21. The terminal communication I / F 21 may include a network controller, such as an Ethernet PRO network interface card, for interfacing with a communication network. The network controller may also be compliant with other direct wireless communication standards, such as Bluetooth (registered trademark), Near Field Communication (NFC), or infrared.

[0065] The terminal storage device 22 is, for example, a semiconductor memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The terminal storage device 22 stores an operating system program, a driver program, a control program, data, etc. used for processing in the terminal processing device 27. The driver programs stored in the terminal storage device 22 include an output device driver program that controls the display device 23 and an input device driver program that controls the input device 24. The control programs stored in the terminal storage device 22 are, for example, application programs for realizing various functions related to various services including video distribution services and information exchange services. The control programs may be transmitted from the server device 3 or other devices.

[0066] The terminal storage device 22 stores, as data, identification information (e.g., a user ID) for uniquely identifying a user operating the terminal device 2. The terminal storage device 22 also stores, as data, background data, model data, etc. The terminal storage device 22 also stores, as data, a user table T1, an object table T2, a group table T3, and a reference information table T4. The terminal storage device 22 may also temporarily store data related to a predetermined process.

[0067] Background data is asset data for constructing a virtual space in which a user's character object exists in a distributed image. The background data includes data for drawing the background of the virtual space, data for drawing various objects included in the distributed image, and data for drawing various other background objects displayed in the distributed image. The background data may also include object position information indicating the positions of various background objects in the virtual space.

[0068] The terminal storage device 22 also stores multiple types of motion data for character-related images. The motion data for character-related images includes body motion data, which is a digital representation of the position and orientation of each part (head, arms, hands (including fingers), chest, torso, legs, or other parts) for causing the character object to perform a predetermined action. The motion data for character-related images may also include face motion data for causing the character object to have a predetermined facial expression.

[0069] The motion data for a character-related image may be a collection of multiple consecutive motion data to represent a character object moving for a predetermined time (e.g., 3 seconds). The motion data for a character-related image may be a single piece of motion data. If the motion data for a character-related image is a single piece of motion data, the character object included in the character-related image generated by the motion data for that character-related image will not move.

[0070] The terminal storage device 22 may also store object data for character-related images, such as object data for representing various background objects, gift objects, and text, along with motion data for the character-related images. The use of object data for character-related images related to the design of such character-related images allows for a greater variety in the expression of the character-related images. In this case, the terminal storage device 22 may store an object ID associated with the object data for the character-related images.

[0071] The display device 23 is a liquid crystal display. The display device 23 may be an organic EL (Electro-Luminescence) display or the like. The display device 23 displays, on a display screen, moving images corresponding to moving image data and / or still images corresponding to still image data supplied from the terminal processing device 27. The display device 23 does not have to be a component of the terminal device 2. In this case, the display device 23 may be a display of an HMD capable of communicating with the server device 3, a projection device for projection mapping, a retinal projection type projection device, or the like capable of communicating with the terminal device 2 via a wired or wireless connection. The terminal device 2 may include a display controller such as a graphics card or a graphics adapter for interfacing with the display device 23.

[0072] The input device 24 is a pointing device such as a touch panel. If the input device 24 is a touch panel, it can detect various touch operations by the user, such as tapping, double-tapping, and dragging. The touch panel may be equipped with a capacitance-type proximity sensor and configured to detect non-contact operations by the user. The input device 24 may also be input keys. The user can use the input device 24 to input letters, numbers, symbols, or positions on the display screen of the display device 23. When operated by the user, the input device 24 generates a signal corresponding to the operation. The input device 24 then supplies the generated signal to the terminal processing device 27 as a user instruction. The terminal device 2 may also include an I / O interface for interfacing with the input device 24. The I / O interface may be connected to various peripheral devices other than the input device 24.

[0073] The imaging device 25 is a camera having an imaging optical system, an imaging element, an image processing unit, etc. The imaging optical system is, for example, an optical lens, and focuses a light beam from a subject on the imaging surface of the imaging element. The imaging element is, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and outputs an image of the subject imaged on the imaging surface. The image processing unit creates moving image data in a predetermined file format from images continuously generated by the imaging element at predetermined intervals and outputs the moving image data as imaging data. Alternatively, the image processing unit creates still image data in a predetermined file format from images generated by the imaging element and outputs the moving image data as imaging data.

[0074] The microphone 26 is a sound collection device configured to acquire voice uttered by the user and convert it into voice data. The microphone 26 is configured to acquire voice input from the user, convert the user's voice input acquired by the microphone 26 into voice data, and output the voice data to the terminal processing device 27.

[0075] The terminal processing device 27 is a processor that loads the operating system program, driver program, and control program stored in the terminal storage device 22 into memory and executes instructions contained in the loaded programs. The terminal processing device 27 is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or a graphics processing unit (GPU), or a combination of various electronic circuits. The terminal processing device 27 may be realized by an integrated circuit such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or a microcontroller unit (MCU). Although FIG. 4 illustrates the terminal processing device 27 as a single component, the terminal processing device 27 may be a collection of multiple physically separate processors. For example, multiple processors operating cooperatively in parallel to execute instructions may be implemented.

[0076] It should be noted that the functions performed by the components disclosed herein (e.g., terminal processing device 27 and / or server processing device 33 (described below)) may be implemented in "circuitry" or "processing circuitry," including general-purpose processors, special-purpose processors, integrated circuits, ASICs, CPUs, conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be "circuitry" or "processing circuitry." A processor may also be a "programmed processor" that executes programs stored in memory.

[0077] As used herein, a "circuitry," "unit," or "means" refers to hardware that is programmed to implement or performs a described function, which may be any hardware disclosed herein or any hardware known to be programmed to implement or perform the described function.

[0078] If the hardware is a processor that is considered to be a type of "Circuitry," the "Circuitry," "unit," or "means" is the combination of the hardware and the software used to configure the hardware and / or processor.

[0079] "Processing Circuitry" may be used to control any computer-based and cloud-based control process. In the operational sequences, operational flows, and flowcharts described below, descriptions or blocks may be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps of the control process. These alternative implementations are also included within the scope of exemplary embodiments of the present advancements. For example, specific logical functions or steps of the control process may be performed in a different order than disclosed, e.g., substantially simultaneously, or in the reverse order of the order disclosed.

[0080] "Processing Circuitry" may include a CPU that executes one or more of the control processes disclosed herein. Process data and instructions may be stored in memory. Process data and instructions may be stored on a storage medium disk, such as a hard disk drive (HDD) or portable storage medium. Process data and instructions may be stored remotely.

[0081] The claimed advancements are not limited by the type of computer-readable medium on which the instructions for the processes of the present invention are stored. For example, the instructions may be stored on a CD, DVD, FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or any other non-volatile computer-readable medium of an information processing device (such as another server device or computer) with which the Processing Circuitry communicates. The instructions may also be stored on network-based storage, cloud-based storage, or other mobile-accessible storage, provided that the instructions are executable by the Processing Circuitry.

[0082] The claimed advancements may be provided as a utility application, a background daemon, a component of an operating system, or any combination thereof, and may execute in conjunction with a CPU and an operating system, including Microsoft Windows, UNIX, Solaris, LINUX, Apple MAC-OS, Apple iOS, and other systems known to those skilled in the art.

[0083] The functions and features disclosed herein may be performed by various distributed components of the information processing system 1. For example, one or more processors may perform these system functions, and each processor may be distributed across multiple components that can communicate within a network. Distributed components may include various human interface and communication devices (such as display monitors, smartphones, tablets, and personal digital assistants (PDAs)), and one or more client and server devices that share processing. Input to the information processing system 1 may be received directly from a user or remotely via real-time or batch processes. Furthermore, some implementations may be performed in modules or hardware other than those disclosed herein, in which case other implementations are within the scope of the claims.

[0084] The terminal processing device 27 executes various commands included in the control program to function as a generation unit 271, a transmission unit 272, a reception unit 273, a display processing unit 274, and an audio output unit 275. The functions of the generation unit 271, the transmission unit 272, the reception unit 273, the display processing unit 274, and the audio output unit 275 will be described later.

[0085] (various tables) 5 is a diagram showing an example of the data structure of the user table T1, object table T2, and group table T3 stored in the terminal storage device 22. At least one of the user table T1, object table T2, and group table T3 may be stored in the server storage device 32 of the server device 3. In this case, the terminal device 2 may receive the latest table stored in the server storage device 32 at a predetermined timing and store it in the terminal storage device 22.

[0086] (User table T1) FIG. 5(a) is an example of the data structure of a user table T1 that manages users who use services provided by the server device 3. The users managed in the user table T1 are broadcast users, but users other than broadcast users may also be managed. In the user table T1, for each user, a user ID, the user's name, the user's character object, the user's owned objects, and the user's used objects are stored in association with one another. In the user table T1, the user's owned coins (information indicating the total amount of virtual monetary value "coins" held by the user) and / or user points may be stored in association with the user's user ID. The user ID is an example of identification data for uniquely identifying each user. The name is an example of data indicating the name of each user.

[0087] For example, a character object is a model ID for identifying model data for generating animation of the character object. The model data is associated with the model ID and stored in the terminal storage device 22. The model data may be three-dimensional model data for generating three-dimensional animation, or two-dimensional model data for generating two-dimensional model data. The model data includes, for example, rig data (so-called "skeleton data") indicating the face and skeleton of the character object's other parts, and surface data indicating the shape and texture of the character object's surface. The model data may include multiple different model data. Each of the multiple model data may have different rig data or the same rig data. Each of the multiple model data may have different surface data or the same surface data. Note that multiple object IDs associated with multiple partial objects constituting the character object may be associated with the model ID. In this case, 3D model data for generating 3D animation may be stored in the terminal storage device 22 and the server storage device 32 for each object ID of the multiple partial objects. In this case, in the display process of the character object, the object IDs of the multiple partial objects associated with the model ID may be extracted, the three-dimensional model data (of each partial object) corresponding to each object ID may be specified, and the character object may be displayed based on the specified three-dimensional model data (of each partial object). Also, image information corresponding to each three-dimensional model data may be stored.

[0088] The owned object is an object ID indicating an object owned by each user. The owned object may be an attachment object that can be associated with a specific part of a character object, a wallpaper object that is placed behind the user's character object in the distribution image, etc. Details of attachment objects and wallpaper objects will be described later. The owned object is not limited to the above examples and may be any object that can be displayed on the distribution screen. The owned object may be an object that the user has acquired based on the result of an event in which the user has participated or the result of a selection game (e.g., a lottery game) played by the user. The owned object may also be an object that the user has acquired by spending "coins," which are virtual currency, using the purchase function of the information processing system 1.

[0089] The used object is an object ID indicating an object used in a distributed image being displayed on the user's terminal device 2. For example, when a user inputs a change instruction during a video distribution service, an attachment object held as one of the user's owned objects is attached to the user's character object included in the distributed image being displayed. In this case, the object ID of the attachment object attached to the user's character object is stored in the user table T1 as a used object in association with the user ID. Also, for example, when a user inputs a change instruction during a video distribution service, a wallpaper object held as one of the user's owned objects is placed behind the user's character object in the distributed image being displayed. In this case, the object ID of the wallpaper object placed in the distributed image is stored in the user table T1 as a used object in association with the user ID.

[0090] (Object table T2) 5(b) shows an example of the data structure of an object table T2 that manages objects selected as additional objects. An additional object is, for example, a gift object given to a broadcasting user by a viewing user. In the object table T2, the object ID, name, image information, rarity, placement location, etc. are stored in association with each other for each additional object.

[0091] The object ID is an example of identification data for uniquely identifying each additional object. The image information is one or more still images corresponding to each additional object. The image information may be one or more types of moving images corresponding to each additional object. The rarity is information indicating the rarity of each additional object. The placement location is information indicating the location where the image information of each additional object is displayed. Note that if the display position and display range of a character object are determined, information indicating the relative position of the character object may be stored as the placement location. Model data of the additional object may be stored instead of image information. The model data of the additional object includes rig data and surface data indicating a three-dimensional shape, and an image of the additional object viewed from a predetermined viewpoint is drawn based on the model data.

[0092] Additional objects are classified into multiple types (categories). For example, additional objects include effect objects used as effect gifts, normal objects used as normal gifts, attachment objects used as attachment gifts, and message objects used as message gifts. Additional objects may also include wallpaper objects that are placed behind character objects in the distribution image. Information indicating the type (category) of an additional object may be stored in the object table T2 in association with the object ID of the additional object. Image information and placement locations will be described below according to the type of additional object.

[0093] The effect object is an object that affects the overall impression of the distribution image. For example, the effect object is an object that resembles confetti. When the effect object is an object that resembles confetti, an image showing multiple pieces of paper is stored as the image information.

[0094] Information indicating "space" as the placement location of the effect object is stored. For example, an object resembling confetti is displayed across the entire broadcast image. The effect object may be displayed across the entire broadcast screen including the broadcast user's broadcast image in accordance with a viewing user's instruction. The effect object may be displayed so as to overlap with a character object included in the broadcast image. Unlike wearable objects, which will be described later, the effect object is displayed without being associated with a specific part of the character. In this way, by displaying an effect object on the broadcast user's broadcast image in accordance with a viewing user's instruction, the terminal device 2 of the information processing system 1 can change the overall impression of the broadcast user's broadcast image compared to before the effect object was displayed.

[0095] A normal object is, for example, an object that resembles a stuffed animal, a bouquet of flowers, an accessory, or an item suitable for a gift or present. Information indicating a "space" is stored as the placement location of the normal object. For example, information regarding a predetermined movement route is associated with the normal object, and the normal object is displayed moving along the predetermined movement route within the delivered image. The information indicating the "space" stored as the placement location of the "normal object" may be classified into multiple types. The multiple types of "space" may be, for example, a "center space" (the central area of the image when the delivered image is divided into three parts horizontally), a "left space" (the left area of the image when the delivered image is divided into three parts horizontally), and a "right space" (the right area of the image when the delivered image is divided into three parts horizontally). Furthermore, the multiple types of "space" may be a "center space" (the central area of the image when the delivered image is divided into three parts vertically), an "upper space" (the upper area of the image when the delivered image is divided into three parts vertically), and a "lower space" (the lower area of the image when the delivered image is divided into three parts vertically). In this case, the movement route associated with the normal object is set within the "space" of the type associated as the placement location of the normal object. For example, if information indicating a "right space" is stored in a normal object, information regarding a movement route for moving within the right area of the image when the delivered image is divided into left and right thirds is associated with the normal object. Furthermore, a normal object may be displayed that moves according to a predetermined movement rule from a position set for each normal object in the delivered image or a position automatically and randomly determined. The predetermined movement rule is, for example, a movement rule set so that the normal object moves as if it were free falling. Note that the normal object may be displayed so as to overlap with a character object. In this case, unlike an attached object described later, the normal object is displayed without being associated with a specific part of the character object. Furthermore, the normal object may be displayed so as to bounce back when it comes into contact with the character object.

[0096] When a normal object is displayed overlapping with a character object included in the distributed image, for example, the normal object may be displayed so as to overlap with "a part of the character object other than the 'head', including the 'face'." In this case, the normal object is displayed so as not to overlap with the "head" of the character object. Also, the normal object may be displayed so as to overlap with "a part of the character object other than the 'upper body', including the 'face'." In this case, the normal object is displayed so as not to overlap with the "upper body" of the character object.

[0097] The attachment object is an object that is displayed in the distributed image in association with a specific part (attachment location) of the character object. For example, the specific part may be the left front, right front, left rear, right rear, front center, back center, left eye, right eye, left ear, right ear, or all of the hair on the character object's head. The specific part may also be the thumb, ring finger, wrist, elbow, shoulder, upper arm, entire hand, entire arm, torso, upper body, lower body, etc.

[0098] An attachment object associated with a specific part of a character object may be displayed in the distributed image so as to be in contact with the specific part of the character object. Furthermore, an attachment object associated with a specific part of a character object may be displayed in the distributed image so as to cover part or all of the specific part of the character object. The specific part may be identified by three-dimensional position information indicating a position in three-dimensional coordinate space, or may be associated with the position information in the three-dimensional coordinate space.

[0099] Image information of the wearing object is, for example, image information that resembles accessories (headbands, necklaces, earrings, etc.) worn by the character object, clothing (T-shirts, dresses, etc.), costumes, and other items that can be worn by the character object.

[0100] As the placement location of the attachment object, information indicating the attachment part, which indicates which part of the character object the attachment object is associated with, is stored. For example, if the attachment object is a "headband," information indicating the "head" of the character object is stored as the placement location of the attachment object. Also, if the attachment object is a "T-shirt," information indicating the "torso" of the character object is stored as the placement location of the attachment object.

[0101] Information indicating multiple attachment positions in a three-dimensional coordinate space may be stored as the attachment position of an attachment object. For example, if the attachment object is a "headband," information indicating two attachment positions of the attachment object, the "left rear of the head" and the "right rear of the head" of a character object, may be stored. As a result, an attachment object resembling a "headband" is displayed as being attached to both the "left rear of the head" and the "right rear of the head" of the character object.

[0102] When a character object is equipped with multiple types of attachment objects that are attached to the same attachment location, the multiple types of attachment objects are attached to the character object at different times. As a result, multiple types of attachment objects that are attached to the same attachment location are not attached to the character object at the same time. For example, if the "head" is stored as the attachment location for both an attachment object representing a "headband" and an attachment object representing a "hat," the "headband" attachment object and the "hat" attachment object will not be displayed on the character object's head at the same time.

[0103] In the object table T2, a display time according to the type of additional object may be stored in association with the object ID of the additional object. For example, the display time of an attached object may be stored as a time longer than the display time of an effect object and the display time of a normal object. For example, if "60 seconds" is stored as the display time of an attached object, "5 seconds" may be stored as the display time of an effect object, and "10 seconds" may be stored as the display time of a normal object.

[0104] (Group table T3) Fig. 5(c) shows an example of the data structure of a group table T3 that manages information exchange groups to which users belong. A group table T3 is provided for each user, and the group table T3 shown in Fig. 5(c) is the group table T3 of the user who operates the terminal device 2 that stores the group table T3.

[0105] In the group table T3, for each information exchange group, the group ID of the information exchange group, the name of the information exchange group, the users belonging to the information exchange group, etc. are stored in association with one another. The group ID is an example of identification data for uniquely identifying each information exchange group. The name is an example of data indicating the name of each information exchange group. The users belonging to the information exchange group are the user IDs of each user belonging to the information exchange group.

[0106] When the group table T3 is stored in the server storage device 32, the user ID of the corresponding user (the user who operates the terminal device 2 storing the group table T3) is stored in association with each group table T3.

[0107] Returning to FIG. 4, the functions of the generation unit 271, transmission unit 272, reception unit 273, display processing unit 274, and audio output unit 275 will be described.

[0108] (Generation part 271) The generation unit 271 functions when the terminal device 2 is used as a broadcasting terminal device operated by a broadcasting user. The generation unit 271 acquires, from the imaging device 25, imaging data continuously output by the imaging device 25. For example, the imaging device 25 of the terminal device 2 (broadcasting terminal device) held by the broadcasting user is provided on the terminal device 2 so that the face of the broadcasting user looking at the display screen of the display device 23 is the imaging direction. The imaging device 25 continuously captures images of the broadcasting user's face in the imaging direction, acquires imaging data of the broadcasting user's face, and outputs the data to the generation unit 271 of the terminal processing device 27. The imaging device 25 may acquire imaging data by capturing images of parts of the broadcasting user other than the face, such as the head, arms, hands (including fingers), chest, torso, legs, or other parts. The imaging device 25 may also be a 3D camera capable of detecting the depth of a person's face.

[0109] The generation unit 271 generates face motion data, which is a digital representation of the broadcast user's facial movements, as time passes, based on continuously acquired imaging data. The face motion data may be generated at predetermined sampling time intervals. In this way, the face motion data generated by the generation unit 271 can digitally represent the broadcast user's facial movements (changes in facial expressions) in chronological order.

[0110] The generation unit 271 may generate body motion data, which is a digital representation of the position and orientation of each part of the distribution user (head, arms, hands (including fingers), chest, torso, legs, or other parts), together with or separately from the facial motion data, based on continuously acquired imaging data.

[0111] The body motion data may be generated based on detection information from a known motion sensor worn by the broadcast user. In this case, the terminal communication I / F 21 of the terminal device 2 (broadcast terminal device) has a predetermined communication circuit for wirelessly acquiring the detection information from the motion sensor worn by the broadcast user. The generation unit 271 then generates the body motion data based on the detection information acquired by the terminal communication I / F 21. The body motion data may be generated at predetermined sampling time intervals. In this way, the body motion data generated by the generation unit 271 can digitally represent the body movements of the broadcast user in chronological order.

[0112] The generation of body motion data based on detection information from a motion sensor worn by a broadcasting user may be performed, for example, in a filming studio. In this case, the filming studio may be equipped with a base station, a tracking sensor, and a display. The base station may be, for example, a multi-axis laser emitter. The motion sensor worn by the broadcasting user may be, for example, a Vive Tracker provided by HTC CORPORATION, and the base station may be, for example, a base station provided by HTC CORPORATION.

[0113] The supporter computer may also be installed in a separate room adjacent to the photography studio. The display of the photography studio may be configured to display information received from the supporter computer. The server device 3 may be installed in the same room as the supporter computer. The room where the supporter computer is installed may be separated from the photography studio by a glass window. In this case, the operator of the supporter computer can see the broadcasting user. The supporter computer may be configured to be able to change the settings of various devices provided in the photography studio in response to the operation of the supporter. The supporter computer can, for example, set the scanning interval by the base station, set the tracking sensor, and change various settings of various other devices. The operator may input a message to the supporter computer, and the input message may be displayed on the display of the photography studio.

[0114] The generation unit 271 generates character video data including facial motion data and / or body motion data generated as needed, and outputs the generated character video data to the display processing unit 274. Hereinafter, facial motion data and body motion data may be collectively referred to as motion data. The generation unit 271 also outputs the generated character video data and the broadcast user's user ID stored in the terminal storage device 22 to the transmission unit 272. When the generation unit 271 acquires audio data of the broadcast user output from the microphone 26, it outputs the generated character video data, the acquired audio data, and the broadcast user's user ID stored in the terminal storage device 22 to the transmission unit 272.

[0115] (Transmitter 272) In response to an instruction to start a broadcast from the broadcast user, the transmitting unit 272 of the broadcast user's terminal device 2 (broadcast terminal device) transmits information regarding a broadcast start instruction for instructing the broadcast user to start a broadcast event, together with the broadcast user's user ID stored in the terminal storage device 22, to the server device 3 via the terminal communication I / F 21. Hereinafter, the broadcast user's user ID may be referred to as the broadcast user ID.

[0116] Additionally, the transmitting unit 272 of the broadcast user's terminal device 2 (broadcast terminal device) transmits the broadcast user's character video data output from the generating unit 271 to the server device 3 via the terminal communication I / F 21 together with the broadcast user ID stored in the terminal storage device 22 as information for displaying the broadcast user's broadcast image in the broadcast event. When the broadcast user's voice data is output from the generating unit 271 together with the broadcast user's character video data, the transmitting unit 272 transmits the broadcast user's character video data and voice data together with the broadcast user ID to the server device 3 via the terminal communication I / F 21.

[0117] In addition, in response to an instruction to end the distribution by the distribution user, the transmitting unit 272 of the distribution user's terminal device 2 (distribution terminal device) transmits information regarding a distribution end instruction to instruct the distribution user to end the distribution event to the server device 3 via the terminal communication I / F 21, together with the distribution user ID stored in the terminal storage device 22.

[0118] Furthermore, in response to an instruction from the viewing user to start viewing a distributed image, the transmitting unit 272 of the viewing user's terminal device 2 (viewing terminal device) transmits information regarding the viewing start instruction to the server device 3 via the terminal communication I / F 21, together with information regarding the distributed image and the user ID of the viewing user stored in the terminal storage device 22. The information regarding the distributed image is information for identifying the distributed image for which the viewing user has been instructed to start viewing, and is, for example, identification information of the distributed image (distributed image ID) or the broadcast user ID of the broadcast user broadcasting the distributed image. Hereinafter, the user ID of the viewing user may be referred to as the viewing user ID.

[0119] Additionally, the transmission unit 272 transmits information indicating various instructions input by the user by operating the input device 24, together with the user ID stored in the terminal storage device 22, to the server device 3 via the terminal communication I / F 21. Various instructions from the user include, for example, addition instructions, modification instructions, and generation instructions. The information indicating the addition instructions is transmitted to the server device 3 together with the object ID of the additional object to be included in the broadcast user's broadcast image, the broadcast user ID, and the viewing user ID stored in the terminal storage device 22 of the viewing user's terminal device 2 (viewing terminal device).

[0120] When a broadcasting user selects partial objects and issues instructions regarding the creation of a character object composed of the selected partial objects, a model ID for identifying the model data of the character object and object IDs of the multiple partial objects that make up the character object may be transmitted to server device 3 via terminal communication I / F 21 together with a user ID and the like stored in terminal storage device 22 of the broadcasting user's terminal device 2 (broadcasting terminal device). In this case, server device 3 may store the user ID, model ID, and object IDs of the multiple partial objects that make up the character object in server storage device 32 in association with one another.

[0121] Furthermore, the viewing user's terminal device 2 (viewing terminal device) that receives, via server device 3, a model ID for identifying model data of the broadcasting user's character object, the object IDs of multiple partial objects that make up the user's character object, and the user's user ID, may associate the received user ID, model ID, and object IDs of the multiple partial objects that make up the character object with one another and store them in terminal storage device 22. In this way, simply by transmitting the broadcasting user's user ID (or model ID) to the viewing user's terminal device 2 (viewing terminal device) via server device 3, the viewing user's terminal device 2 (viewing terminal device) can extract the object IDs of the multiple partial objects associated with the received user ID (or model ID). The viewing user's terminal device 2 (viewing terminal device) can then display a broadcast image including the broadcasting user's character object based on the three-dimensional model data (of each partial object) that corresponds to each extracted object ID.

[0122] In response to an instruction from the user to start a new information exchange, the transmitting unit 272 of the user's terminal device 2 transmits information regarding the new start instruction to the server device 3 via the terminal communication I / F 21. The information regarding the new start instruction is transmitted to the server device 3 along with the user ID of the other user designated by the user and the user ID stored in the terminal storage device 22 of the terminal device 2. The other users designated by the user are one or more users designated by the user as partners with whom the user will exchange information. Hereinafter, a group to which the user and one or more other users belong may be referred to as an information exchange group.

[0123] In response to an instruction from the user to start information exchange (for example, text chat) within the information exchange group, the transmitting unit 272 of the terminal device 2 of the user belonging to the information exchange group transmits information related to the information exchange start instruction to the server device 3 via the terminal communication I / F 21. The group ID of the information exchange group to which the user belongs, which is stored in the terminal storage device 22 of the terminal device 2, is transmitted as destination information together with the information related to the information exchange start instruction to the server device 3. Note that if the server device 3 does not store the user IDs of the users belonging to the information exchange group, destination information including the group ID indicating the information exchange group or the user IDs of other users belonging to the information exchange group may be transmitted to the server device 3 together with the information related to the information exchange start instruction.

[0124] When input information is input by the user while the information exchange screen is being displayed, the transmitting unit 272 of the terminal device 2 of a user belonging to the information exchange group transmits the input information to the server device 3 via the terminal communication I / F 21. Together with the input information, the user ID and destination information stored in the terminal storage device 22 of the terminal device 2 of the user belonging to the information exchange group are transmitted to the server device 3. The destination information is a group ID indicating the information exchange group, or the user ID of another user belonging to the information exchange group.

[0125] (Receiving unit 273) The receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives the character video data and user ID of the broadcasting user transmitted from the server device 3 via the terminal communication I / F 21. The receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives the character video data, audio data, and user ID of the broadcasting user transmitted from the server device 3 via the terminal communication I / F 21. The receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receives the audio data and user ID of the broadcasting user transmitted from the server device 3 via the terminal communication I / F 21. In addition, the receiving unit 273 of the broadcasting user's terminal device 2 (distribution terminal device) and the receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) receive information indicating various instructions from other users, etc. transmitted from the server device 3 via the terminal communication I / F 21. The receiving unit 273 passes the received information to the terminal processing device 27.

[0126] The broadcast user's character video data and user ID transmitted from server device 3 were transmitted to server device 3 by the transmitting unit 272 of the broadcast user's terminal device 2 (broadcast terminal device). Also, the broadcast user's character video data, voice data, and user ID transmitted from server device 3 were transmitted to server device 3 by the transmitting unit 272 of the broadcast user's terminal device 2 (broadcast terminal device). Also, the broadcast user's voice data and user ID transmitted from server device 3 were transmitted to server device 3 by the transmitting unit 272 of the broadcast user's terminal device 2 (broadcast terminal device).

[0127] The receiving unit 273 of the terminal device 2 of a user belonging to the information exchange group receives information regarding the new start instruction sent from the server device 3 and the user ID of the user belonging to the information exchange group in response to information regarding the new start instruction sent from the terminal device 2 of another user belonging to the information exchange group.

[0128] The receiving unit 273 of the terminal device 2 of the user belonging to the information exchange group receives, via the terminal communication I / F 21, the input information and user IDs of the other users belonging to the information exchange group transmitted from the server device 3.

[0129] (Display processing unit 274) In response to an instruction to start a distribution event, the display processing unit 274 of the distribution user's terminal device 2 (distribution terminal device) displays, on the display device 23, an image drawn based on the character video data including the motion data generated by the generation unit 271 as a distribution image of the distribution user. The motion data is transmitted to the viewing user's terminal device 2 (viewing terminal device) as character video data. The display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) also displays, on the display device 23, an image drawn based on the distribution user's character video data (motion data) received by the reception unit 273 as a distribution image of the distribution user.

[0130] The following describes the process of drawing a distributed image based on motion data. In the following description, the process of drawing a distributed image based on motion data in the distribution terminal device uses the user ID stored in the terminal storage device 22, and the process of drawing a distributed image based on motion data in the viewing terminal device uses the user ID received from the server device 3.

[0131] First, the display processing unit 274 extracts the model ID of the character object associated with the user ID and the object ID of the use object from the user table T1. Next, the display processing unit 274 reads from the terminal storage device 22 the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the use object. Next, the display processing unit 274 generates an animation of the character object in which the use object is attached to the placement location based on the read model data, the image information, placement location, and motion data of the use object. Note that if the model ID of the character object is associated with the object IDs of multiple partial objects that make up the character object, the display processing unit 274 extracts model data corresponding to the object IDs of the multiple partial objects associated with the extracted model ID. Next, the display processing unit 274 generates an animation of the character object in which the use object is attached to the placement location based on the read model data of the multiple partial objects, the image information, placement location, and motion data of the use object. Note that the use object may be included among multiple partial objects.

[0132] If the motion data includes facial motion data, the display processing unit 274 generates animation of the character object so that the facial expression of the character object changes based on the facial motion data. This allows the display processing unit 274 to generate animation of the character object that moves in synchronization with the facial movements of the broadcast user based on the read model data and facial motion data. If the motion data includes facial motion data and body motion data, the display processing unit 274 generates animation of the character object that moves in synchronization with the facial movements and body movements of the broadcast user based on the read model data, facial motion data, and body motion data. Next, the display processing unit 274 draws a distribution image including the generated animation (moving image) of the character object and a background image that shows a background generated using background data stored in the terminal storage device 22. Then, the display processing unit 274 outputs drawing data that shows the drawn distribution image, and ends the drawing process of the distribution image based on the motion data. If three-dimensional model data is used in the drawing process of the distribution image, the distribution image drawn in the drawing process of the distribution image is a three-dimensional moving image drawn in a three-dimensional virtual space including a character object that moves in synchronization with the facial movements and body movements of the broadcast user. The following description will mainly focus on delivered images that include animations (moving images) of character objects, but the delivered images are not limited to those that include animations (moving images) of character objects. The delivered images may also include still images of character objects. Furthermore, the delivered images may be images in which a new still image of a character object is displayed at predetermined intervals.

[0133] Furthermore, in response to a change instruction from the broadcast user, the display processing unit 274 may execute a change process to display on the display device 23 a character object to which an attachment object that is an object owned by the broadcast user or an attachment object sent by the viewing user as an attachment gift has been applied. For example, if the attachment object instructed to be changed by the broadcast user is a "headband," the display processing unit 274 extracts the object ID of the attachment object ("headband") from the owned objects associated with the broadcast user's user ID in the user table T1, and stores it as an object used by the broadcast user. The display processing unit 274 extracts image information and a placement location associated with the object ID stored as the object used by the broadcast user from the object table T2. Next, based on the extracted image information and placement location, the display processing unit 274 generates a broadcast image including a character object of the broadcast user wearing the attachment object ("headband") on their head (the placement location associated with the object ID of the "headband"). Note that the display processing unit 274 may also generate a broadcast image including a character object wearing an attachment object that moves in conjunction with the movement of the attachment location (placement location) of the character object. For example, in the delivered image, the attachment object "headband" is displayed to move in conjunction with the movement of the character object's head.

[0134] Furthermore, the display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) displays a delivery image including the additional object in response to an instruction to add the additional object from the viewing user. Furthermore, the display processing unit 274 of the broadcasting user's terminal device 2 (distribution terminal device) displays a delivery image including the additional object in response to information indicating an instruction to add the additional object received from the viewing user's terminal device 2.

[0135] When the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with the information indicating an instruction to add an additional object is the object ID of one of the effect objects, the display processing unit 274 of the broadcasting user's terminal device 2 (broadcast terminal device) references the object table T2 and extracts from the terminal storage device 22 a still image or video (image information) of the effect object associated with the object ID received along with the information indicating the instruction to add, and generates a broadcast image for the viewing user that includes the extracted still image or video. For example, when an instruction to add an effect object representing confetti or fireworks is given, the display processing unit 274 of the broadcasting user's terminal device 2 (broadcast terminal device) generates a broadcast image that includes a video of an effect object (not shown) that resembles confetti or fireworks. When the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with the information indicating an instruction to add an additional object is the object ID of one of the effect objects, the display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) similarly references the object table T2 and extracts from the terminal storage device 22 a still image or video (image information) of the effect object associated with the object ID received along with the information indicating the instruction to add, and generates a broadcast image for the viewing user that includes the extracted still image or video.

[0136] When the object ID received from the viewing user's terminal device 2 (viewing terminal device) along with the information indicating an instruction to add an additional object is the object ID of a normal object, the display processing unit 274 of the broadcasting user's terminal device 2 (viewing terminal device) refers to the object table T2 and extracts a still image or video (image information) of the normal object and its placement location associated with the object ID of the normal object. Next, the display processing unit 274 of the broadcasting user's terminal device 2 (distribution terminal device) generates a broadcast image of the viewing user including a still image or video of the normal object placed at the position indicated by the placement location. Furthermore, when the object ID received from the viewing user along with the information indicating an instruction to add an additional object is the object ID of a normal object, the display processing unit 274 of the viewing user's terminal device 2 (viewing terminal device) similarly refers to the object table T2 and extracts a still image or video (image information) of the normal object and its placement location associated with the object ID of the normal object, and generates a broadcast image of the viewing user including a still image or video of the normal object placed at the position indicated by the placement location.

[0137] The display processing unit 274 may generate a distributed image including a moving image of a normal object moving within the display area of the distributed image. For example, the display processing unit 274 may generate a distributed image including a moving image of a normal object falling from top to bottom of the distributed image. In this case, the normal object may be displayed within the display area of the distributed image from the start of its fall until it reaches the bottom edge of the distributed image, and may disappear from the distributed image after it reaches the bottom edge of the distributed image. Note that the movement route of the normal object may be from left to right, from right to left, or from top left to bottom left of the distributed image, or a route in any other direction. The route may also be a linear trajectory, a circular trajectory, an elliptical trajectory, a spiral trajectory, or any other trajectory.

[0138] If the object ID received together with the information indicating an instruction to add an additional object from the viewing user is the object ID of a worn object, the display processing unit 274 of the broadcasting user's terminal device 2 (broadcasting terminal device) displays notification information regarding the wearing instruction within the broadcast image. Next, the display processing unit 274 references the object table T2 and extracts image information and a placement location of the worn object associated with the object ID received together with the information indicating the addition instruction. Next, the display processing unit 274 generates a broadcast image including the broadcasting user's character object wearing the worn object at the attachment location indicated by the placement location based on the extracted image information and placement location. Furthermore, if the object ID received together with the information indicating an instruction to add an additional object from the viewing user is the object ID of a worn object, the display processing unit 274 displays notification information regarding the wearing instruction within the broadcast image. Next, the display processing unit 274 references the object table T2 and extracts image information and a placement location of the worn object associated with the object ID received together with the information indicating the addition instruction. Next, based on the extracted image information and placement location, the display processing unit 274 generates a distribution image including a character object of the broadcast user wearing the attachment object at the attachment part indicated by the placement location.

[0139] Furthermore, in response to an instruction from the user to start a new information exchange, the display processing unit 274 of the user's terminal device 2 displays an information exchange screen related to a new information exchange group to which the user and all other users designated by the user belong. Furthermore, when the receiving unit 273 receives information related to the new start instruction from other users and the user IDs of each user belonging to the new information exchange group, the display processing unit 274 of the user's terminal device 2 displays the information exchange screen related to the information exchange group.

[0140] Furthermore, the display processing unit 274 of the terminal device 2 of a user belonging to an information exchange group displays an information exchange screen related to the information exchange group in response to an instruction from the user to start information exchange (for example, text chat, etc.) within the information exchange group. Furthermore, when the receiving unit 273 receives information related to an instruction to start information exchange from other users belonging to the information exchange group and the user IDs of each user belonging to the information exchange group, the display processing unit 274 of the terminal device 2 of a user belonging to the information exchange group displays an information exchange screen related to the information exchange group.

[0141] Furthermore, when input information is input by the user while the information exchange screen is displayed, the display processing unit 274 of the terminal device 2 of the user belonging to the information exchange group displays the input information below the most recently input information in the user display area of the information exchange screen. Furthermore, when input information by another user is received by the receiving unit 273 while the information exchange screen is displayed, the display processing unit 274 of the terminal device 2 of the user belonging to the information exchange group displays the received input information below the most recently input information in the other user display area of the information exchange screen.

[0142] (Audio output unit 275) When the audio output unit 275 of the viewing user's terminal device 2 (viewing terminal device) receives the broadcast user's audio data via the receiving unit 273, it outputs the broadcast user's audio, generated based on the audio data, from a speaker (not shown). Furthermore, when the audio output unit 275 of the viewing terminal device receives the broadcast user's character video data and audio data, it outputs the broadcast user's audio, based on the audio data, from a speaker along with the display of the broadcast user's broadcast image via the display processing unit 274. This allows the broadcast user's audio to be output while the broadcast user's character object included in the broadcast image is moving its lips, allowing the viewing user to view a broadcast image including the broadcast user's character object that appears to be making an audio sound.

[0143] (Server device 3) FIG. 6 is a diagram illustrating an example of a schematic configuration of a server device 3. The server device 3 provides various services, such as a video distribution service and an information exchange service, to the terminal devices 2 of multiple users. The server device 3 transmits character video data of the broadcasting user and information indicating various instructions transmitted from the terminal device 2 of the broadcasting user to the terminal devices 2 of one or more viewing users. The server device 3 also transmits input information (text information, stamp images, etc.) received from one user's terminal device to the terminal devices 2 of one or more other users who belong to the same information exchange group as the one user. The server device 3 may also transmit information received from one user's terminal device 2 to the terminal device 2 of a user identified by the user ID included in destination information received simultaneously with the information. To this end, the server device 3 includes a server communication I / F 31, a server storage device 32, and a server processing device 33.

[0144] The server communication I / F 31 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination of these. The server device 3 can transmit information to other devices and receive information from other devices via the server communication I / F 31.

[0145] The server storage device 32 is, for example, a semiconductor memory device such as a ROM or RAM. The server storage device 32 may be, for example, a magnetic disk, an optical disk, or any other storage device capable of storing data. The server storage device 32 stores an operating system program, a driver program, an application program, data, etc. used for processing in the server processing device 33. The server storage device 32 may also store, as data, a user table T1, an object table T2, a group table T3, and a reference information table T4. The server storage device 32 may also store, as data, the broadcast user ID of a broadcast user currently broadcasting, and store, in association with the broadcast user ID, the user ID of a viewing user currently viewing a broadcast image of that broadcast user. The broadcast user ID is a type of information related to a broadcast image. The information related to a broadcast image may be identification information for the broadcast image (a broadcast image ID). For example, a broadcast image ID, a broadcast user ID, and a viewing user ID may be stored in association with each other.

[0146] The server processing device 33 is a processor that loads the operating system program, driver program, and control program stored in the server storage device 32 into memory and executes instructions contained in the loaded programs. The server processing device 33 is, for example, an electronic circuit such as a CPU, MPU, DSP, or GPU, or a combination of various electronic circuits. The server processing device 33 may be implemented by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. While FIG. 6 illustrates the server processing device 33 as a single component, the server processing device 33 may also be a collection of multiple physically separate processors. For example, the server processing device 33 may be implemented with multiple processors that operate cooperatively in parallel to execute instructions. The server processing device 33 functions as a server receiving unit 331 and a server transmitting unit 332 by executing various instructions contained in the control program.

[0147] (Server receiving unit 331) The server receiving unit 331 receives information regarding a broadcast start instruction sent from the broadcast user's terminal device 2 (broadcast terminal device) together with the broadcast user ID via the server communication I / F 31. The server receiving unit 331 receives information regarding a broadcast end instruction sent from the broadcast user's terminal device 2 (broadcast terminal device) together with the broadcast user ID via the server communication I / F 31. The server receiving unit 331 also receives information for displaying the broadcast user's broadcast image sent from the broadcast user's terminal device 2 (broadcast terminal device) together with the broadcast user ID via the server communication I / F 31.

[0148] The server receiving unit 331 also receives information regarding a viewing start instruction transmitted from the viewing user's terminal device 2 (viewing terminal device), together with information regarding the distributed image and the viewing user ID, via the server communication I / F 31. The server receiving unit 331 receives information indicating an additional instruction transmitted from the viewing user's terminal device 2 (viewing terminal device), together with the object ID, distribution user ID, and viewing user ID, via the server communication I / F 31.

[0149] The server receiving unit 331 also receives, from the terminal device 2 of the user who wishes to start a new information exchange, information related to a new start instruction, the user ID of the user, and the user IDs of other users selected as partners in the information exchange, via the server communication I / F 31. In this case, the server receiving unit 331 sets a group ID of the new information exchange group. The server receiving unit 331 then stores the received user ID of the user and the user IDs of the other users as user IDs belonging to the new information exchange group in the server storage device 32 in association with the group ID of the new information exchange group.

[0150] Furthermore, the server receiving unit 331 receives information related to the information exchange start instruction, the user ID of the user, and destination information from the terminal device 2 of a user belonging to the information exchange group via the server communication I / F 31. If the received destination information is a group ID indicating the information exchange group, the server receiving unit 331 refers to the server storage device 32 and extracts the user ID of the user associated with the group ID received as the destination information. Then, the server receiving unit 331 specifies the extracted user ID as the user ID of the user of the destination terminal device 2. If the received destination information is a user ID, the server receiving unit 331 specifies the user ID received as the destination information as the user ID of the user of the destination terminal device 2. Then, the server receiving unit 331 passes the received information related to the information exchange start instruction and the user ID of the user of the destination terminal device 2 to the server transmitting unit 332.

[0151] The server receiving unit 331 also receives input information entered by a user, the user ID of the user, and destination information from the terminal device 2 of a user belonging to the information exchange group via the server communication I / F 31. Next, the server receiving unit 331 extracts the user ID of the user of the destination terminal device 2 based on the received destination information. The server receiving unit 331 passes the received input information, the user ID of the user who entered the input information, and the user ID of the user of the destination terminal device 2 to the server transmitting unit 332.

[0152] (Server transmission unit 332) The server transmitting unit 332 transmits the broadcast user's character video data and user ID received by the server receiving unit 331 to the viewing user's terminal device 2 (viewing terminal device) indicated by the viewing user ID associated with the broadcast user ID of that broadcast user, stored in the server storage device 32, via the server communication I / F 31. The server transmitting unit 332 transmits the broadcast user's character video data, audio data, and user ID received by the server receiving unit 331 to the viewing user's terminal device 2 (viewing terminal device) indicated by the viewing user ID associated with the broadcast user ID of that broadcast user, stored in the server storage device 32, via the server communication I / F 31.

[0153] Additionally, the server transmitting unit 332 transmits information indicating various instructions from the broadcast user received by the server receiving unit 331 to the terminal device 2 (viewing terminal device) of the viewing user indicated by the broadcast user ID associated with the broadcast user ID of that broadcast user, stored in the server storage device 32, via the server communication I / F 31. Additionally, the server transmitting unit 332 transmits information indicating various instructions from the viewing user received by the server receiving unit 331 to the terminal device 2 of the broadcast user indicated by the broadcast user ID associated with the user ID of that viewing user, and to the terminal devices 2 of viewing users indicated by the user IDs of other viewing users associated with that broadcast user ID, via the server communication I / F 31.

[0154] Furthermore, in response to the setting of the group ID of the new information exchange group by the server receiving unit 331, the server transmitting unit 332 extracts the user IDs of the users who belong to the new information exchange group and are associated with the group ID of the new information exchange group from the server storage device 32. Then, the server transmitting unit 332 transmits information relating to a display instruction for an information exchange screen for the new information exchange group to the terminal device 2 of the user identified by the extracted user ID via the server communication I / F 31.

[0155] In addition, when the server transmitting unit 332 receives information regarding an information exchange start instruction and destination information from the server receiving unit 331, it transmits information regarding an instruction to display an information exchange screen regarding the received information exchange start instruction to the terminal device 2 of the user identified by the user ID related to the destination information via the server communication I / F 31.

[0156] In addition, when the server transmitting unit 332 receives input information, the user ID of the user who input the input information, and the user ID of the user of the destination terminal device 2 from the server receiving unit 331, it transmits the received input information via the server communication I / F 31 to the terminal device 2 of the user identified by the user ID of the user of the destination terminal device 2 other than the user ID of the user who input the input information.

[0157] (Examples of various screens) 7 to 14(a) and 15 to 16, examples of various screens displayed on the display device 23 of the user's terminal device 2 will be described below. In FIGS. 7 to 14(a) and 15 to 16, multiple components denoted by the same reference numerals are components having equivalent functions.

[0158] Fig. 7(a) is a diagram showing an example of a distribution screen 600 displayed on the display device 23 of the distribution user's terminal device 2 (distribution terminal device). The distribution screen 600 includes a distribution image 602 including a character object 601, a response information display area 603, an acquire button 604, and an end distribution button 605. Fig. 7(b) is a diagram showing an example of a distribution screen 610 displayed on the display device 23 of the viewing user's terminal device 2 (viewing terminal device). The distribution screen 610 includes a distribution image 612 including a character object 611, a rating button 613, an add button 614, a comment button 615, a response information display area 616, and an end viewing button 617.

[0159] In response to the start of a distribution event, the broadcasting user's terminal device 2 (broadcasting terminal device) executes a process of transmitting information related to a distribution start instruction to the server device 3 by the transmitting unit 272. The distribution event starts when the broadcasting user inputs a start instruction for the distribution event from the input device 24 to the terminal processing device 27 in response to an operation by the broadcasting user on a predetermined screen (not shown), such as a home screen, displayed on the display device 23. Next, the terminal device 2 (broadcast terminal device) simultaneously (in parallel) executes a process of transmitting character video data, etc. by the transmitting unit 272 and a process of displaying the distribution screen 600 by the display processing unit 274.

[0160] When the receiving unit 273 of the terminal device 2 (viewing terminal device) of the viewing user receives information indicating one or more broadcast users who are currently broadcasting, the display processing unit 274 of the terminal device 2 (viewing terminal device) displays a broadcast event selection screen (not shown) on the display device 23. The broadcast event selection screen includes option objects for selecting one or more broadcast users who are currently broadcasting. Next, the transmitting unit 272 of the terminal device 2 (viewing terminal device) acquires, from the input device 24, information indicating a broadcast user selected from the one or more broadcast users included in the broadcast event selection screen in response to the viewing user's operation of the input device 24. In this way, the transmitting unit 272 receives, from the input device 24, an instruction from the viewing user to start viewing the broadcast image. Then, the transmitting unit 272 transmits, to the server device 3 via the terminal communication I / F 21, information regarding the viewing start instruction including information indicating the selected broadcast user, information regarding the broadcast image, and the user ID of the viewing user stored in the terminal storage device 22.

[0161] The server receiving unit 331 of the server device 3 receives information regarding the viewing start instruction, information regarding the distributed image, and the user ID of the viewing user transmitted from the terminal device 2 (viewing terminal device) via the server communication I / F 31. Next, the server transmitting unit 332 of the server device 3 specifies the terminal device 2 (viewing terminal device) of the viewing user identified by the user ID received together with the information regarding the viewing start instruction. Next, the server transmitting unit 332 distributes, to the specified terminal device 2 (viewing terminal device), information for displaying the distributed image identified by the information regarding the distributed image received together with the information regarding the viewing start instruction, via the server communication I / F 31. Note that the information for displaying the distributed image is character video data of the broadcasting user, or character video data and audio data of the broadcasting user.

[0162] When the receiving unit 273 of the terminal device 2 (viewing terminal device) receives character video data, the display processing unit 274 of the terminal device 2 (viewing terminal device) displays the distribution screen 610 based on the received character video data. When the receiving unit 273 of the terminal device 2 (viewing terminal device) receives character video data and audio data, the display processing unit 274 of the terminal device 2 (viewing terminal device) displays the distribution screen 610 based on the received character video data, and the audio output unit 275 of the terminal device 2 (viewing terminal device) outputs the audio of the distribution user generated based on the audio data from a speaker (not shown). Note that the server receiving unit 331 of the server device 3 registers (stores in the server storage device 32) the viewing user ID of the viewing user of the terminal device 2 (viewing terminal device) that transmitted information related to the viewing start instruction as the viewing user ID of the viewing user currently viewing.

[0163] Each display element included in the distribution screen 600 will be described below.

[0164] The character object 601 included in the distribution screen 600 is an animation (moving image) of the broadcast user's character object generated by the display processing unit 274. The display processing unit 274 generates animation of the broadcast user's character object based on the generated motion data of the broadcast user and the character object associated with the broadcast user's user ID stored in the user table T1. If the motion data includes face motion data, the display processing unit 274 changes the facial expression of the character object 601 to synchronize with the broadcast user's facial movements. If the motion data includes body motion data, the display processing unit 274 moves the corresponding part of the character object 601 to synchronize with the broadcast user's body movements. The display processing unit 274 displays a distribution image 602 including the generated character object 601 on the display device 23.

[0165] The response information display area 603 included in the distribution screen 600 includes response information (comment information, rating information, and / or information related to additional objects) transmitted from each terminal device 2 (viewing terminal device) displaying the distribution screen 610.

[0166] The example of the response information display area 603 shown in Fig. 7(a) shows that comment information indicating the comment "cool" has been sent from the terminal device 2 (viewing terminal device) of a viewing user named "User R". Next, it shows that evaluation information has been sent from the terminal device 2 (viewing terminal device) of a viewing user named "User F". It also shows that evaluation information has most recently been sent from the terminal device 2 (viewing terminal device) of a viewing user named "User A".

[0167] The broadcast end button 605 included in the broadcast screen 600 is an object image such as a button object for ending the display of the broadcast screen 600 and ending the broadcast of the broadcast image 612. When the broadcast user selects the broadcast end button 605 by operating the input device 24, the display processing unit 274 of the broadcast user's terminal device 2 (broadcast terminal device) ends the display of the broadcast screen 600. Then, the transmission unit 272 of the terminal device 2 (broadcast terminal device) transmits information regarding the broadcast end instruction to the server device 3 via the terminal communication I / F 21, together with the broadcast user ID stored in the terminal storage device 22. When the server receiving unit 331 of the server device 3 receives the information regarding the broadcast end instruction transmitted from the broadcast user's terminal device 2 (broadcast terminal device), it deletes the broadcast user ID received together with the information regarding the broadcast end instruction from the server storage device 32. Furthermore, the server receiving unit 331 deletes the viewing user ID associated with the deleted broadcast user ID from the server storage device 32. This ends the broadcast of the broadcast image of the broadcast user who selected the broadcast end button 605. Note that the broadcast user ID of a broadcast user who has ended their broadcast may not be deleted from the server storage device 32. In this case, the broadcast user ID of a broadcast user who has ended their broadcast may be stored in the server storage device 32 as a broadcast end user ID. Also, the viewing user ID associated with the broadcast user ID of a broadcast user who has ended their broadcast may not be deleted from the server storage device 32. In this case, the viewing user ID associated with the broadcast user ID of a broadcast user who has ended their broadcast may be stored in the server storage device 32 as a viewing end user ID. Also, the server transmission unit 332 may send information regarding a broadcast end instruction to the terminal device 2 (viewing terminal device) of the viewing user identified by the viewing user ID associated with the broadcast user ID of the broadcast user who has ended their broadcast. In this case, the terminal device 2 (viewing terminal device) of the viewing user that is displaying the broadcast image of the broadcast user will also end displaying the broadcast screen 610 in response to receiving information regarding the broadcast end instruction.

[0168] Below, each display element included in the distribution screen 610 will be explained.

[0169] Character object 611 included in distribution screen 610 is an animation (moving image) of the broadcast user's character object generated by display processing unit 274. Display processing unit 274 generates animation of the broadcast user's character object based on the broadcast user's character moving image data (motion data) received by receiving unit 273 and the character object associated with the broadcast user's user ID stored in user table T1. An example of display processing of distribution image 612 including character object 611 will be described below.

[0170] When the server device 3 receives information regarding the instruction to start viewing the distribution image of the distribution user, information regarding the distribution image (distribution user ID of the distribution image), and the user ID of the viewing user from the terminal device 2 (viewing terminal device) of the viewing user, the server device 3 stores the received viewing user ID of the viewing user in association with the distribution user ID of the received distribution image.

[0171] If the broadcast user's character video data (motion data) received by the receiving unit 273 of the viewing user's terminal device 2 (viewing terminal device) includes the broadcast user's facial motion data, a broadcast image 612 is displayed, including a character object 611 whose facial expression changes in sync with the broadcast user's facial movements. For example, if the broadcast user makes a sound, a broadcast image 612 is displayed, including a character object 611 whose lips are moving. Furthermore, if the broadcast user's motion data received by the receiving unit 273 includes the broadcast user's body motion data, a broadcast image 612 is displayed, including a character object 611 whose corresponding body part moves in sync with the broadcast user's body movements. If the broadcast user's character video data and audio data are received, a broadcast image 612 based on the received character video data is displayed by the display processing unit 274, and the broadcast user's audio, generated based on the received broadcast user's audio data, is output as audio by the audio output unit 275. This allows user A to hear the broadcast user's audio while watching the broadcast user's character object 611, which is moving as if making a sound. This completes the display process for the delivered image 612. The display process for the delivered image 612 is executed every time output information from a broadcast user is received at predetermined time intervals (for example, every two seconds).

[0172] Rating button 613 included on distribution screen 610 is an object image for instructing the transmission of rating information regarding character object 611 included on distribution screen 610, or the transmission of rating information regarding the broadcast user associated with character object 611. When rating button 613 is selected in response to the viewing user's operation of input device 24, transmission unit 272 of terminal device 2 (viewing terminal device) transmits rating information indicating that a "positive rating has been given to character object 611 included on distribution screen 610" or rating information indicating that a "positive rating has been given to the broadcast user associated with character object 611 included on distribution screen 610" to server device 3 via terminal communication I / F 21. Server receiving unit 331 of server device 3 may store the received rating information in server storage device 32, and server transmitting unit 332 of server device 3 may transmit the received rating information to terminal device 2 (distribution terminal device) of the broadcast user.

[0173] Note that rating button 613 may be an object image for instructing the transmission of rating information indicating that a "negative rating has been given to character object 611 included in distribution screen 600" or rating information indicating that a "negative rating has been given to a broadcast user related to character object 611 included in distribution screen 600." Furthermore, distribution screen 610 may include multiple types of rating buttons 613 (for example, two types of rating buttons: a rating button for sending positive rating information and a rating button for sending negative rating information).

[0174] When the distribution screen 610 is displayed on a plurality of terminal devices 2 (viewing terminal devices), aggregated information that aggregates the rating information transmitted from each terminal device 2 (viewing terminal device) may be included on the distribution screen 600 and / or the distribution screen 610. For example, the server device 3 may count the number of rating information transmitted from each of the plurality of terminal devices 2 (viewing terminal devices) and generate display information for including the number of rating information on the distribution screen 600 and / or the distribution screen 610. The aggregated information is not limited to the number of rating information, but may also be information regarding the time period in which the rating information was transmitted. The aggregation period may be the period from the start of the distribution event to the current time, the period from the start of displaying the distribution screen 600 to the current time, the period from the start of distribution of the distribution screen 610 to the current time, or a predetermined period (for example, the period from 9:00 PM to midnight). Multiple aggregation periods may also be set. When the server device 3 can transmit rating information to the terminal device 2 (distribution terminal device) of the broadcasting user, the aggregation process of the rating information may be performed by the terminal device 2 (distribution terminal device). In this case, the process of including the aggregate information in the distribution screen 600 may be performed by the terminal device 2 (distribution terminal device), and the process of including the aggregate information in the distribution screen 610 may be performed by the terminal device 2 (viewing terminal device) displaying the distribution screen 610.

[0175] The add button 614 included in the distribution screen 610 is an object image for instructing the display of a selection screen 700 for selecting an additional object to be displayed on the distribution screen 600 and the distribution screen 610. When the add button 614 is selected in response to an operation of the input device 24 by the viewing user, the selection screen 700 is displayed on the terminal device 2 (viewing terminal device) of the viewing user. Details of the selection screen 700 will be described later.

[0176] The comment button 615 included in the distribution screen 610 is an object image for instructing the display of a comment input window (not shown) for inputting a comment (character string) to be displayed on the distribution screen 600 and the distribution screen 610. When the comment button 615 is selected in response to an operation of the input device 24 by the viewing user, the comment input window (not shown) is superimposed on the distribution screen 610.

[0177] The comment input window is displayed only on the display device 23 of the terminal device 2 of the viewing user who selected the comment button 615, and the character string being input into the comment input window is not displayed on the display device 23 of other terminal devices 2. For example, the display processing unit 274 of the terminal device 2 of the viewing user creates comment information indicating the comment (character string) input into the comment input window in response to the viewing user's operation of the input device 24. Comment information is a type of response information.

[0178] The response information display area 616 included in the distribution screen 610 includes response information (comment information, rating information, and / or information regarding additional objects (described below)) created in response to operations by the viewing user, as well as response information sent from the terminal devices 2 of other viewing users who are viewing the distribution image 612 of the same distribution user.

[0179] The example of the response information display area 616 shown in Fig. 7(b) shows that comment information indicating the comment (character string) "Cool" was sent from the terminal device 2 of a viewing user named "User R." It also shows that evaluation information was sent from the terminal device 2 of a viewing user named "User F." It also shows that evaluation information created by the terminal device 2 of User A who operates this terminal device 2 was sent.

[0180] When response information is created by the viewing user of the terminal device 2 and a transmission instruction is input, the transmitting unit 272 of the terminal device 2 displaying the distribution screen 610 transmits the created response information together with information relating to the instruction to display the response information to the server device 3 via the terminal communication I / F 21. Note that the viewing user ID of the viewing user stored in the terminal storage device 22 is transmitted together with the information relating to the instruction to display the response information.

[0181] The server receiving unit 331 of the server device 3 receives information regarding a display instruction for response information transmitted from the terminal device 2 displaying the distribution screen 610, along with the response information and the viewing user ID. The server transmitting unit 332 of the server device 3 extracts the broadcast user ID stored in association with the received viewing user ID from the server storage device 32. The server transmitting unit 332 then transmits the received response information and viewing user ID to the terminal device 2 of the broadcast user identified by the extracted broadcast user ID and the terminal device 2 of the viewing user identified by the extracted viewing user ID. The receiving unit 273 of each terminal device 2 displaying the distribution screen 610 receives the response information and viewing user ID transmitted from the server device 3 via the terminal communication I / F 21. The display processing unit 274 of each terminal device 2 displaying the distribution screen 610 displays the distribution screen 610 in a response information display area 616, chronologically arranging the response information and the name of the viewing user who input the response information, based on the received response information and the name associated with the received viewing user ID. The name associated with the received user ID is information extracted by the display processing unit 274 by referring to the user table T1. In this way, the display processing unit 274 of each terminal device 2 displaying the distribution screen 610 displays the received response information and the name of the viewing user who input the response information in chronological order in the response information display area 616 every time response information is received.

[0182] An upper limit may be set for the number of pieces of response information that can be displayed in the response information display area 603 and / or the response information display area 616. In the example of the response information display area 603 shown in Fig. 7(a) and the response information display area 616 shown in Fig. 7(b), the upper limit for the number of pieces of response information that can be displayed is "3." If the number of pieces of response information displayed on each terminal device 2 exceeds the set upper limit, the display processing unit 274 of each terminal device 2 displays the distribution screen 600 in which the three pieces of newly received response information are included in the response information display area 603, or the distribution screen 610 in which the three pieces of newly received response information are included in the response information display area 616, in order.

[0183] Note that the response information in the response information display area 616 may be displayed in a scrollable manner. In this case, in response to a swipe operation by the user (for example, an operation in which a finger touches the screen and moves from the bottom to the top of the screen), response information that is not currently included in the response information display area 616 is scrolled and displayed within the response information display area 616. The response information in the response information display area 603 included in the delivery screen 600 may also be displayed in a scrollable manner, similar to the response information display area 616.

[0184] The viewing end button 617 is an object image, such as a button object, for ending the display of the distribution screen 610 and ending viewing of the distribution image 612. When the viewing user selects the viewing end button 617 by operating the input device 24, the display processing unit 274 of the viewing user's terminal device 2 ends the display of the distribution screen 610. Then, the transmission unit 272 of the terminal device 2 transmits information regarding the viewing end instruction to the server device 3 via the terminal communication I / F 21, together with the viewing user ID and destination information stored in the terminal storage device 22. When the server receiving unit 331 of the server device 3 receives the information regarding the viewing end instruction and the viewing user ID transmitted from the user's terminal device 2, the server receiving unit 331 stores, as a viewing end user ID, the viewing user ID that is identical to the received viewing user ID and associated with the broadcast user ID of the broadcast user associated with the ended distribution screen 610 in the server storage device 32. Note that the viewing end user ID is stored in association with the broadcast user ID of the broadcast user associated with the ended distribution screen 610. This ends viewing of the distributed image by the viewing user who selected the viewing end button 617.

[0185] 8(a) is a diagram showing an example of a selection screen 700 displayed on the display device 23 of the terminal device 2 (viewing terminal device) of the viewing user. As shown in FIG. 8(a), the selection screen 700 is displayed so as to be superimposed on the distribution screen 610. The selection screen 700 may be displayed later while the display of the distribution screen 610 is temporarily suspended, and the distribution screen 610 may be redisplayed when the display of the selection screen 700 ends.

[0186] In the example shown in FIG. 8(a), images representing eight types of candidate objects for addition are included in the display area of the selection screen 700. Names indicating the candidate objects for addition may be arranged near the images representing the candidate objects for addition. The number of candidate objects for addition included in the display area of the selection screen 700 is not limited to eight, and may be seven or less, or nine or more. When some of the images representing the candidate objects for addition cannot be included in the display area of the selection screen 700, the display processing unit 274 of the terminal device 2 may display the images representing the candidate objects for addition in a scrollable manner. In this case, in response to a swipe operation by the viewing user (for example, an operation in which a finger is moved from right to left while touching the screen), images representing candidate objects for addition that are not currently included in the display area of the selection screen 700 are scrolled and displayed in the display area of the selection screen 700.

[0187] Each of the additional candidate objects represents an additional object. Images representing the additional candidate objects are, for example, icon images, thumbnail images, etc., representing gift objects, which are a type of additional object. Note that, on the selection screen 700, images of each additional candidate object may be displayed categorized by type of additional object (free gifts, paid gifts, accessories, support goods, appealing gifts, variety gifts, cooperation gifts that require the cooperation of users belonging to a group, etc.). Note that additional objects belonging to paid gifts are additional objects that a user can purchase by spending virtual currency value "coins." Note that additional objects belonging to free gifts are additional objects that can be obtained without spending currency value, by spending points that can be acquired by, for example, watching a broadcast. Note that the additional object may be an object that a viewing user can send to a broadcasting user by playing and clearing a predetermined mission executed on the terminal device 2. Note that the additional object may be an object that a viewing user can send to a broadcasting user by playing a known lottery game executed on the terminal device 2 and winning the lottery game. Note that the additional object may be managed (stored) as an object owned by the viewing user.

[0188] When an addition candidate object is selected in response to an operation of the input device 24 by the viewing user, the input device 24 inputs an addition instruction together with the object ID corresponding to the selected addition candidate object to the terminal processing device 27. An example of an addition process in response to an addition instruction will be described below.

[0189] First, when the display processing unit 274 acquires the input addition instruction and object ID, it references the object table T2 and extracts a still image or video image (image information) of the added object associated with the acquired object ID from the terminal storage device 22. Next, the display processing unit 274 generates a new delivery image 612 that includes the extracted still image or video image in the delivery image 612 that includes the delivery user's character object 611.

[0190] Next, upon acquiring the input addition instruction and object ID, the transmitting unit 272 transmits information indicating the addition instruction, together with the acquired object ID, the viewing user ID of the viewing user stored in the terminal storage device 22, and information related to the distributed image (distributor user ID of the distributed image), to the server device 3 via the terminal communication I / F 21. Next, the server receiving unit 331 of the server device 3 receives the information indicating the addition instruction, the object ID, the viewing user ID, and information related to the distributed image (distributor user ID of the distributed image) via the server communication I / F 31. Note that if information related to the distributed image (distributor user ID of the distributed image) is not transmitted, the server transmitting unit 332 of the server device 3 extracts a broadcast user ID stored in association with the received viewing user ID from the server storage device 32. The server transmitting unit 332 extracts viewing user IDs other than the received viewing user ID from information related to the distributed image (distributor user ID of the distributed image) or viewing user IDs associated with the extracted broadcast user ID. Then, the server transmitting unit 332 transmits information indicating the received addition instruction, the object ID, and the received user ID (user ID of the viewing user who issued the addition instruction) to the terminal device 2 of the viewing user indicated by the extracted viewing user ID, via the server communication I / F 31. In addition, the server transmitting unit 332 transmits information regarding the distributed image (distributor user ID of the distributed image) or information indicating the received addition instruction, the object ID, and the received viewing user ID to the terminal device 2 of the distributor user indicated by the extracted distributor user ID, via the server communication I / F 31.

[0191] Then, the receiving units 273 of the terminal devices 2 of the viewing users (excluding the viewing user who issued the addition instruction) and the broadcast user receive information indicating the addition instruction, the user ID of the viewing user who issued the addition instruction, and the object ID via the terminal communication I / F 21. Next, the display processing unit 274 of the terminal device 2 references the object table T2 and extracts a still image or video image (image information) of the additional object associated with the received object ID from the terminal storage device 22. Next, the display processing unit 274 generates a new broadcast image 612 that includes the extracted still image or video image in the broadcast image 602, 612 that includes the broadcast user's character object 601, 611. Note that response information including the name of the viewing user who issued the addition instruction and the name of the additional object associated with the received object ID may be newly displayed in the new broadcast image 602, 612. This completes the addition process.

[0192] 8(b) is a diagram showing another example of a distribution screen 710 displayed on the display device 23 of the terminal device 2. Similar to the distribution screen 610, the distribution screen 710 includes a distribution image 612 including a character object 611, a rating button 613, an add button 614, a comment button 615, a response information display area 616, and an end viewing button 617.

[0193] Unlike the distribution screen 610, the distribution screen 710 includes an additional object 711 and an additional object 712. The additional object 711 and the additional object 712 are both still images or moving images showing the additional objects. An example of the display process for the additional object 712 will be described below, taking as an example the case where the viewing user who instructs to add the additional object 712 is user A.

[0194] When an addition candidate object (corresponding to the additional object 712) is selected in response to user A's operation of the input device 24, the input device 24 of user A's terminal device 2 inputs an addition instruction to the terminal processing device 27 together with the object ID corresponding to the selected addition candidate object. Upon receiving the input addition instruction and object ID, the display processing unit 274 of user A's terminal device 2 references the object table T2 and extracts a still image or video (image information) of the additional object associated with the obtained object ID from the terminal storage device 22. Then, the display processing unit 274 of user A's terminal device 2 generates a new delivery image 612 that includes the broadcast user's character object 611 and adds the additional object 712 based on the extracted still image or video.

[0195] The transmitting unit 272 of user A's terminal device 2 transmits information indicating the addition instruction for the additional object 712, the acquired object ID, and user A's viewing user ID stored in the terminal storage device 22 to the viewing users (excluding the viewing user who issued the addition instruction) and the broadcast user's terminal device 2 via the server device 3. When the information indicating the addition instruction for the additional object 712, user A's viewing user ID, and object ID are received by the receiving units 273 of the viewing users (excluding user A) and the broadcast user's terminal device 2, the display processing unit 274 of the viewing users (excluding user A) and the broadcast user's terminal device 2 references object table T2 and extracts a still image or video (image information) of the additional object associated with the received object ID from the terminal storage device 22. The display processing unit 274 of the terminal device 2 then generates a new distributed image 612 based on the extracted still image or video in the distributed image 612 including the broadcast user's character object 611. This completes the display processing for the additional object 712.

[0196] When a predetermined change condition is satisfied, the shape, size, arrangement, color, brightness, and / or design of the additional object added to the distribution screen may change while the distribution screen is being displayed. The predetermined change condition may be a condition related to information indicating an instruction to add an additional object. For example, the predetermined change condition may be that a condition related to the timing of an instruction to add an additional object by any of all viewing users viewing the same distribution screen is satisfied, and / or that at least two of all viewing users viewing the same distribution screen have issued instructions to add multiple specific types of additional objects that are different from each other. For example, the condition related to the timing of the instruction to add an additional object being satisfied means that the difference in the timing of the instructions to add from multiple viewing users is within a predetermined period. In response to an instruction to add an additional object by a viewing user, the server device 3 awards the viewing user with predetermined points. When the predetermined change condition is satisfied, the viewing user who issued the instruction to add an additional object that satisfies the predetermined change condition may be awarded more points than the predetermined points. The predetermined change condition may be that a condition relating to the timing of an instruction to add by any of a plurality of viewing users having a predetermined relationship is satisfied, and / or that a plurality of viewing users having a predetermined relationship have issued instructions to add a plurality of specific types of additional objects that are different from one another. In this case, even if a condition relating to the timing of an instruction to add by a viewing user who does not have a predetermined relationship is satisfied, the predetermined change condition is not satisfied.

[0197] A predetermined relationship among multiple viewing users means, for example, that one viewing user among the multiple viewing users is a mutual follower of another viewing user among the multiple viewing users. Note that if one user follows another user and the other user also follows the one user (the one user and the other user are mutually following each other), the one user and the other user are mutual followers. Furthermore, a predetermined relationship among multiple viewing users may mean that one viewing user among the multiple viewing users is in a mutual follow relationship with a predetermined viewing user who is mutually following another viewing user among the multiple viewing users.

[0198] Furthermore, the predetermined relationship between multiple viewing users may be that the multiple viewing users are friends. Furthermore, the predetermined relationship between multiple viewing users may be that specific information (such as a phone number, email address, or predetermined ID (identification)) of another viewing user among the multiple viewing users is stored in a terminal device of one of the multiple viewing users. Furthermore, the predetermined relationship between multiple viewing users may be a relationship in which the number of times the multiple viewing users have exchanged information using the same information exchange service is equal to or greater than a predetermined number. Furthermore, the predetermined relationship between multiple viewing users may be a relationship in which the number of times each of the multiple viewing users has participated in the same and / or different information exchange services is equal to or greater than a predetermined number.

[0199] Furthermore, the predetermined relationship among multiple viewing users may be a relationship regarding operations of other viewing users among the multiple viewing users when one of the multiple viewing users distributes a distribution image as a distribution user. Here, the relationship regarding operations of other viewing users may be a relationship in which the other viewing users have viewed the distribution image distributed by the one viewing user a predetermined number of times or more. Furthermore, the relationship regarding operations of other viewing users may be a relationship in which the other viewing users have sent comment information regarding the distribution image distributed by the one viewing user a predetermined number of times or more.

[0200] Furthermore, the predetermined relationship between multiple viewing users may be a relationship regarding a predetermined event in which each of the multiple viewing users is participating. Here, the relationship regarding the predetermined event is a relationship in which each of the multiple viewing users is currently participating in the same event. The relationship regarding the predetermined event may also be a relationship in which each of the multiple viewing users has participated in the same event in the past. The relationship regarding the predetermined event may also be a relationship in which another viewing user among the multiple viewing users has participated in an event in which one or more viewing users among the multiple viewing users are participating in.

[0201] When a change button (not shown) on the distribution image 602 is selected by the broadcasting user operating the input device 24, a selection screen 800 is displayed. For example, the change button is included in the distribution screen 600. The change button may also be included in a predetermined screen, such as a home screen, displayed on the display device 23. FIG. 9(a) is a diagram showing an example of the selection screen 800 displayed on the display device 23 of the broadcasting user's terminal device 2 (broadcast terminal device). The selection screen 800 shown in FIG. 9(a) is displayed superimposed on the distribution screen 600. The selection screen 800 may also be displayed after the display of the distribution screen 600 has ended.

[0202] The selection screen 800 includes a plurality of placement candidate objects 801 indicating each of a plurality of placement locations, and attachment candidate objects 802 which are candidates for attachment objects to be newly attached to the character object 601. When one of the plurality of placement candidate objects 801 is selected by the broadcasting user operating the input device 24, an attachment candidate object 802 corresponding to the object ID of an object owned by the broadcasting user which is associated with the placement candidate corresponding to the selected placement candidate object 801 is displayed on the selection screen 800. When one of the plurality of attachment candidate objects 802 is selected by the broadcasting user operating the input device 24, the input device 24 inputs a change instruction to the terminal processing device 27, and the display processing unit 274 executes the change processing described above. Then, the display processing unit 274 displays a distribution screen 810 ( FIG. 9( b) ) including a distribution image 812 including a character object 811 wearing the new attachment object.

[0203] 9(b) is a diagram showing another example of a broadcast screen 810 displayed on the display device 23 of the broadcast user's terminal device 2 (broadcast terminal device). The broadcast screen 810 includes at least a broadcast image 812 including the broadcast user's character object 811 wearing a new attachment object. In the example shown in FIG. 9(b), a character object 811 is displayed in which the clothing worn by the broadcast user's character object 601 has been changed.

[0204] Information regarding the change instruction input by a broadcasting user who has started a video broadcasting service is transmitted via server device 3 to terminal device 2 (viewing terminal device) of the viewing user who is viewing the broadcast image of that broadcasting user. As a result, a change process according to the change instruction input by the broadcasting user is also executed on terminal device 2 (viewing terminal device) of the viewing user. As a result, a broadcast image including the broadcasting user's character object equipped with the new attachment object is displayed on terminal device 2 (viewing terminal device) of the viewing user.

[0205] 8 and 9, at least a portion of the broadcast image including the broadcaster's character object is changed in response to an additional instruction from the viewing user, and at least a portion of the broadcaster's character object is changed in response to a change instruction from the broadcaster. This prevents viewing users from becoming bored during the broadcast event and increases users' motivation to continue using the video broadcast service.

[0206] Furthermore, in conventional information processing systems, after starting a distribution event for a viewing user via a character object, a broadcasting user had to interrupt the distribution event in order to change the distribution image including the character object. After the distribution event was interrupted, the broadcasting user had to activate a function to change the appearance of the character object and / or the background image in the distribution image including the character object. The broadcasting user then had to use the change function to change the character object and / or the distribution image, and then resume the distribution event, a cumbersome process. In contrast, the information processing system 1 of the present embodiment allows the broadcasting user to change the character object of the broadcasting user without interrupting the distribution event and resetting the character object of the broadcasting user. As a result, the information processing system 1 of the present embodiment does not require a complex user interface for changing the broadcasting user's character object during a distribution event, thereby improving usability compared to conventional information processing systems. Furthermore, the information processing system 1 of the present embodiment allows the user's character object to be changed without interrupting the distribution event, thereby reducing the number and volume of communications between the server device 3 and the terminal device 2. In this way, the information processing system 1 of the present embodiment allows for improved communication between the server device 3 and the terminal device 2.

[0207] FIG. 10(a) is a diagram showing an example of a group setting screen 900 displayed on the display device 23 of the user's terminal device 2. The group setting screen 900 is displayed in response to a user's operation on a predetermined screen, such as a home screen, displayed on the display device 23. A user who can operate a terminal device 2 capable of displaying the group setting screen 900 is a broadcasting user or a viewing user. A user who can operate a terminal device 2 capable of displaying the group setting screen 900 is not limited to a broadcasting user or a viewing user, and may be any user who is able to receive services provided by the server device 3. A new information exchange group is set in response to a user's operation on the group setting screen 900.

[0208] As shown in FIG. 10(a), the group setting screen 900 includes a candidate display area 901, a selection object 902, and a start button 903. The candidate display area 901 displays information indicating one or more candidate users who can belong to the information exchange group. For example, the candidate users who can belong to the information exchange group are users who are mutual followers of the user operating the terminal device 2 that displayed the group setting screen 900. Alternatively, the candidate users who can belong to the information exchange group may be users who are friends with the user operating the terminal device 2 that displayed the group setting screen 900. The candidate users who can belong to the information exchange group are not limited to the above, and may be any users who have a predetermined relationship with the user operating the terminal device 2 that displayed the group setting screen 900. In the example shown in FIG. 10(a), the display processing unit 274 of the user's terminal device 2 displays thumbnail images and names of character objects of multiple users who can belong to the information exchange group in the candidate display area 901 as information indicating the candidate users.

[0209] The selection objects 902 are operation objects for selecting each candidate user displayed in the candidate display area 901. In the example shown in FIG. 10(a), the display processing unit 274 of the user's terminal device 2 displays check box objects corresponding to each candidate user as the selection objects 902. For example, when the selection object 902 is selected by the user's operation of the input device 24, the selection object 902 is displayed with a "check" mark. In this case, the candidate user corresponding to the selection object 902 with the "check" mark is selected. Furthermore, when the selection object 902 with the "check" mark is selected by the user's operation of the input device 24, the selection object 902 is displayed without the "check" mark. In this case, the candidate user corresponding to the selection object 902 with the "check" mark is not selected. Each of the selection objects 902 is associated with the user ID of the corresponding candidate user.

[0210] The start button 903 is a button object or the like that allows the user to create a new information exchange group and start exchanging information with other users who belong to the newly created information exchange group. When the user selects the start button 903 by operating the input device 24, a new information exchange group is created to which the user operating the terminal device 2 and the candidate users selected by the selection object 902 belong as affiliated users. For example, the display processing unit 274 of the user's terminal device 2 identifies the user ID of the user stored in the terminal storage device 22 and the user IDs of the candidate users associated with the selection object 902 selected by the selection object 902. Then, the display processing unit 274 associates the user ID of the identified user and the user IDs of the candidate users with the newly created group ID as affiliated users and stores them in the group table T3. At this time, the display processing unit 274 may associate the name of the information exchange group created automatically with the newly created group ID, or may associate the name of the information exchange group input by the user with the newly created group ID. When a new information exchange group is created in response to selection of the start button 903, the display processing unit 274 displays an information exchange screen 1000 for users belonging to the new information exchange group.

[0211] FIG. 10(b) is a diagram showing an example of an information exchange start screen 910 displayed on the display device 23 of the user's terminal device 2. The information exchange start screen 910 is displayed in response to a user's operation on a predetermined screen, such as a home screen, displayed on the display device 23. A user who can operate a terminal device 2 capable of displaying the information exchange start screen 910 is a broadcasting user or a viewing user. A user who can operate a terminal device 2 capable of displaying the information exchange start screen 910 is not limited to a broadcasting user or a viewing user, and may be any user who is capable of receiving services provided by the server device 3. In response to a user's operation on the information exchange start screen 910, information exchange by the selected information exchange group is started.

[0212] 10(b), the information exchange start screen 910 includes a candidate group area 911. The candidate group area 911 displays information indicating an information exchange group to which the user operating the terminal device 2 displaying the information exchange start screen 910 belongs. The information indicating the information exchange group can be selected by the user, and the information indicating the information exchange group is associated with the group ID of the corresponding information exchange group.

[0213] When the user operates the input device 24 to select information indicating one of the information exchange groups included in the information exchange start screen 910, the group ID associated with the information indicating the selected information exchange group is identified. The display processing unit 274 displays the information exchange screen 1000 for each user identified by the user ID associated with the identified group ID (each user belonging to the information exchange group indicated by the identified group ID).

[0214] 11(a) is a diagram showing an example of an information exchange screen 1000 displayed on the display device 23 of the user's terminal device 2. On the information exchange screen 1000, information exchange is carried out among a plurality of users belonging to an information exchange group newly set up by the user selecting the start button 903 included in the group setting screen 900. Alternatively, on the information exchange screen 1000, information exchange is carried out among a plurality of users belonging to an information exchange group selected by the user on the information exchange start screen 910. Hereinafter, among the plurality of users belonging to the information exchange group performing information exchange, users other than the user operating the terminal device 2 displaying the information exchange screen 1000 may be referred to as "other users." The information exchange screen 1000 (and each information exchange screen described later) may be realized by a so-called chat screen or the like that realizes text chat or the like.

[0215] The information exchange screen 1000 includes other user input information 1001, other user information 1002, a text input object 1003, a candidate display button 1004, and user input information 1005. The other user input information 1001 is input information input by another user. The other user information 1002 is information indicating another user (such as an icon image of a character object) that is displayed in association with each other user input information 1001. The text input object 1003 is an operation object that allows the user to input text information as input information. The candidate display button 1004 is an operation object that displays candidate information 1101 corresponding to a character-related image so that the user can select it. The user input information 1005 is input information input by the user.

[0216] Each time input information of another user transmitted from the terminal device 2 of the other user is received via the server device 3, the display processing unit 274 displays other user input information 1001 indicating the received input information of the other user in the other user display area of the information exchange screen 1000. For example, the other user display area is an area near the left side of the information exchange screen 1000, and the other user input information 1001 is displayed in the other user display area in chronological order from the oldest to the newest, from top to bottom of the screen. Each time user input information is input by the user's input operation on the text input object 1003, the display processing unit 274 displays user input information 1005 indicating the input user input information in the user display area of the information exchange screen 1000. For example, the user display area is an area near the right side of the information exchange screen 1000, and the user input information 1005 is displayed in the user display area in chronological order from the oldest to the newest, from top to bottom of the screen.

[0217] When the information exchange screen 1000 related to the newly set information exchange group is displayed, the transmitting unit 272 of the user's terminal device 2 transmits information related to the new information exchange group to the server device 3 via the terminal communication I / F 21. The information related to the new information exchange group includes the group ID of the new information exchange group, the name of the new information exchange group, and the user IDs of the users who belong to the new information exchange group.

[0218] The server receiving unit 331 of the server device 3 receives information related to the information exchange group via the server communication I / F 31. The server receiving unit 331 stores the received information related to the new information exchange group in the server storage device 32. For example, the server receiving unit 331 stores the name of the new information exchange group and the user IDs of the users belonging to the new information exchange group in the server storage device 32, in association with the group ID of the new information exchange group.

[0219] The server transmission unit 332 of the server device 3 transmits information regarding the new information exchange group via the server communication I / F 31 to the terminal devices 2 of other users who belong to the new information exchange group and are identified by the user IDs of users other than the user who transmitted the information regarding the new information exchange group.

[0220] When the receiving unit 273 of the other user's terminal device 2 receives information about the new information exchange group transmitted from the user's terminal device 2 via the server device 3, it stores the name of the new information exchange group and the user IDs of the users belonging to the new information exchange group in association with the group ID of the new information exchange group in the group table T3. Note that when the display processing unit 274 of the other user's terminal device 2 receives information about the new information exchange group, it may automatically display the information exchange screen 1000 related to the new information exchange group.

[0221] On the information exchange screen 1000, when new text information is input as input information into the text input object 1003 by a user operation, the display processing unit 274 displays the input new text information in the user display area as user input information 1005. On the information exchange screen 1000, when image information, video information, or the like is input as input information by a user operation, the display processing unit 274 displays the input image information, video information, or the like in the user display area as user input information 1005. The information input as input information is not limited to text information, image information, and video information, and may be information in a format that can be used by a predetermined application program.

[0222] When input information entered by another user's operation on the information exchange screen 1000 displayed by the other user's terminal device 2 is received via the server device 3 together with the other user's user ID, the received input information is displayed in the other user display area as other user input information. In this case, an image or the like related to the received user ID of the other user is displayed as other user information 1002.

[0223] The transmitting unit 272 of the user's terminal device 2 transmits the input new text information and the user's user ID to the terminal device 2 of the other user via the server device 3. As a result, the input new text information is also displayed as other user input information 1001 on the information exchange screen 1000 displayed on the terminal device 2 of the other user who is a user belonging to the user other than the user who input the text information.

[0224] When the user operates the input device 24 to select the candidate display button 1004 on the information exchange screen 1000, the input device 24 sends to the terminal processing device 27 an instruction to display candidate information for selecting a character-related image. Next, the display processing unit 274 determines whether a character-related image including the user's character object has been generated in the past and stored in the terminal storage device 22. If a character-related image including the user's character object has not been stored, the display processing unit 274 displays a generation instruction object 1006 for instructing the generation of a character-related image. Furthermore, if a character-related image including the user's character object has been stored, for example, the display processing unit 274 displays the information exchange screen 1100 including the four types of stored candidate information 1101.

[0225] 11(b) is a diagram showing an example of an information exchange screen 1010 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 1010 is a screen that is displayed when the candidate display button 1004 included in the information exchange screen 1000 is selected by the user and it is determined that a character-related image including the user's character object is not stored in the terminal storage device 22.

[0226] Similar to the information exchange screen 1000, the information exchange screen 1010 includes other user input information 1001, other user information 1002, a text input object 1003, a candidate display button 1004, and user input information 1005. Furthermore, the information exchange screen 1010 includes a generation instruction object 1006. In the example shown in FIG. 11(b), the information exchange screen 1010 displays information prompting the generation of a character-related image together with the generation instruction object 1006.

[0227] When a generation instruction object 1006 is selected on the information exchange screen 1010 by a user operating the input device 24, the input device 24 sends an instruction to generate a character-related image to the terminal processing device 27. When the generation unit 271 receives the instruction to generate a character-related image from the input device 24, it executes a process to generate the character-related image. An example of the process to generate a character-related image will be described below.

[0228] First, the generation unit 271 extracts from the user table T1 the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 the motion data for the character-related image and the object data for the character-related image. Next, the generation unit 271 generates a character-related image including a character object with a predetermined motion and facial expression, and object data such as a background and text, based on the read model data, the image information and placement location of the used object, and the motion data for the character-related image and the object data for the character-related image. Then, the generation unit 271 associates the generated character-related image with a related image ID for uniquely identifying the character-related image and stores the associated image in the terminal storage device 22. The generation unit 271 also generates candidate information indicating the generated character-related image, associates the generated candidate information with a related image ID for identifying the character-related image, and stores the associated information in the terminal storage device 22. The generation unit 271 also associates the object ID of the object used when generating the character-related image with the related image ID and stores the associated image ID in the terminal storage device 22, and the character-related image generation process ends. Note that some or all of the generated character-related images may be set so as not to be usable in the information exchange service. In this case, a usable group made up of character-related images usable in the information exchange service may be set. Then, when a new character-related image usable in the information exchange service is generated, the new character-related image is set so as to be included in the usable group.

[0229] An example of a character-related image transmission process will be described below. First, the transmission unit 272 of the terminal device 2 that generated the character-related image transmits the character-related image, candidate information, and object ID of the used object associated with the related image ID stored in the terminal storage device 22 to the server device 3 via the terminal communication I / F 21. The user ID stored in the terminal storage device 22 is transmitted to the server device 3 together with the character-related image, candidate information, and object ID of the used object associated with the related image ID. When the server receiving unit 331 of the server device 3 receives the character-related image, candidate information, and object ID of the used object associated with the related image ID, and the user ID, it stores the related image ID in association with the user ID, and also stores the character-related image, candidate information, and object ID of the used object in association with the related image ID in the server storage device 32. The server transmitting unit 332 then transmits the received related image ID associated with the user's user ID, and the character-related image, candidate information, and object ID of the used object associated with the related image ID, via the server communication I / F 31, to the terminal devices 2 of other users who belong to the information exchange group to which the user of the terminal device 2 that transmitted the character-related image, etc., belongs. The terminal devices 2 of the other users store the transmitted related image ID associated with the user ID, and the character-related image, candidate information, and object ID of the used object associated with the related image ID, in the terminal storage device 22. This completes the character-related image transmission process. Note that the transmitting unit 272 of the terminal device 2 that generated the character-related image does not need to transmit the candidate information and object ID of the used object associated with the related image ID stored in the terminal storage device 22 to the server device 3. Furthermore, the candidate information and object ID of the used object associated with the user ID of a user who belongs to the same information exchange group do not need to be transmitted to the terminal devices 2 of other users who belong to the same information exchange group.Furthermore, the timing at which a character-related image associated with the user ID of a user belonging to the same information exchange group is transmitted to the terminal devices 2 of other users belonging to the same information exchange group may be the timing at which the user selects the character-related image as input information. The character-related image transmission process may be performed every time the character-related image generation process and / or additional generation process is performed.

[0230] The display processing unit 274 reads out the candidate information generated by the generation unit 271 from the terminal storage device 22 and displays it. Fig. 12(a) is a diagram showing an example of an information exchange screen 1100 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 1100 is a screen that is displayed when the generation instruction object 1006 included in the information exchange screen 1010 is selected by the user.

[0231] Like the information exchange screen 1000, the information exchange screen 1100 includes other user input information 1001, other user information 1002, a text input object 1003, a candidate display button 1004, and user input information 1005. Furthermore, the information exchange screen 1010 includes four types of generated candidate information 1101 and an additional generation button 1102. The number of types of candidate information 1101 included in the information exchange screen 1100 is not limited to four, and may be three or fewer, or five or more. When some of the multiple types of candidate information 1101 cannot be included within the display area of the information exchange screen 1100, the display processing unit 274 of the terminal device 2 may display the candidate information 1101 in a scrollable manner. In this case, in response to a swipe operation by the user (for example, an operation in which a finger is moved from right to left while touching the screen), candidate information 1101 that is not currently included within the display area of the information exchange screen 1100 is scrolled and displayed within the display area of the information exchange screen 1100.

[0232] When one piece of candidate information is selected from the four pieces of candidate information 1101 in response to the user's operation of the input device 24 (FIG. 12(b)), the input device 24 sends the related image ID associated with the selected candidate information to the terminal processing device 27. Next, the display processing unit 274 extracts, from the terminal storage device 22, a character-related image identified by the related image ID received from the input device 24, and displays the extracted character-related image in the user display area of the information exchange screen 1200 as user input information 1201 (FIG. 13). The character-related image displayed as the user input information 1201 is a two-dimensional still image or a three-dimensional still image depicting a three-dimensional virtual space including a character object. When the character-related image displayed as the user input information 1201 is generated based on multiple consecutive motion data, the character-related image may be a two-dimensional moving image depicting a character object moving for a predetermined time (e.g., three seconds). In this case, the character-related image may be a three-dimensional moving image depicting a three-dimensional virtual space including a character object moving for a predetermined time (e.g., three seconds). 13 , the display size of the character-related image displayed as user input information 1201 on the display screen of the display device 23 is smaller than the display size of the delivered image 602 on the display screen of the display device 23. Thus, the drawing processing load of drawing the character-related image on the display screen of the display device 23 is smaller than the drawing processing load of drawing the delivered image 602 on the display screen of the display device 23. Furthermore, because the character moving image data including the motion data for displaying the delivered image 602 is continuously transmitted, the transmission volume of the character moving image data is larger than the transmission volume of the related image ID for displaying the character-related image. In this way, by using character-related data in various services, it is possible to display a user's character object on another user's terminal device 2 while reducing the processing load of the terminal device 2 and / or the server device 3 and the transmission load between the terminal device 2 and the server device 3 compared to when a delivered image is used.

[0233] Along with the display of the user input information 1201 by the display processing unit 274, the transmission unit 272 transmits the related image ID received from the input device 24 and the user ID stored in the terminal storage device 22 to the server device 3 via the terminal communication I / F 21. The server transmission unit 332 of the server device 3 extracts, from the server storage device 32, the user IDs of other users associated with the same information exchange group as the user ID received by the server receiving unit 331. Furthermore, the server transmission unit 332 transmits, via the server communication I / F 31, the related image ID and user ID received by the server receiving unit 331 to the terminal device 2 of the other user identified by the extracted user ID.

[0234] Then, the receiving unit 273 of the terminal device 2 of the other user receives the related image ID and user ID transmitted from the server device 3 via the terminal communication I / F 21. Next, the display processing unit 274 of the terminal device 2 of the other user extracts a character-related image identified by the related image ID received by the receiving unit 273 from the terminal storage device 22, and displays the extracted character-related image in the other user display area of the distribution screen as other user input information. In this way, the same image as the character-related image used by the user is also displayed on the terminal device 2 of the other user associated with the information exchange group to which the user belongs. Note that if the character-related image identified by the received related image ID is not stored in the terminal storage device 22 by the display processing unit 274 of the terminal device 2 of the other user, the transmitting unit 272 of the terminal device 2 of the other user transmits an image transmission request to the server device 3. The image transmission request is transmitted together with the received related image ID and / or user ID, and the user ID of the other user. When the server receiving unit 331 receives an image transmission request, the server transmitting unit 332 of the server device 3 extracts, from the server storage device 32, a character-related image associated with the related image ID received together with the image transmission request. Next, the server transmitting unit 332 transmits the extracted character-related image together with the received related image ID and user ID to the terminal device 2 of the other user. The receiving unit 273 of the terminal device 2 of the other user receives the related image ID, character-related image, and user ID transmitted from the server device 3 via the terminal communication I / F 21, associates them with each other, and stores them in the terminal storage device 22. Then, the display processing unit 274 of the terminal device 2 of the other user displays the stored character-related image in the other user display area of the distribution screen as other user input information.

[0235] 12(a), when the user selects the additional generation button 1102 in response to an operation of the input device 24, the display processing unit 274 and the generation unit 271 of the user's terminal device 2 execute an additional generation process. The additional generation process is a process of generating a new character-related image based on reference information related to a character object representing the user. The reference information is information acquired by the reference information acquisition process, and includes, for example, information indicating the displayed delivery image.

[0236] An example of the reference information acquisition process executed by the broadcasting user's terminal device 2 will be described with reference to Fig. 14(a). Hereinafter, the broadcasting user's terminal device 2 may be referred to as broadcasting terminal device 2a, and the viewing user's terminal device 2 may be referred to as viewing terminal device 2b. Fig. 14(a) is a diagram showing an example of a broadcasting screen 600 displayed on the display device 23 of the broadcasting terminal device 2a. The broadcasting screen 600 shown in Fig. 14(a) has the same display elements as the broadcasting screen 600 shown in Fig. 7(a).

[0237] As shown in FIG. 14(a), when the broadcaster user operates the input device 24 of the broadcast terminal device 2a to select the acquisition button 604, the input device 24 inputs an instruction to the terminal processing device 27 to acquire the reference information.

[0238] The reference information includes at least a portion of the image information of the delivery image 602. For example, the display processing unit 274 of the streaming terminal device 2a acquires at least a portion of the image information of the delivery image 602 as reference information in response to an instruction to acquire the reference information. The image information acquired as reference information is capture image information (screenshot) of the delivery screen 600 captured by the display processing unit 274 at the time when the instruction to acquire the reference information is issued, or image information corresponding to a partial image area of the screenshot. The image information acquired as reference information may be multiple screenshots (or a portion thereof) acquired in chronological order by the display processing unit 274 within an acquisition period (e.g., a period from the time when the instruction to acquire the reference information is acquired until two seconds later). The acquisition period may be the duration of touching the acquire button 604 in response to the broadcast user's operation of the input device 24 of the streaming terminal device 2a. For example, the input device 24 of the streaming terminal device 2a may output the period from the time when the broadcast user's finger touches the acquire button 604 to the time when the broadcast user's finger releases the acquire button 604 as the acquisition period. Hereinafter, the image information acquired as reference information may be referred to as reference image information. Then, the display processing unit 274 stores the reference image information in the terminal storage device 22, and the reference information acquisition process ends. At least a portion of the image information of the delivered image 602 is an example of information indicating a delivered image.

[0239] The reference information may include motion data of the character object 601 in the delivered image 602. In the example shown in FIG. 14(a), the display processing unit 274 may acquire, as reference information, motion data of the character object 601 in the delivered image 602 in response to an instruction to acquire reference information. For example, each time the generation unit 271 generates motion data, the generation unit 271 temporarily stores, in the terminal storage device 22, motion data from the current time up to a predetermined time ago (e.g., if the current time is 8:05:30, motion data from 8:05:00 to 8:05:30). Then, the display processing unit 274 acquires, as reference information, motion data of the character object 601 in the delivered image 602 displayed at the time when the instruction to acquire reference information was issued. The display processing unit 274 may acquire, as reference information, motion data of the character object 601 in the delivered image 602 displayed within an acquisition period (e.g., a period up to 2 seconds before) from the time when the instruction to acquire reference information was issued. Hereinafter, motion data acquired as reference information may be referred to as reference motion data. Then, the display processing unit 274 stores the reference motion data in the terminal storage device 22, and the reference information acquisition process ends. The motion data of the character object 601 of the delivered image 602 is an example of information indicating the delivered image.

[0240] The reference information acquisition process related to the reference image information and reference motion data is not limited to being executed in response to the broadcasting user selecting the acquisition button 604. For example, the reference information acquisition process related to the reference image information and reference motion data may be executed automatically in response to an instruction from the viewing user. In this case, an acquisition button (not shown) is included on the distribution screen 610 displayed by the viewing terminal device 2b of the viewing user viewing the broadcasting user's distributed image 612. When the acquisition button is selected in response to the viewing user's operation of the input device 24 of the viewing terminal device 2b, the input device 24 inputs an instruction to acquire the reference information together with acquisition date and time information indicating the date and time at which the acquisition button was selected to the terminal processing device 27. Upon receiving the reference information acquisition instruction and the acquisition date and time information, the transmission unit 272 of the viewing terminal device 2b transmits information indicating the reference information acquisition instruction and the acquisition date and time information to the server device 3 via the terminal communication I / F 21. The viewing user ID of the viewing user stored in the terminal storage device 22 may be transmitted together with the information indicating the reference information acquisition instruction and the acquisition date and time information. Next, the server receiving unit 331 of the server device 3 receives information indicating an instruction to acquire the reference information and the acquisition date and time information via the server communication I / F 31. The server transmitting unit 332 of the server device 3 transmits the received information indicating an instruction to acquire the reference information and the acquisition date and time information to the distribution terminal device 2a via the server communication I / F 31. The receiving unit 273 of the distribution terminal device 2a receives the received information indicating an instruction to acquire the reference information and the acquisition date and time information via the terminal communication I / F 21. Based on the information indicating an instruction to acquire the reference information and the acquisition date and time information, the receiving unit 273 executes a reference information acquisition process related to the reference image information and reference motion data at the date and time indicated by the acquisition date and time information. Then, the receiving unit 273 acquires the reference image information and / or reference motion data at the date and time indicated by the acquisition date and time information and stores the reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0241] Furthermore, the reference information acquisition process for the reference image information and reference motion data may be automatically executed when a predetermined acquisition condition is met. The predetermined acquisition condition is a condition related to the response information included in the response information display area 603 on the distribution screen 600. For example, the display processing unit 274 of the distribution terminal device 2a acquires the reference image information and / or reference motion data corresponding to the period with the largest number of pieces of response information.

[0242] An example of the reference information acquisition process related to the reference image information and reference motion data corresponding to the period with the largest number of pieces of response information will be described below. First, the display processing unit 274 sets multiple periods for each predetermined unit time, and stores the number of pieces of response information displayed within each period in the terminal storage device 22. The predetermined unit time is, for example, 10 seconds. The predetermined unit time may also be 5 seconds, 30 seconds, etc. When the predetermined unit time is 10 seconds, for example, the number of pieces of response information for each period from 8:05:00 to 8:05:10, from 8:05:10 to 8:05:20, from 8:05:20 to 8:05:30, etc. may be stored. Each period may have overlapping periods (e.g., from 8:05:00 to 8:05:10, from 8:05:5 to 8:05:15, from 8:05:10 to 8:05:20, etc.). The display processing unit 274 acquires reference image information and / or reference motion data at specific dates and times within each of the periods up to Ma ranking in terms of the number of pieces of response information displayed within the past Na days. Na days may be the 1st, 2nd, 5th, 10th, etc. If Ma is "1", reference image information and / or reference motion data at specific dates and times within one period in which the most pieces of response information were displayed is acquired. If Ma is "5", five pieces of reference image information and / or reference motion data at specific dates and times within each of the periods in which the first to fifth most pieces of response information were displayed are acquired. The specific dates and times within a period are the start date and time of the period, the end date and time of the period, or any date and time within the period. Then, the display processing unit 274 stores the acquired reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0243] The display processing unit 274 may store the number of predetermined types of response information among the response information displayed within each period in the terminal storage device 22. For example, the display processing unit 274 may store the number of comment information, or the display processing unit 274 may store the number of evaluation information. Furthermore, the display processing unit 274 may store the total number of multiple types of response information.

[0244] The predetermined acquisition condition may be a condition related to a viewing user who is viewing the distribution screen 610 or a viewing user who has viewed the distribution screen 610. For example, the display processing unit 274 of the distribution terminal device 2a acquires reference image information and / or reference motion data corresponding to a timing when many viewing users are simultaneously viewing the distribution image of the distribution user.

[0245] An example of a reference information acquisition process related to reference image information and reference motion data corresponding to timings at which a large number of viewing users are simultaneously viewing a broadcast user's broadcast image will be described below. First, the server transmission unit 332 of the server device 3 counts, at predetermined time intervals, the number of user IDs (viewing user IDs) of viewing users currently viewing the broadcast user's broadcast image as the concurrent viewer count. Each time the server transmission unit 332 counts the concurrent viewer count, it transmits concurrent viewer count information indicating the concurrent viewer count to the broadcast terminal device 2a via the server communication I / F 31. The display processing unit 274 references the concurrent viewer count information received at predetermined time intervals, and acquires reference image information and / or reference motion data for the timings with the highest number of concurrent viewers within the past Na days, up to Ma. Na days are the 1st, 2nd, 5th, 10th, etc. If Ma is "1," reference image information and / or reference motion data for the timing with the highest number of concurrent viewers is acquired. If Ma is "5," five pieces of reference image information and / or reference motion data for the timings with the highest number of concurrent viewers, one to five, are acquired. Then, the display processing unit 274 stores the acquired reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0246] The predetermined acquisition condition may be a condition related to the additional objects displayed on the distribution screen 600. For example, the display processing unit 274 of the distribution terminal device 2a acquires reference image information and / or reference motion data corresponding to the period in which the number of additional objects displayed is the largest.

[0247] An example of the reference information acquisition process for the reference image information and reference motion data corresponding to the period with the largest number of displayed additional objects will be described below. First, the display processing unit 274 sets multiple periods, each consisting of a predetermined unit of time (5 seconds, 10 seconds, 30 seconds, etc.), and stores the number of displayed additional objects within each period in the terminal storage device 22. There may be overlapping periods within each period. The display processing unit 274 acquires reference image information and / or reference motion data at specific dates and times within each period up to the Math highest number of displayed additional objects within the past Na days (Na days may be 1 day, 2 days, 5 days, 10 days, etc.). The specific dates and times within a period are the start date and time of the period, the end date and time of the period, or any date and time within the period. Then, the display processing unit 274 stores the acquired reference image information and / or reference motion data in the terminal storage device 22, and the reference information acquisition process ends.

[0248] The display processing unit 274 may store in the terminal storage device 22 the number of additional objects of a predetermined type among the additional objects displayed within each period. For example, the display processing unit 274 may store the number of "bear" gift objects, or the display processing unit 274 may store the number of "accessory" gift objects. The display processing unit 274 may also store the total number of multiple types of additional objects. The display processing unit 274 of the distribution terminal device 2a may also acquire reference image information and / or reference motion data corresponding to the period in which the total amount of consideration for the displayed additional objects was the highest. In this case, the additional objects can be transmitted to the distribution terminal device 2a by the viewing user paying a consideration (such as "coins" having virtual monetary value held by the viewing user). For example, the display processing unit 274 stores in the terminal storage device 22 the total amount of consideration for the displayed additional objects within each period. The display processing unit 274 acquires reference image information and / or reference motion data at specific dates and times within each period up to Ma highest in total amount of consideration displayed within the past Na days (Na days are 1st, 2nd, 5th, 10th, etc.). When reference motion data is acquired, the object ID of the additional object displayed in the distributed image at the time of acquisition may also be acquired. Then, by using the reference motion data and the object ID of the additional object, a character-related image similar to the distributed image at the time of acquisition can be generated. In the video distribution service of this embodiment, the higher the price, the more luxurious the appearance, the more difficult it is to obtain, and / or the more popular the additional object provided. This allows the distributed user to use character-related images that have a high total price for the additional objects, are luxurious in appearance, are difficult to obtain, and / or are very popular.

[0249] The reference information may include response information included in the response information display area 603 in the distribution screen 600. For example, every time comment information is displayed in the response information display area 603, the display processing unit 274 stores the displayed comment information as reference information in the terminal storage device 22. In this case, date and time information such as display date and time information or reception date and time information of the comment information may be stored in the terminal storage device 22 together with the comment information. Furthermore, every time rating information is displayed in the response information display area 603, the display processing unit 274 may store the displayed rating information as reference information in the terminal storage device 22. In this case, date and time information such as display date and time information or reception date and time information of the rating information may be stored in the terminal storage device 22 together with the rating information.

[0250] The comment information stored as reference information is not limited to that included in the response information display area 603. For example, it may be comment information entered by a broadcasting user and / or a viewing user in a specific game that may be played during the video distribution service. The specific game is, for example, a word game (a quiz, a word association game, a word chain game, a word collection game, a riddle, an answer matching game, etc.). The specific game is a network game in which a broadcasting user operating multiple terminal devices 2 connected to the network can participate. The specific game may also be a network game in which a viewing user operating multiple terminal devices 2 connected to the network can participate. The specific game may also be a network game in which a broadcasting user and a viewing user operating multiple terminal devices 2 connected to the network can participate. The specific game is not limited to a word game, and may be a competitive game, a sports simulation game, a shooting game, etc.

[0251] For example, comment information input to each terminal device 2 by multiple users (broadcast users and / or viewing users) participating in a word game is transmitted to the server device 3. The server device 3 transmits the received comment information to the terminal devices 2 of the multiple users participating in the word game. At the end of the word game, the server device 3 may transmit the comment information received during the word game to the terminal devices 2 of the multiple users participating in the word game. Every time comment information is received, the receiving unit 273 of each terminal device 2 stores the received comment information as reference information in the terminal storage device 22. In this case, date and time information, such as information on the date and time when the comment information was received, may be stored in the terminal storage device 22 together with the comment information.

[0252] The comment information stored as reference information may be limited to those that satisfy a predetermined selection condition, such as comment information that has received high ratings from other users, comment information in the top Mb ranking with the most high ratings from other users, etc.

[0253] The reference information may include information about the additional object selected by the viewing user. For example, the object ID of the additional object transmitted together with information indicating an addition instruction in response to an operation on the selection screen 700 of the viewing terminal device 2b by the viewing user viewing the distribution image 612 of the distribution user may be stored as reference information in the terminal storage device 22. For example, each time the receiving unit 273 of the distribution terminal device 2a receives information indicating an addition instruction, the user ID of the viewing user, and an object ID transmitted from the viewing terminal device 2b via the server device 3, the receiving unit 273 stores the received object ID as reference information in the terminal storage device 22. In this case, date and time information, such as display date and time information of the additional object or reception date and time information of the object ID of the additional object, may be stored in the terminal storage device 22 together with the object ID of the additional object. In this way, information about the additional object displayed in the distribution screen 600 is stored as reference information in the terminal storage device 22.

[0254] The object ID of the additional object stored as reference information may be the object ID of an additional object that is not displayed on the distribution screen 600. In this case, a selection screen (not shown) for selecting an additional object, similar to the selection screen 700, is displayed on the viewing terminal device 2b of the viewing user viewing the distribution image 612 of the broadcast user. When the viewing user selects an additional object on the selection screen in response to an operation of the input device 24 of the viewing terminal device 2b by the viewing user, the input device 24 inputs an instruction to acquire reference information and the object ID of the selected additional object to the terminal processing device 27. Upon receiving the instruction to acquire reference information and the object ID of the additional object, the transmitting unit 272 of the viewing terminal device 2b transmits information indicating the instruction to acquire reference information and the object ID of the additional object to the server device 3 via the terminal communication I / F 21. The viewing user ID of the viewing user stored in the terminal storage device 22 may also be transmitted together with the information indicating the instruction to acquire reference information and the object ID of the additional object. Next, the server receiving unit 331 of the server device 3 receives the information indicating the instruction to acquire reference information and the object ID of the additional object via the server communication I / F 31. The server transmitting unit 332 of the server device 3 transmits the received information indicating the instruction to acquire the reference information and the object ID of the additional object to the distribution terminal device 2a via the server communication I / F 31. The receiving unit 273 of the distribution terminal device 2a receives the received information indicating the instruction to acquire the reference information and the object ID of the additional object via the terminal communication I / F 21. Based on the information indicating the instruction to acquire the reference information and the object ID of the additional object, the receiving unit 273 executes a reference information acquisition process to store the received object ID of the additional object as reference information in the terminal storage device 22. As the reference information, the object ID of the received additional object may be stored, and the date and time of reception of the information indicating the instruction to acquire the reference information may also be stored.

[0255] The reference information may include information about owned objects stored in association with the broadcast user. For example, the object IDs of owned objects stored in association with the broadcast user's user ID may be stored in the terminal storage device as reference information.

[0256] The reference information may include a profile image of the broadcast user's character object. For example, a profile image is a type of profile information and includes an image of the broadcast user's character object. The profile image does not have to include an image of the broadcast user's character object; for example, the profile image may be an image that includes an image representing the broadcast user. In addition to the profile image, the profile information may include the broadcast user's name, various information input by the broadcast user (broadcast category, broadcast time slot, favorite additional objects, etc.), etc. The broadcast user's profile information is stored in the terminal storage device 22 of the broadcast user's broadcast terminal device 2a. The server device may also store profile information transmitted from the broadcast terminal device 2a in the server storage device 32. When the broadcast user changes the profile information by operating the input device 24 of the broadcast terminal device 2a, the reference information may also be automatically changed in accordance with the changed profile information. In this case, the user may change the reference information (profile image) by operating the input device 24 of the broadcast terminal device 2a.

[0257] The reference information may include image information showing the virtual space in which the character object exists. For example, the broadcasting user's character object can move freely within the world of the metaverse (Internet space) described below in response to the broadcasting user's operation. In this case, image information showing the world may be acquired as reference information in response to the broadcasting user's operation or automatically.

[0258] The reference information may include image information showing the inside of the game space in which the character object appears. For example, the broadcasting user's character object can move freely within the two-dimensional or three-dimensional game space in which the broadcasting user plays, in response to the broadcasting user's operation. In this case, image information showing the inside of the game space may be acquired as reference information, either in response to the broadcasting user's operation or automatically.

[0259] The reference information is not limited to the above examples, and may include any information as long as it is information about the broadcast user or information about a character object representing the broadcast user.

[0260] (Reference information table T4) Fig. 14(b) shows an example of the data structure of a reference information table T4 that manages reference information. A reference information table T4 is provided for each user, and the reference information table T4 shown in Fig. 14(b) is the reference information table T4 of the user who operates the terminal device 2 that stores the reference information table T4.

[0261] In the reference information table T4, reference information IDs, reference information, date and time information, etc. are stored in association with one another. The reference information ID is an example of identification data for uniquely identifying each piece of reference information. Each time the reference information acquisition process is executed, a new reference information ID is assigned to the acquired reference information, and the new reference information ID and the acquired reference information are stored in association with one another in the reference information table T4. In addition, the date and time information is date and time information acquired together with the reference information in the reference information acquisition process, and is stored in the reference information table T4 together with the acquired reference information in association with the new reference information ID. Note that if date and time information is not acquired together with the reference information in the reference information acquisition process, the new reference information ID and the date and time information associated with the acquired reference information are not stored.

[0262] (Reference information selection screen) 15 is a diagram showing an example of a reference information selection screen 1300 displayed on the display device 23 of the user's terminal device 2. The reference information selection screen 1300 is a screen that is displayed when the additional generation process is started in response to the user selecting the additional generation button 1102 included in the information exchange screen 1100. For example, the display processing unit 274 of the terminal device displays the reference information selection screen 1300 in response to the user selecting the additional generation button 1102 by operating the input device 24. Note that if no reference information is stored in the reference information table T4, the reference information selection screen 1300 is not displayed.

[0263] The reference information selection screen 1300 includes candidate reference information 1301a and 1301b indicating reference information, object data 1302a and 1302b for character-related images, an additional generation instruction button 1303, and an end button 1304.

[0264] Candidate reference information 1301a and 1301b indicating reference information are image information or text information indicating reference information stored in the reference information table T4, and are displayed so that each can be selected by the user. Candidate reference information 1301a shown in FIG. 15 is an example of candidate reference information displayed when the reference information is reference image information or reference motion data. For example, when the reference information is reference image information, the candidate reference information 1301a is reference image information. Furthermore, when the reference information is reference image information, the candidate reference information 1301a may be an image generated based on the reference image information (e.g., an image with a reduced number of pixels of the reference image information). When the reference information is reference motion data, the candidate reference information 1301a is an image indicating a character object generated based on the reference motion data and model data of the character object. Candidate reference information 1301b shown in FIG. 15 is an example of an image displayed when the reference information is an additional object. For example, when the reference information is an additional object, the candidate reference information 1301b is an image indicating the additional object. Furthermore, when the reference information is comment information, the candidate reference information is image information or text information indicating the comment information.

[0265] The number of candidate reference information indicating reference information included in the reference information selection screen 1300 is not limited to two, and may be one, three, or more. When some of the candidate reference information cannot be included in the display area of the reference information selection screen 1300, the display processing unit 274 of the terminal device 2 may display the candidate reference information in a scrollable manner. In this case, in response to a swipe operation by the user, candidate reference information that is not currently included in the display area of the reference information selection screen 1300 may be scrolled and displayed within the display area of the reference information selection screen 1300.

[0266] The candidate reference information included in the reference information selection screen 1300 may be information indicating only a specific type of reference information. For example, the candidate reference information included in the reference information selection screen 1300 may be only image information indicating reference image information. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information indicating reference motion data. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information or text information indicating comment information. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information or text information indicating attachment objects. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information or text information indicating attachment objects transmitted as additional objects. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information or text information indicating gift objects transmitted as additional objects. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information or text information indicating owned objects. Alternatively, the candidate reference information included in the reference information selection screen 1300 may be only image information indicating profile images. Furthermore, the candidate reference information included in the reference information selection screen 1300 may be only image information showing the inside of a virtual space. Note that the reference information selection screen 1300 may include two or more types of candidate reference information.

[0267] The candidate reference information included in the reference information selection screen 1300 may be a portion of a specific type of reference information. For example, the candidate reference information included in the reference information selection screen 1300 may be an image showing comment information extracted by a predetermined extraction method from the comment information stored as reference information in the reference information table T4.

[0268] In this case, the predetermined extraction method may be a method of extracting identical comment information from the comment information stored as reference information, counting the number of such comment information, and extracting the top Mc-ranked comment information based on the counted number. For example, if the Mc-rank is "2" and the reference information table T4 stores five pieces of comment information with the word "cool," three pieces of comment information with the word "cute," and two pieces of comment information with the word "good luck," the two types of comment information, "cool" and "cute," are extracted. Alternatively, the predetermined extraction method may be a method of extracting comment information associated with date and time information for a predetermined period (e.g., 5 seconds before and after the time) related to the time when a specific additional object is displayed from the comment information stored as reference information. For example, if the candidate reference information includes information indicating a specific additional object and comment information associated with date and time information for a predetermined period related to the time when the specific additional object is displayed, the user can select the specific additional object and comment information that matches the specific additional object. This allows the information processing system 1 to provide a character-related image including a specific additional object and comment information desired by the user.

[0269] Furthermore, the candidate reference information included in the reference information selection screen 1300 may be an image showing an additional object (attachment object) extracted by a predetermined extraction method from the additional objects (attachment objects) stored as reference information in the reference information table T4. In this case, the predetermined extraction method is a method of extracting an additional object (attachment object) at a placement location that does not overlap with the display position of object data for the character-related image (data of objects indicating background, characters, etc.) used for the character-related image.

[0270] The reference information selection screen 1300 may not display the candidate reference information 1301a and 1301b. If the candidate reference information 1301a and 1301b are not displayed on the reference information selection screen 1300, reference information selected from the reference information stored in the reference information table T4 according to a predetermined selection method is determined, and a character-related image is generated based on the determined reference information. The predetermined selection method may be a method of randomly selecting a predetermined number of reference information from the reference information, or a method of selecting a predetermined number of reference information from the reference information according to the order previously associated with each reference information. The predetermined number is any natural number equal to or greater than "1" or "2." Note that the reference information may be selected according to a predetermined selection method from a specific type of reference information stored in the reference information table T4. For example, a predetermined number of comment information may be selected according to a predetermined selection method from the comment information stored as reference information. The predetermined selection method is a method of randomly selecting a predetermined number of comment information from the comment information stored as reference information. Alternatively, the predetermined selection method may be a method of extracting identical comment information from the comment information stored as reference information, counting the number of such comment information, and selecting the comment information with the largest number of counts. For example, if five pieces of comment information saying "cool," three pieces of comment information saying "cute," and two pieces of comment information saying "do your best" are stored in the reference information table T4, the comment information saying "cool" is selected (determined).

[0271] The object data 1302a and 1302b for character-related images are image information indicating object data for character-related images stored in the terminal storage device 22, and each is displayed so as to be selectable by the user. The number of types of object data for character-related images included in the reference information selection screen 1300 is not limited to two, and may be one type, or three or more types. When some of the object data for multiple types of character-related images cannot be included in the display area of the reference information selection screen 1300, the display processing unit 274 of the terminal device 2 may display the object data for character-related images in a scrollable manner. In this case, in response to a swipe operation by the user, object data for character-related images that are not currently included in the display area of the reference information selection screen 1300 may be scrolled and displayed within the display area of the reference information selection screen 1300.

[0272] Object data 1302a, 1302b for character-related images may not be displayed on reference information selection screen 1300. When object data 1302a, 1302b for character-related images are not displayed on reference information selection screen 1300, a character-related image is generated that does not use object data for character-related images. Alternatively, when object data 1302a, 1302b for character-related images are not displayed, a character-related image may be generated that uses predetermined object data for character-related images.

[0273] The additional generation instruction button 1303 is an example of an operation object for inputting an additional generation instruction to generate a new character-related image. For example, when the additional generation instruction button 1303 is selected in response to a user's operation of the input device 24, the input device 24 inputs an additional generation instruction to the generation unit 271.

[0274] When the generation unit 271 receives an additional generation instruction from the input device 24, it determines reference information to be used in the generation process of a new character-related image. If either of the candidate reference information 1301a or 1301b has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 determines reference information corresponding to the selected image. If neither of the candidate reference information 1301a nor 1301b has already been selected when the additional generation instruction button 1303 is selected, the generation unit 271 determines reference information selected according to a predetermined selection method from the reference information stored in the reference information table T4.

[0275] Next, the generation unit 271 executes a generation process for generating a new character-related image based on the determined reference information. Hereinafter, an example of the generation process for generating a new character-related image when the determined reference information is reference image information will be described.

[0276] If the determined reference information is reference image information, the generation unit 271 sets the reference information (reference image information) as a new character-related image, associates it with a related image ID for uniquely identifying the new character-related image, and stores it in the terminal storage device 22. Alternatively, the generation unit 271 may cut out an image within a predetermined image area of the reference information (reference image information), set the image as a new character-related image, associates it with a related image ID for uniquely identifying the new character-related image, and stores it in the terminal storage device 22. Then, the generation unit 271 generates candidate information indicating the generated new character-related image, associates the generated candidate information with a related image ID for identifying the new character-related image, and stores it in the terminal storage device 22, thereby completing the generation process. This enables the information processing system 1 to provide a new character-related image including a character object in the same pose (posture) as the character object in the distribution image at the timing of acquiring the reference information.

[0277] An example of a generation process for generating a new character-related image when the determined reference information is reference motion data will be described below. First, the generation unit 271 extracts from the user table T1 the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 the determined reference information (reference motion data) and object data for the character-related image. Next, the generation unit 271 generates a new character-related image including a character object with a predetermined action and facial expression, and object data such as a background and text, based on the read model data, the image information and placement location of the used object, the reference information (reference motion data), and the object data for the character-related image. If either object data 1302a or 1302b for a character-related image has already been selected when additional generation instruction button 1303 is selected, generation unit 271 generates a new character-related image using the selected object data. This enables information processing system 1 to provide a new character-related image including a character object in the same pose (posture) as the character object in the delivery image at the time of acquiring the reference information.

[0278] An example of a generation process for generating a new character-related image when the reference information is response information will be described. First, the generation unit 271 extracts from the user table T1 the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 motion data for the character-related image and object data for the character-related image. Next, the generation unit 271 generates a new character-related image including a character object with a predetermined motion and facial expression, object data such as a background and text, and object data corresponding to the reference information, based on the read model data, the image information and placement location of the used object, the motion data for the character-related image and the object data for the character-related image, and the reference information. If either object data 1302a or 1302b for a character-related image has already been selected when additional generation instruction button 1303 is selected, generation unit 271 generates a new character-related image using the selected object data, thereby enabling information processing system 1 to provide a character-related image using information related to the character object on the distribution screen.

[0279] When the reference information is comment information, the object data corresponding to the reference information is an object on which the comment information is written (for example, a dial object on which the text "Cool" is displayed). In this case, the generated new character-related image includes the object on which the comment information is written.

[0280] An example of a generation process for generating a new character-related image when the reference information is information about an additional object will be described. First, the generation unit 271 extracts, from the user table T1, the model ID of the character object associated with the user ID stored in the terminal storage device 22 and the object ID of the used object. Next, the generation unit 271 reads, from the terminal storage device 22, model data associated with the extracted model ID and image information and placement location associated with the extracted object ID of the used object. Next, the generation unit 271 reads, from the terminal storage device 22, motion data for the character-related image and object data for the character-related image. Next, based on the read model data, image information and placement location of the used object, motion data for the character-related image and object data for the character-related image, and the reference information, the generation unit 271 generates a new character-related image including a character object with a predetermined motion and facial expression, object data such as a background and text, and an additional object corresponding to the reference information. If either object data 1302a or 1302b for a character-related image has already been selected when additional generation instruction button 1303 is selected, generation unit 271 generates a new character-related image using the selected object data.

[0281] If the reference information is information about an attachment object among the additional objects, the additional object corresponding to the reference information is an attachment object that is applied to the character object. For example, if the additional object corresponding to the reference information is an attachment object "headband," the generated new character-related image includes a character object wearing the headband. In this case, if the extracted use object (the attachment object already applied to the character object) is of the same type as the attachment object corresponding to the reference information, the attachment object corresponding to the reference information may be applied to the character object instead of the extracted use object. This enables the information processing system 1 to provide a character-related image using information about the character object on the distribution screen.

[0282] If the reference information is information about a gift object among the additional objects, the additional object corresponding to the reference information is a gift object held by the character object. For example, if the additional object corresponding to the reference information is a gift object "bear," the generated new character-related image includes a character object holding the "bear." In this case, motion data for the character-related image may be applied to the character object, resulting in a pose according to the attributes of the character object and / or the type of gift object. This enables the information processing system 1 to provide a character-related image using information from the viewing user regarding the character object on the distribution screen.

[0283] In this case, multiple types of motion data for character-related images may be stored in the terminal storage device 22, and any of the motion data may be automatically applied. For example, different types of motion data for character-related images may be stored for each parameter of the character object. The parameters may include gender (female, male, etc.), size (height above a predetermined value, below a predetermined value, etc.), personality (calm, lively, etc.), age (30 years old or above, under 30 years old, etc.). For example, if the character object is a female type, motion data MF for character-related images may be automatically applied, and if the character object is a male type, motion data MM for character-related images may be automatically applied. If the character object's height is above a predetermined value, motion data MT for character-related images may be automatically applied, and if the character object's height is below a predetermined value, motion data ML for character-related images may be automatically applied. Furthermore, if the additional object corresponding to the reference information is the gift object "bear," motion data MG for character-related images may be automatically applied, and if the additional object corresponding to the reference information is the gift object "light stick," motion data MP for character-related images may be automatically applied. Furthermore, if the character object is female and the additional object corresponding to the reference information is the gift object "bear," motion data MF1 for the character-related image is automatically applied; if the character object is male and the additional object corresponding to the reference information is the gift object "bear," motion data MF2 for the character-related image is automatically applied; if the character object is female and the additional object corresponding to the reference information is the gift object "light stick," motion data MP1 for the character-related image is automatically applied; and if the character object is male and the additional object corresponding to the reference information is the gift object "light stick," motion data MP2 for the character-related image is automatically applied.

[0284] Note that when the reference information is information about a gift object among the additional objects, the motion data for the character-related image is not necessarily automatically applied to the character object. For example, motion data selected by the user from a plurality of types of motion data for character-related images may be applied to the character object.

[0285] An example of a generation process for generating a new character-related image when the reference information is information about a owned object will be described. First, the generation unit 271 extracts from the user table T1 the model ID of a character object associated with a user ID stored in the terminal storage device 22 and the object ID of a used object. Next, the generation unit 271 reads from the terminal storage device 22 model data associated with the extracted model ID and image information and placement location associated with the extracted object ID of the used object. Next, the generation unit 271 reads from the terminal storage device 22 motion data for the character-related image and object data for the character-related image. Next, based on the read model data, image information and placement location of the used object, motion data for the character-related image and object data for the character-related image, and reference information, the generation unit 271 generates a new character-related image including a character object with a predetermined motion and facial expression, object data such as a background and text, and a owned object corresponding to the reference information. If either object data 1302a or 1302b for a character-related image has already been selected when additional generation instruction button 1303 is selected, generation unit 271 generates a new character-related image using the selected object data, thereby enabling information processing system 1 to provide a character-related image using information about the user related to the character object.

[0286] In this case, the generated new character-related image includes the character object holding the owned object. In this case, motion data for the character-related image may be applied to the character object, such that the pose corresponds to the attributes of the character object and / or the type of the gift object, just as in the case where the reference information is information about a gift object among the additional objects.

[0287] If the determined reference information is a profile image, the generation unit 271 sets the profile image as a new character-related image. If the determined reference information is image information indicating the inside of a virtual space, the generation unit 271 sets the image information indicating the inside of the virtual space as a new character-related image. If the determined reference information is image information indicating the inside of a game space, the generation unit 271 sets the image information indicating the inside of the game space as a new character-related image. This enables the information processing system 1 to provide a character-related image using various information related to a character object.

[0288] Then, the generation unit 271 associates the generated new character-related image with a related image ID for uniquely identifying the new character-related image and stores the associated image ID in the terminal storage device 22. The generation unit 271 also generates candidate information indicating the generated new character-related image, associates the generated candidate information with a related image ID for identifying the new character-related image and stores the associated image ID in the terminal storage device 22, and the process of generating the new character-related image ends.

[0289] If the additional object (attached object) stored as reference information in the reference information table T4 is a specific attached object, the generation unit 271 may automatically apply the specific attached object in the generation process of all character-related images. The automatic application process of the specific attached object may be executed only within an application period from the time the specific attached object is stored as reference information. The automatic application process of the specific attached object may be executed only when the number of times a character-related image generated with the specific attached object applied is used is equal to or less than a predetermined upper limit. For example, if a broadcast user uses a character-related image generated with the specific attached object applied 10 times (an example of the upper limit), the automatic application process of the specific attached object will not be executed thereafter, and the generation unit 271 will execute a generation process in which the specific attached object is not automatically applied to all character-related images.

[0290] When a specific attachment object is stored as reference information, a generation process for applying the specific attachment object to an already generated character-related image may be automatically re-executed. Furthermore, a character-related image to which a specific attachment object is applied may be usable in the information exchange service for only a predetermined usage period. For example, after the predetermined usage period has elapsed, a generation process for not applying the specific attachment object to an already generated character-related image may be automatically re-executed. Alternatively, after the predetermined usage period has elapsed, candidate information for a character-related image to which a specific attachment object is applied may be deleted from candidate information 1403 (described later) to make the character-related image to which a specific attachment object is applied unusable.

[0291] If the additional object (attachment object) stored as reference information is a specific attachment object, the generation unit 271 may not automatically apply the specific attachment object in the generation process of some character-related images. For example, the specific attachment object may not be applied to a character-related image where, as a result of applying the specific attachment object, the display position of the specific attachment object overlaps with the display position of object data for the character-related image (data of objects indicating background, text, etc.) used in the character-related image. Furthermore, the specific attachment object may not be applied to a character-related image where, as a result of applying the specific attachment object, the display position of the specific attachment object overlaps with the display position of another attachment object that is already attached.

[0292] The end button 1304 is an operation object for ending the display of the reference information selection screen 1300. When the end button 1304 is selected by the user operating the input device 24, the display processing unit 274 of the terminal device 2 ends the display of the reference information selection screen 1300.

[0293] 16 is a diagram showing an example of an information exchange screen 1400 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 1400 is displayed when a new character-related image is generated by the additional generation process.

[0294] Similar to the information exchange screen 1100, the information exchange screen 1400 includes other user input information 1401, other user information 1402, a text input object 1003, a candidate display button 1004, user input information 1404, and an additional generation button 1102. The candidate information 1403 on the information exchange screen 1400 includes the four types of candidate information 1101 on the information exchange screen 1100 and candidate information corresponding to two types of new character-related images. Note that if candidate information corresponding to two types of new character-related images is included in the candidate information 1403, one or more types of candidate information out of the four types of candidate information 1101 that were previously included may be deleted from the candidate information 1403. For example, deleting three or more types of candidate information reduces the number of options for candidate information, making it easier for the user to select information.

[0295] When one piece of candidate information is selected from six types of candidate information 1403 in response to a user's operation of input device 24, input device 24 sends the related image ID associated with the selected candidate information to terminal processing device 27. Next, display processing unit 274 extracts, from terminal storage device 22, a character-related image identified by the related image ID received from input device 24, and displays the extracted character-related image in the user display area of information exchange screen 1400 as user input information 1404 (FIG. 16).

[0296] Along with the display of the user input information 1404 by the display processing unit 274, the transmission unit 272 transmits the related image ID received from the input device 24 and the user ID stored in the terminal storage device 22 to the server device 3 via the terminal communication I / F 21. The server transmission unit 332 of the server device 3 extracts, from the server storage device 32, the user IDs of other users associated with the same information exchange group as the user ID received by the server receiving unit 331. Furthermore, the server transmission unit 332 transmits, via the server communication I / F 31, the related image ID and user ID received by the server receiving unit 331 to the terminal device 2 of the other user identified by the extracted user ID.

[0297] Then, the receiving unit 273 of the terminal device 2 of the other user receives the related image ID and user ID transmitted from the server device 3 via the terminal communication I / F 21. Next, the display processing unit 274 of the terminal device 2 of the other user extracts a character-related image identified by the related image ID received by the receiving unit 273 from the terminal storage device 22, and displays the extracted character-related image in the other user display area of the distribution screen as other user input information. In this way, the same image as the character-related image used by the user is also displayed on the terminal device 2 of the other user associated with the information exchange group to which the user belongs. Note that if the character-related image identified by the received related image ID is not stored in the terminal storage device 22 by the display processing unit 274 of the terminal device 2 of the other user, the transmitting unit 272 of the terminal device 2 of the other user transmits an image transmission request to the server device 3. The image transmission request is transmitted together with the received related image ID and / or user ID, and the user ID of the other user. When the server receiving unit 331 receives an image transmission request, the server transmitting unit 332 of the server device 3 extracts, from the server storage device 32, a character-related image associated with the related image ID received together with the image transmission request. Next, the server transmitting unit 332 transmits the extracted character-related image together with the received related image ID and user ID to the terminal device 2 of the other user. The receiving unit 273 of the terminal device 2 of the other user receives the related image ID, character-related image, and user ID transmitted from the server device 3 via the terminal communication I / F 21, associates them with each other, and stores them in the terminal storage device 22. Then, the display processing unit 274 of the terminal device 2 of the other user displays the stored character-related image in the other user display area of the distribution screen as other user input information.

[0298] (Operation sequence of information processing system 1) 17 is a diagram showing an example of an operation sequence of the information processing system 1. This operation sequence is executed based on control programs previously stored in the terminal storage device 22 and the server storage device 32, respectively, mainly by the terminal processing device 27 and the server processing device 33 in cooperation with each element of the two terminal devices 2 and the server device 3. Hereinafter, of the two terminal devices 2, the distribution terminal device 2a will be referred to as the terminal device 2 operated by the distribution user, and the viewing terminal device 2b will be referred to as the terminal device 2 operated by the viewing user.

[0299] First, the transmitting unit 272 of the broadcasting terminal device 2a transmits the character video data generated by the generating unit 271 and the broadcast user ID of the broadcast user to the server device 3 via the terminal communication I / F 21 (step S101). The transmitting unit 272 may also transmit the broadcast user's voice data along with the character video data to the server device 3. From the start to the end of the video broadcasting service, the processing of step S101 is executed continuously at predetermined time intervals (for example, every two seconds). Then, in response to the execution of the processing of step S101, the processing of steps S102 to S104 is executed intermittently.

[0300] Every time the server receiving unit 331 receives character moving image data, the server transmitting unit 332 of the server device 3 distributes the received character moving image data to the viewing terminal device 2b via the server communication I / F 31 (step S102). The display processing unit 274 of the distribution terminal device 2a generates drawing data for displaying a distribution image based on the character moving image data generated by the generating unit 271, and displays a distribution screen on the display device 23 based on the generated drawing data (step S103). The display processing unit 274 of the viewing terminal device 2b generates drawing data for displaying a distribution image based on the character moving image data received by the receiving unit 273, and displays a distribution screen on the display device 23 based on the generated drawing data (step S104).

[0301] Next, the transmitting unit 272 of the viewing terminal device 2b transmits information regarding a display instruction for displaying the response information on the distribution screen and / or information indicating an addition instruction for displaying the additional object on the distribution screen, which information has been input in response to the viewing user's operation of the input device 24, to the server device 3 via the terminal communication I / F 21 (step S105). Every time the server receiving unit 331 receives information regarding various instructions (display instruction, addition instruction, etc.), the server transmitting unit 332 of the server device 3 distributes the received information regarding the various instructions to the distribution terminal device 2a via the server communication I / F 31 (step S106). The display processing unit 274 of the viewing terminal device 2b generates drawing data for displaying the response information and / or the additional object based on the various instructions input in response to the viewing user's operation of the input device 24, and displays the distribution screen on the display device 23 based on the generated drawing data (step S107). The display processing unit 274 of the distribution terminal device 2a generates drawing data for displaying the response information and / or additional objects based on the information regarding the various instructions received by the receiving unit 273, and displays the distribution screen on the display device 23 based on the generated drawing data (step S108).

[0302] Next, the display processing unit 274 of the distribution terminal device 2a executes a reference information acquisition process (step S109). The reference information acquisition process will be described in detail later. When acquiring reference information transmitted from the viewing terminal device 2b, the reference information acquisition process may be executed by the receiving unit 273 of the distribution terminal device 2a. The reference information acquisition process is not limited to a process that can be executed only by the distribution terminal device 2a. For example, the display processing unit 274 or the receiving unit 273 of the viewing terminal device 2b, for which a character object representing a user is available, may execute the candidate information display process.

[0303] Next, the display processing unit 274 of the distribution terminal device 2a executes candidate information display processing (step S110). Details of the candidate information display processing will be described later. Note that the candidate information display processing is not limited to processing that can be executed only by the distribution terminal device 2a. For example, the display processing unit 274 of the viewing terminal device 2b, which has a character object representing a user available, may execute the candidate information display processing.

[0304] Next, the generation unit 271 and the display processing unit 274 of the distribution terminal device 2a execute the additional generation process (step S111). The additional generation process will be described in detail later. The additional generation process is not limited to a process that can be executed only by the distribution terminal device 2a. For example, the generation unit 271 and the display processing unit 274 of the viewing terminal device 2b, which has available a character object representing a user, may execute the candidate information display process.

[0305] Next, the transmitting unit 272 of the distribution terminal device 2a transmits the input information input in response to the operation of the input device 24 by the distribution user to the server device 3 via the terminal communication I / F 21 (step S112). Next, every time the server receiving unit 331 receives input information, the server transmitting unit 332 of the server device 3 distributes the received input information to the viewing terminal device 2b via the server communication I / F 31 (step S113). The display processing unit 274 of the distribution terminal device 2a generates drawing data for displaying the input information input in response to the operation of the input device 24 by the distribution user, and displays an information exchange screen on the display device 23 based on the generated drawing data (step S114). The display processing unit 274 of the viewing terminal device 2b generates drawing data for displaying the input information received by the receiving unit 273, and displays an information exchange screen on the display device 23 based on the generated drawing data (step S115).

[0306] 18(a) is a diagram showing an example of the operational flow of the reference information acquisition process. The candidate information display process shown in FIG. 18(a) is executed in step S109 of FIG.

[0307] First, the display processing unit 274 or the receiving unit 273 determines whether an instruction to acquire reference information has been acquired (step S201). If an instruction to acquire reference information has not been acquired (step S201-No), the display processing unit 274 or the receiving unit 273 ends the reference information acquisition process. If an instruction to acquire reference information has been acquired (step S201-Yes), the display processing unit 274 or the receiving unit 273 acquires the reference information (step S202), stores the acquired reference information in the terminal storage device 22 (step S203), and ends the reference information acquisition process.

[0308] 18(b) is a diagram showing an example of the operational flow of the candidate information display process The candidate information display process shown in FIG.

[0309] First, the display processing unit 274 determines whether or not a display instruction for candidate information has been acquired (step S301). If a display instruction for candidate information has not been acquired (step S301-No), the process returns to step S301 (waits until a display instruction for candidate information is acquired).

[0310] When an instruction to display candidate information is acquired (step S301-Yes), the display processing unit 274 determines whether or not a character-related image has been generated in the past and stored in the terminal storage device 22 (step S302). When a character-related image has not been stored (step S302-No), the generation unit 271 generates a character-related image (step S303) and proceeds to step S304.

[0311] If a character-related image has been generated in the past and stored in the terminal storage device 22 (step S302-Yes), the generation unit 271 generates candidate information corresponding to the character-related image, and the display processing unit 274 displays the generated candidate information (step S304), and ends the candidate information display process.

[0312] 19 is a diagram showing an example of the operation flow of the additional generation process. The additional generation process shown in FIG. 19 is executed in step S111 of FIG.

[0313] First, the display processing unit 274 determines whether or not reference information is stored in the reference information table T4 (step S401). If reference information is not stored (step S401-No), the display processing unit 274 ends the additional generation process. If reference information is stored (step S401-Yes), the display processing unit 274 displays the reference information selection screen 1300 (step S402).

[0314] Next, the display processing unit 274 determines whether an additional generation instruction has been input by the input device 24 (step S403). If an additional generation instruction has not been input (step S403-No), the display processing unit 274 ends the additional generation process. If an additional generation instruction has been input (step S403-Yes), the generation unit 271 determines reference information (step S404).

[0315] Next, the generation unit 271 generates a new character-related image based on the determined reference information (step S405). Then, the display processing unit 274 displays the generated new character-related image (step S406), and the additional generation process ends.

[0316] As described above in detail, the information processing system 1 of the present embodiment can increase the types of character-related images that can be used based on information about the broadcasting user or information about a character object representing the broadcasting user. In this way, the information processing system 1 of the present embodiment can improve the user interface that determines the information to be provided to other users without requiring the user to perform complicated operations. Furthermore, the information processing system 1 of the present embodiment can change the types of character-related images based on information about the broadcasting user or information about a character object representing the broadcasting user. In this way, the information processing system 1 of the present embodiment can change the information that the user provides to other users, preventing the user from getting bored of making selections, without requiring the user to perform complicated operations.

[0317] (Variation 1) It should be noted that the present invention is not limited to this embodiment. For example, stamp images may be categorized by type and displayed on the information exchange screen. For example, stamp images include stamp images for conveying "OK," stamp images for conveying "NO," stamp images with a "laughing" expression, etc. Furthermore, stamp images may include stamp images generated as character-related images, stamp images generated as new character-related images, etc.

[0318] FIG. 20 is a diagram showing an example of information exchange screens 1500 and 1510 including stamp images generated as character-related images. The information exchange screens 1500 and 1510 include type selection buttons 1501 for selecting the type of stamp image. When candidate information corresponding to one of the type selection buttons 1501 is selected in response to a user's operation of the input device 24, a stamp image of the type corresponding to the selected type selection button 1501 is displayed in a display area below the type selection button 1501. Although FIG. 20 shows an example of a button as a widget for selecting the type of stamp image, the widget for selecting the type of stamp image may also be a tab. In this way, the information exchange screens 1500 and 1510 switch between and display a plurality of stamp images by type.

[0319] 20(a) is a diagram showing an example of an information exchange screen 1500 when a type selection button 1501 corresponding to a stamp image generated as a character-related image is selected. On the information exchange screen 1500, candidate information 1101 corresponding to the stamp image generated as a character-related image is displayed. Note that when the type selection button 1501 corresponding to the stamp image for conveying "OK" is selected, candidate information corresponding to the stamp image for conveying "OK" is displayed on the information exchange screen 1500.

[0320] 20(b) is a diagram showing an example of an information exchange screen 1510 when a type selection button 1501 corresponding to a stamp image generated as a new character-related image is selected. On the information exchange screen 1510, the type selection button 1501 corresponding to the stamp image generated as a new character-related image is selected, and in response to this selection, candidate information 1511 corresponding to the stamp image generated as a new character-related image is displayed. In the example shown in FIG. 20(b), the type selection button 1501 corresponding to the stamp image generated as a new character-related image is a button object labeled "Add."

[0321] In this way, the information exchange screen is configured to display stamp images categorized by type, thereby improving the user interface. Such an information exchange screen makes it possible to improve the visibility of multiple stamp images even if the size of the display screen of the terminal device 2 is limited. Furthermore, such information exchange screens 1500 and 1510 allow the user to intuitively and / or easily find the stamp image they wish to use. Furthermore, such information exchange screens 1500 and 1510 allow the user to know at a glance whether or not there are any stamp images generated as new character-related images.

[0322] (Variation 2) At least some of the functions implemented by the terminal processing device 27 of the terminal device 2 may be implemented by a processor other than the terminal processing device 27 of the terminal device 2. At least some of the functions implemented by the terminal processing device 27 may be implemented, for example, by the server processing device 33 of the server device 3. Specifically, some of the functions of the generation unit 271 and the display processing unit 274 may be executed in the server device 3. For example, the terminal device 2 transmits continuously acquired imaging data and audio data to the server device 3 along with the user ID of the user operating the terminal device 2. Then, the server device 3 generates character video data using the functions of the generation unit 271 and the display processing unit 274, and generates display information for displaying a communication screen including distribution images of all users on the display device 23. Then, the server device 3 transmits the display information together with audio information (audio data) to each of the terminal devices 2 of all users. Each terminal device 2 outputs the information received from the server device 3 (so-called "server rendering").

[0323] (Variation 3) The terminal device 2 has the function of displaying communication screens and the like by executing various commands contained in a control program stored in the terminal device 2. To provide an information exchange service and / or a video distribution service, the terminal device 2 may also invoke a browser function incorporated in a web application, use this browser function to receive a web page (a document written in a markup language such as HTML (HyperText Markup Language)) from a server device, and execute the control program incorporated in the web page (so-called "browser rendering"). For example, by using HTML 5 as the markup language, the terminal device 2 can easily execute new information processing. By employing such a web application to provide an information exchange service and / or a video distribution service in the terminal device 2, a program creator can provide new information processing to the client side (terminal device side) simply by incorporating a new program into a web page transmitted by the server device, thereby significantly reducing the man-hours required to develop a new program. Furthermore, the client side can receive the new service simply by receiving the web page without downloading a new control program. This reduces the load on the communication network, communication costs, and / or communication time compared to downloading a control program, and simplifies the user interface.

[0324] (Variation 4) The generation unit 271 of user A's terminal device 2 may generate output information including facial motion data without using imaging data. For example, facial motion data corresponding to voice data may be stored in the terminal storage device 22 of the terminal device 2, and the generation unit 271 may extract facial motion data corresponding to the acquired voice data of user A from the terminal storage device 22 and generate output information including the extracted facial motion data, the voice data, and user A's user ID. Alternatively, the generation unit 271 of user A's terminal device 2 may generate output information including user A's voice data output from the microphone 26 and the user ID stored in the terminal storage device 22, without including facial motion data. In this case, the display processing unit 274 of user B's terminal device 2 may extract facial motion data corresponding to user A's voice data included in the output information of user A received via the server device 3, and generate animation of user A's character object. The process of generating facial motion data corresponding to user A's voice data may be a process using a known lip synchronization algorithm.

[0325] For example, when the terminal device 2 of user A uses an HMD as the display device 23, communication with user B can be performed via a character object without acquiring image data of user A.

[0326] (Variation 5) The broadcast images 602, 612, 812 displayed on each terminal device 2 are not limited to moving images or still images including animations of 3D or 2D character objects generated based on the broadcast user's actions. The broadcast images 602, 612, 812 may include animations of character objects generated in response to the broadcast user's operations, and may also include moving images or still images of the broadcast user himself. Note that during the video broadcast service, each terminal device 2 does not have to display these character object animations or moving images or still images of the broadcast user himself. In this case, each terminal device 2 may output only the broadcast user's voice from a speaker.

[0327] In this way, the "broadcast user" disclosed in this embodiment is not limited to a user who at least uses the function of the terminal device to transmit character video data to the server device, but may also be a user who transmits information and / or audio data for displaying various broadcast images. Information for displaying various broadcast images is, for example, character video data, video data of the broadcast user himself, etc.

[0328] (Variation 6) Furthermore, the "broadcasting user" disclosed in this embodiment is not limited to a user who hosts or hosts a single video distribution service. The "broadcasting user" may also be a user who hosts or hosts a collaborative distribution service in which multiple people can participate, or a video chat service or voice chat service in which multiple people can participate and / or watch. Alternatively, the "broadcasting user" may be a user who hosts or hosts an event (such as a party) service in a virtual space in which multiple people can participate and / or watch. In this way, the "broadcasting user" may be a user who primarily performs these services, and may be referred to as a host user, organizer user, or organizer user.

[0329] The "viewing user" disclosed in this embodiment is not limited to a user who at least uses the functions of a terminal device to receive character video data from a distribution terminal device and display images distributed by the distribution user. The "viewing user" may also be a user who receives information and / or audio data for displaying various distributed images.

[0330] A "viewing user" may be not only a user who receives information and / or audio data for displaying various distribution images, but also a user who is capable of reacting. For example, a "viewing user" may be a user who watches a video distribution service or a collaboration distribution service, or a user who participates in a video chat service, a voice chat service, or an event service and / or a user who watches these services. Therefore, the "viewing user" in the present disclosure may be rephrased as a guest user, a participating user, a listener, a viewing user, a supporting user, or the like.

[0331] (Variation 7) Furthermore, the information processing system 1 according to the embodiment of the present disclosure may provide a next-generation internet space (metaverse) in which multiple people can simultaneously participate in a digital world through character objects (e.g., avatars) and freely engage in social activities such as social interaction, work, and / or play at a level similar to that of the real world, transcending the gap between reality and virtuality. In such a space, users' avatars can freely roam within the world and communicate with each other.

[0332] The information processing system 1 may distribute a distribution image including one avatar (character object) among multiple avatars in the space as the character object of the distribution user. In other words, one-to-many video distribution may be performed in a many-to-many metaverse space. In such a space, there may be no particular distinction between the distribution user and the viewing user.

[0333] Although the present embodiment and its modifications have been described in detail above, the present invention is not limited to the specific examples, and various changes, substitutions, and modifications can be made to the present invention without departing from the scope of the present invention. [Additional Notes] [1] An information processing system comprising one or more computer processors, the one or more computer processors: transmitting information regarding display of a moving image including a first object representing the first user to a second terminal device of a second user; transmitting to the second terminal device information regarding display of the related image selected by the first user from a group including a plurality of related images related to the first object; Run the one or more computer processors: Obtaining reference information for the first object; an image including the first object, generated based on the reference information, included in the group as the related image; An information processing system characterized by executing the above. [2] the one or more computer processors: including an image that includes the first object and that is generated without using the reference information in the group as the related image; The information processing system according to claim 1, [3] In obtaining the reference information, the one or more computer processors: acquiring information about the displayed video as the reference information; 3. The information processing system according to claim 1, wherein the information processing system executes the above. [4] The information processing system described in claim 3, wherein the information regarding the video acquired as the reference information includes motion data for displaying the related image including the first object in a pose identical to the pose of the first object included in the displayed video at a predetermined timing. [5] An information processing system as described in claim 3, wherein the information regarding the video obtained as the reference information includes captured image information of the displayed video. [6] the one or more computer processors: including text information entered by the second user in the video; Run 4. The information processing system according to claim 3, wherein the information about the video acquired as the reference information includes the text information. [7] The information processing system described in claim 3, wherein the information regarding the video obtained as the reference information includes information regarding the second object selected by the first user from among second objects that can be selected to be displayed together with the first object in the video. [8] The one or more computer processors: including in the video an image showing a third object selected by the second user; Run In obtaining the reference information, the one or more computer processors: acquiring information of an image showing the third object as the reference information; The information processing system according to claim 1, [9] The one or more computer processors: storing information of an image showing a held object held by the first user; Run In obtaining the reference information, the one or more computer processors: acquiring image information representing the owned object as the reference information; The information processing system according to claim 1,

[10] The one or more computer processors: storing an image representing a profile of the first object; Run In obtaining the reference information, the one or more computer processors: acquiring information of an image indicating the profile as the reference information; The information processing system according to claim 1,

[11] The one or more computer processors: Displaying the first object existing in a virtual space; Run In obtaining the reference information, the one or more computer processors: acquiring predetermined image information showing the inside of the virtual space as the reference information; The information processing system according to claim 1,

[12] The information processing system of claim 1, wherein the rendering processing load for rendering the video on the display screen is greater than the rendering display load for rendering the related image on the display screen.

[13] The information processing system of claim 1, wherein the video is a three-dimensional video image depicting a three-dimensional space including the first object, and the related image is a two-dimensional image.

[14] The information processing system of claim 1, wherein the display size of the video on the display screen is larger than the display size of the related image on the display screen.

[15] The information processing system of claim 1, wherein the transmission volume of the display information for displaying the video is greater than the transmission volume of the information regarding the display of the related image.

[16] A method for controlling an information processing system having one or more computer processors, the one or more computer processors: transmitting information regarding display of a moving image including a first object representing the first user to a second terminal device of a second user; transmitting to the second terminal device information regarding display of the related image selected by the first user from a group including a plurality of related images related to the first object; Including, the one or more computer processors: Obtaining reference information for the first object; an image including the first object, generated based on the reference information, included in the group as the related image; A control method comprising:

[17] A server device that communicates with a first terminal device of a first user and a second terminal device of a second user, receiving information regarding a display of a video including a first object representing the first user, the information being transmitted from the first terminal device; receiving information regarding display of the related image selected by the first user from a group including a plurality of related images related to the first object, the information being transmitted from the first terminal device; a processing unit that transmits, to the second terminal device, information regarding the display of the video and information regarding the display of the related image received from the first terminal device. Equipped with an image including the first object, which is generated based on reference information about the first object, is included in the group as the related image; A server device characterized by: [Explanation of symbols]

[0334] 1. Information Processing Systems 2. Terminal Device 21 Terminal communication I / F 22 Terminal storage device 23 Display device 24 Input Devices 25 Imaging device 26. Mike 27 Terminal Processing Device 271 Generation part 272 Transmitter 273 Receiving Unit 274 Display processing section 275 Audio Output Unit 3. Server equipment 31 Server communication I / F 32 Server storage device 33 Server processing unit 331 Server Receiving Unit 332 Server transmission unit 4 base station 5. Mobile communication networks 6 Gateway 7. Internet

Claims

1. 1. An information processing system comprising one or more computer processors, the one or more computer processors: storing an image showing a profile of a first object representing the first user; transmitting information relating to the display of the video including the first object to a second terminal device of a second user; including in the video an image showing text information input by the second user or a third object selected by the second user; transmitting information regarding the display of the related image selected by the first user from a group including a plurality of related images related to the first object to the second terminal device and / or a third terminal device of a third user; acquiring information about the video, which is information about an image indicating the profile, as reference information; an image including the first object, generated based on the reference information, included in the group as the related image; Run An information processing system comprising:

2. the one or more computer processors: The information processing system according to claim 1 , wherein reference information relating to the first object is acquired when response information in the video satisfies a predetermined condition.

3. 3. The information processing system according to claim 2, wherein the predetermined condition is a condition regarding the number of pieces of response information.

4. When an image showing the third object is included in the moving image, an image showing the third object associated with a predetermined part of the first object is displayed in the moving image; the one or more computer processors: If a predetermined condition is satisfied, an image including the first object, which is generated based on the reference information including an image showing the third object, is included in the group as the related image; if the predetermined condition is not satisfied, an image including the first object, which is generated based on the reference information that does not include an image showing the third object, is included in the group as the related image; The information processing system according to claim 1 , wherein the information processing system executes the following:

5. The information processing system of claim 1, wherein the information regarding the video acquired as the reference information includes motion data for displaying the related image including the first object in the same pose as the pose of the first object included in the displayed video at a predetermined timing.

6. The information processing system according to claim 1 , wherein the information about the moving image acquired as the reference information includes captured image information of the displayed moving image.

7. the one or more computer processors: Displaying the first object existing in a virtual space; Run The information processing system according to claim 1 , wherein the reference information includes predetermined image information showing the inside of the virtual space.

8. The information processing system according to claim 1 , wherein a rendering processing load for rendering the video on the display screen is greater than a rendering display load for rendering the related image on the display screen.

9. The information processing system according to claim 1 , wherein the video is a three-dimensional video image in which a three-dimensional space including the first object is depicted, and the related image is a two-dimensional image.

10. The information processing system according to claim 1 , wherein a display size of the moving image on the display screen is larger than a display size of the related image on the display screen.

11. The information processing system according to claim 1 , wherein a transmission amount of the display information for displaying the moving image is greater than a transmission amount of the information for displaying the related image.

12. A method for controlling an information processing system having one or more computer processors, comprising: the one or more computer processors: storing an image showing a profile of a first object representing the first user; transmitting information relating to the display of the video including the first object to a second terminal device of a second user; including in the video an image showing text information input by the second user or a third object selected by the second user; transmitting information regarding the display of the related image selected by the first user from a group including a plurality of related images related to the first object to the second terminal device and / or a third terminal device of a third user; acquiring information about the video, which is information about an image indicating the profile, as reference information; an image including the first object, generated based on the reference information, included in the group as the related image; Contains A control method characterized by:

13. A server device that communicates with a first terminal device of a first user and a second terminal device of a second user, receiving information regarding a display of a video including a first object representing the first user, the information being transmitted from the first terminal device; storing an image showing a profile of the first object; receiving information regarding display of the related image selected by the first user from a group including a plurality of related images related to the first object, the information being transmitted from the first terminal device; a processing unit that transmits information regarding the display of the video and information regarding the display of the related image received from the first terminal device to the second terminal device and / or a third terminal device of a third user. Equipped with the video includes text information input by the second user or an image showing a third object selected by the second user; The image including the first object, which is generated based on reference information acquired as information about the video and represents the profile, is included in the group as the related image. A server device characterized by:

Images