Terminal device control program, terminal device, terminal device control method, server device control program, server device, and server device control method

Abstract

To improve a motivation of a user to continue a communication with the other user.SOLUTION: A control program for a terminal device causes a terminal device of a first user to: display at least a first image including a first object indicating the first user and a second image including a second object indicating a second user; transmit information relating to the display of the first image to a terminal device of the second user; in a case where a first voice of the first user is acquired, transmit the first voice to the terminal device of the second user; display the first image including the first object which is changed in accordance with an action or the first voice of the first user; in a case where a second voice of the second user is received, output the second voice; display the second image including the second object which is changed in accordance with an action or the second voice of the second user; display the first object changed in accordance with an instruction from the first user; and display the second image changed in accordance with the instruction from the first user.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The embodiments disclosed in this specification relate to a control program for a terminal device, a terminal device, a control method for a terminal device, a control program for a server device, a server device, and a control method for a server device. [Background technology]

[0002] Conventionally, there has been known an information system in which a plurality of users communicate with each other by using a terminal device such as a personal computer (PC) to transmit various information such as an image of each user and / or the voice of each user to the terminal device of another user. For example, Patent Document 1 discloses a technology of an information system in which each user's PC, connected to the Internet, shares the image and voice of each user acquired from a camera and microphone provided on the PC, thereby realizing a voice chat between users. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Nissho Electronics Corporation, "What is Zoom in 3 minutes? A web conferencing tool you'll be missing out on if you don't know about it," [online], July 12, 2019, [Retrieved January 26, 2022], Internet<URL:https: / / zoom.nissho-ele.co.jp / blog / practice / zoom-meeting.html> Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional information systems, there is a lack of variety in the output from the user's terminal device when the user is communicating with other users, which creates the problem that the user becomes bored with communicating with other users.

[0005] The disclosed terminal device control program, terminal device, terminal device control method, server device control program, server device, and server device control method make it possible to increase a user's motivation to continue communicating with other users. [Means for solving the problem]

[0006] The control program of the terminal device is a control program for the terminal device of a first user, and causes the terminal device of the first user to display at least a first image including a first object representing the first user and a second image including a second object representing a second user different from the first user, to transmit information regarding the display of the first image to the terminal device of the second user, and when a first voice of the first user is acquired, to transmit the first voice to the terminal device of the second user and display a first image including a first object that changes in accordance with the action or first voice of the first user, and when a second voice of the second user is received, to output the second voice and display a second image including a second object that changes in accordance with the action or second voice of the second user, to display the first object that has changed in accordance with an instruction from the first user, and to display the second image that has changed in accordance with an instruction from the first user.

[0007] Furthermore, it is preferable that the control program of the terminal device, when displaying the changed first object, causes the terminal device of the first user to display the first object performing a first action corresponding to an instruction from the first user, and when displaying the changed second image, causes the terminal device of the first user to display a second object performing a second action related to the first action in accordance with the first action of the first object, or information regarding the first action of the first object.

[0008] Furthermore, in the control program of the terminal device, it is preferable that the instruction by the first user is an instruction to select a first action to be made by the first object from a plurality of first actions, and an instruction to select a second object to be made to perform a second action or a second image including the second object to be made to perform the second action.

[0009] Furthermore, in the control program of the terminal device, when a condition regarding the association between the first user and the second user is satisfied, it is preferable that a specific first action is included in the plurality of first actions.

[0010] Furthermore, in the control program of the terminal device, it is preferable that the second object to be caused to perform the second action is selected by the first user specifying the second object in the displayed second image, and that multiple selection objects corresponding to each of the multiple first actions are displayed in response to the selection instruction for the second object to be caused to perform the second action, and that the first action to be caused by the first object is the first action corresponding to the selection object selected and instructed by the first user from among the multiple selection objects displayed.

[0011] Furthermore, in the control program of the terminal device, it is preferable that the second image including the second object that is to perform the second action is selected by the first user performing a designation operation on the displayed second image, and that multiple selection objects corresponding to each of the multiple first actions are displayed in response to a selection instruction for the second image including the second object that is to perform the second action, and that the first action to be performed by the first object is the first action corresponding to the selection object selected and instructed by the first user from among the multiple selection objects displayed.

[0012] Furthermore, it is preferable that the control program of the terminal device automatically displays each selected object corresponding to each of a plurality of first actions in the second image within a first period after a second image including a second object is displayed on the terminal device of the first user, and that the first action to be performed by the first object is a first action corresponding to a selected object selected and instructed by the first user from among the selected objects, and that the second object to perform the second action is a second object in the second image in which one selected object selected and instructed by the first user is displayed.

[0013] Furthermore, in the control program of the terminal device, it is preferable that the second action performed by the second object is a second action selected by the second user from among a plurality of second actions on the second user's terminal device within a second period after the first user gives an instruction to the first object to perform the first action or after the first object performing the first action is displayed.

[0014] Furthermore, in the control program of the terminal device, it is preferable that the second action performed by the second object is a second action related to the first action that is automatically identified when the first user selects and instructs the first action to be performed by the first object.

[0015] Furthermore, it is preferable that the control program of the terminal device executes the following: within a third period after a first object performing a first action is displayed on the terminal device of a first user, information indicating a third action selected by the second user from among a plurality of third actions is received on the terminal device of a second user; after displaying a second object performing a second action, the second object performing the third action indicated by the received information is displayed.

[0016] Furthermore, it is preferable that the control program of the terminal device, in displaying a first object performing a first action, displays on the terminal device of the first user a selection screen for selecting one specific action from among a plurality of types of specific actions in accordance with instructions from the first user or the second user, selects one specific action from among the plurality of types of specific actions as the first action in accordance with instructions from the first user, and displays the first object performing the selected first action; and, in displaying a second object performing a second action, receives on the terminal device of the first user specific action information indicating the specific action selected as the second action by the second user from among the plurality of types of specific actions from the terminal device of the second user on which a selection screen for selecting one specific action from among the plurality of types of specific actions is displayed, and displays the second object performing the second action indicated by the specific action information.

[0017] Furthermore, it is preferable that the control program for the terminal device executes the following: displaying a selection screen on the terminal device of a first user in response to an instruction from the first user or the second user, for selecting one specific action from among a plurality of types of specific actions, and designating the specific action selected by the first user from among the plurality of types of specific actions as a first action; receiving specific action information indicating the specific action selected by the second user from among the plurality of types of specific actions from the terminal device of a second user on which the selection screen for selecting one specific action from among the plurality of types of specific actions is displayed in response to an instruction from the first user or the second user, and designating the specific action indicated by the specific action information as a second action; and, in displaying the first object performing the first action and the second object performing the second action, if the first action and the second action are designated within a third period from the display of the selection screen, displaying the first object performing the designated first action and the second object performing the designated second action on the terminal device of the first user.

[0018] Furthermore, in the control program of the terminal device, when displaying the second object performing the second action, it is preferable to cause the terminal device of the first user to execute the following: when a condition regarding the number of times the second object has performed the second action is satisfied, displaying the second object performing a fourth action different from the second action, either without performing the second action or together with the second action, in accordance with the first action of the first object.

[0019] Furthermore, it is preferable that the control program of the terminal device is configured to, when displaying a second object performing a second action, display a second object that does not perform a second action corresponding to the first action of the first object when a predetermined setting instruction is given by the second user to the terminal device of the first user.

[0020] The control program of the server device is a control program of the server device capable of communicating with each of a terminal device of a first user and a terminal device of a second user different from the first user, the control program causing the server device to receive information relating to display of a first image including a first object representing the first user from the terminal device of the first user, receive information relating to display of a second image including a second object representing the second user from the terminal device of the second user, transmit information for displaying at least the first image to at least the terminal device of the second user, transmit information for displaying at least the second image to at least the terminal device of the first user, and, when a first voice of the first user is received, transmit the first voice to at least the terminal device of the second user. to the terminal device of the second user, transmit information for displaying a first image including a first object that changes in response to the action or first voice of the first user to at least the terminal device of the second user, and when the second voice of the second user is received, transmit the second voice to the terminal device of the first user and transmit information for displaying a second image including a second object that changes in response to the action or second voice of the second user to at least the terminal device of the first user, receive information for displaying the first object that has changed in response to an instruction by the first user, and transmit information for displaying the changed first object to at least the terminal device of the second user.

[0021] The terminal device is a terminal device of a first user, and includes a processing unit that displays at least a first image including a first object representing the first user and a second image including a second object representing a second user different from the first user, transmits information related to the display of the first image to the terminal device of the second user, and when a first voice of the first user is acquired, transmits the first voice to the terminal device of the second user and displays the first image including the first object that changes in accordance with the action of the first user, and when a second voice of the second user is received, outputs the second voice and displays the second image including the second object that changes in accordance with the action of the second user, displays the first object that has changed in accordance with an instruction from the first user, and displays the second image that has changed in accordance with an instruction from the first user.

[0022] The server device is a server device capable of communicating with each of a terminal device of a first user and a terminal device of a second user different from the first user, and receives information relating to the display of a first image including a first object representing the first user from the terminal device of the first user, receives information relating to the display of a second image including a second object representing the second user from the terminal device of the second user, transmits information for displaying at least the first image to at least the terminal device of the second user, transmits information for displaying at least the second image to at least the terminal device of the first user, and, when receiving a first voice of the first user, transmits the first voice to the terminal device of the second user. the terminal device of the first user, and transmits information for displaying a first image including a first object that changes in response to the action or first voice of the first user to at least the terminal device of the second user; when receiving the second voice of the second user, transmits the second voice to the terminal device of the first user and transmits information for displaying a second image including a second object that changes in response to the action or second voice of the second user to at least the terminal device of the first user; receives information for displaying the first object that has changed in response to an instruction by the first user; and transmits information for displaying the changed first object to at least the terminal device of the second user.

[0023] A method for controlling a terminal device is a method for controlling a terminal device of a first user, and includes the steps of: the terminal device of the first user displaying at least a first image including a first object representing the first user and a second image including a second object representing a second user different from the first user; transmitting information related to the display of the first image to the terminal device of the second user; when acquiring a first voice of the first user, transmitting the first voice to the terminal device of the second user and displaying a first image including a first object that changes in accordance with the action or first voice of the first user; when receiving a second voice of the second user, outputting the second voice and displaying a second image including a second object that changes in accordance with the action or second voice of the second user; displaying the first object that has changed in accordance with an instruction from the first user; and displaying the second image that has changed in accordance with an instruction from the first user.

[0024] A control method for a server device is a control method for a server device capable of communicating with each of a terminal device of a first user and a terminal device of a second user different from the first user, the control method including the steps of: receiving, from the terminal device of the first user, information relating to display of a first image including a first object representing the first user; receiving, from the terminal device of the second user, information relating to display of a second image including a second object representing the second user; transmitting information for displaying at least the first image to at least the terminal device of the second user; transmitting information for displaying at least the second image to at least the terminal device of the first user; and, upon receiving a first voice of the first user, transmitting the first voice to the terminal device of the second user. transmitting information to at least the terminal device of the second user for displaying a first image including a first object that changes in response to the action or first voice of the first user, and, when receiving the second voice of the second user, transmitting the second voice to the terminal device of the first user and transmitting information to at least the terminal device of the first user for displaying a second image including a second object that changes in response to the action or second voice of the second user, receiving information for displaying the first object that has changed in response to an instruction by the first user, and transmitting information for displaying the changed first object to at least the terminal device of the second user. [Effects of the Invention]

[0025] The control program for the terminal device, the terminal device, the control method for the terminal device, the control program for the server device, the server device, and the control method for the server device make it possible to increase a user's motivation to continue communicating with other users. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an overview of an information system. [Figure 2] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information system. [Figure 3] FIG. 2 is a diagram illustrating an example of a schematic configuration of a terminal device. [Figure 4]3A to 3C are diagrams illustrating examples of data structures of various tables. [Figure 5] FIG. 2 illustrates an example of a schematic configuration of a server device. [Figure 6] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 7] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 8] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 9] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 10] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 11] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 12] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 13] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 14] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 15] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 16] FIG. 10 is a diagram illustrating an example of an operation sequence of the information system. [Figure 17] FIG. 10 is a diagram illustrating an example of a screen displayed on a display device of the terminal device. [Figure 18] FIG. 2 is a schematic diagram for explaining the inside of a virtual space. DETAILED DESCRIPTION OF THE INVENTION

[0027] Various embodiments of the present invention will be described below with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions set forth in the claims and their equivalents.

[0028] (Information System Overview) FIG. 1 is a schematic diagram illustrating an example of an information system. The information system includes a terminal device operated by each of a plurality of users and a server device. For example, the terminal device is an information processing device such as a multi-function mobile phone (a so-called "smartphone") owned by the user. For example, the server device is a computer for providing communication services between the terminal devices via a communication network.

[0029] The terminal device stores a control program such as an application program, loads the control program into memory in response to a start operation from a user, and starts a communication service by executing instructions contained in the loaded control program. When the communication service starts, the terminal device realizes multiple functions by executing instructions contained in the control program.

[0030] For example, a user's terminal device realizes a generation function that generates output information including character animation data that includes motion data based on various input data entered by the user. For example, the input data is a plurality of image data acquired at predetermined sampling time intervals by an imaging device provided in the user's terminal device, and in this case, each image data shows the user. The character animation data is an example of information related to the display of a user output image that includes a character object representing the user. During a communication service, output information is generated at predetermined time intervals.

[0031] Furthermore, for example, the user's terminal device displays a user output image based on character video data and transmits the generated output information to the terminal devices of other users via the server device, thereby realizing an output function of transmitting the output information to the server device at predetermined time intervals. Furthermore, for example, the user's terminal device realizes a function of displaying other user output images including character objects representing other users based on the output information of other users transmitted from the server device at predetermined time intervals.

[0032] When a communication service is started, for example, a user output image (1) and another user output image (2) are displayed on the user's terminal device as shown in Fig. 1(a). If another user is not participating in the communication service when the communication service is started, only the user output image (1) is displayed on the user's terminal device. If a user is not participating in the communication service when the communication service is started, only the other user output image (2) is displayed on the other user's terminal device.

[0033] The user output image (1) includes a character object of the user that moves in accordance with the user's motion data included in the character video data. The other user output image (2) includes a character object of another user that moves in accordance with the motion data of the other user included in the received output information of the other user. In this way, the user's terminal device displays the user output image (1) including the user's character object that changes in accordance with the user's motion, and the other user output image (2) including the character object of the other user that changes in accordance with the motion of the other user.

[0034] Since both the user's terminal device and the other user's terminal device are equipped with microphones, the user's terminal device acquires the voice uttered by the user, and the other user's terminal device acquires the voice uttered by the other user.The user's terminal device then includes the acquired user's voice data in output information and transmits the output information including the user's voice data to the other user's terminal device via the server device.The user's terminal device also receives from the server device the output information including the other user's voice data that was transmitted to the server device by the other user's terminal device.

[0035] For example, when a user produces a sound, the user's motion data included in the character video data includes face motion data indicating the user's lip movements. In this case, the user's terminal device displays a user output image (1) including the user's character object whose lips move in approximate synchronization with the sound produced by the user. The output information of other users received by the user's terminal device may include the other user's voice data along with the other user's motion data. In this case, the user's terminal device outputs the other user's voice and displays a other user output image (2) including the other user's character object whose lips move in approximate synchronization with the other user's voice.

[0036] In addition to the above-mentioned functions, the user's terminal device may also realize a function to change a character object included in the user output image (1) in response to a user instruction and display the changed character object. The user's terminal device may also realize a function to change another user output image (2) in response to a user instruction and display the changed other user output image (2).

[0037] For example, the user's terminal device changes the user's character object in response to a change instruction from the user, thereby changing the user output image (1). As shown in FIG. 1(b), for example, the user's terminal device causes the user's character object to move in a manner such that its arms extend toward the other user's character object in response to the user's change instruction. In this way, the user's terminal device automatically changes the user's character object. In this case, the user's terminal device automatically changes the other user output image (2) of the other user by including the hand and arm of the user's character object extending from the direction of the user's character object in the other user output image (2).

[0038] Note that the user's terminal device may cause the outstretched hand of the user's character object in the other user output image (2) to perform a smacking motion on the other user's character object. Furthermore, the user's terminal device may cause the other user's character object to perform a reaction motion in response to this motion. Furthermore, during the time when the user's character object automatically changes in response to a change instruction from the user (for example, the three seconds from when the arm is extended to when it is returned), the user's terminal device may not cause the user's character object to perform a motion in accordance with the user's motion data. Similarly, during the time when the other user's output image (2) of the other user automatically changes in response to a change instruction from the user (for example, the three seconds from when the arm appears and extends to the other user's character object until the display of the arm ceases), the other user's terminal device may not cause the other user's character object to perform a motion in accordance with the other user's motion data.

[0039] A change instruction by a user is input, for example, by a selection operation in which the user selects a predetermined operation object (such as a button object). Furthermore, a change instruction by a user may be input in response to an operation performed by the user from among a plurality of types of operations. Each of the plurality of types of operations is associated with one of a plurality of actions that the user's character object can perform. The plurality of actions may include, as shown in FIG. 1(b), an action in which the user's character object hits another user's character object, an action in which the user's character object tickles another user's character object, or an action in which the user's character object strokes another user's character object.

[0040] As shown in FIG. 1(c), the terminal device of the other user, like the user's terminal device, displays at least the other user output image (2) and the user output image (1). The other user output image (2) includes the other user's character object that moves in accordance with the other user's motion data included in the other user's character video data. The user output image (1) also includes the user's character object that moves in accordance with the user's motion data included in the user's output information transmitted by the user's terminal device. In this way, the other user's terminal device displays the other user output image (2) including the other user's character object that changes in accordance with the other user's motion, and displays the user output image (1) including the user's character object that changes in accordance with the user's motion.

[0041] When the change instruction sent by the user to the server device is received by the terminal device of the other user via the server device, the user output image (2) of the other user changes, and the user's character image changes, as in Fig. 1(b). In the example shown in Fig. 1(c), the terminal device of the other user displays the user output image (1) and the other user output image (2) in an order different from the order in which the user output image (1) and the other user output image (2) are displayed by the user's terminal device. Alternatively, the terminal device of the other user may display the user output image (1) and the other user output image (2) in the same order as the user output image (1) and the other user output image (2) displayed by the user's terminal device.

[0042] Note that changes to the user's character object and the other user's output image (2) due to a change instruction from the user are not limited to changes due to the movement of the character object. For example, the change to the user's character object may be a change in the color and / or brightness of the character object, or a change in the size of the character object. Furthermore, the change to the other user's output image may be a change in the color and / or brightness of the other user's output image, a change in the size of the other user's character object, or a change due to the addition of a new character string, a new icon image, a new mark image, etc.

[0043] As explained above with reference to Figure 1, the control program for the user's terminal device, the user's terminal device, the control method for the user's terminal device, the control program for the server device, the server device, and the control method for the server device display a changed user character and a changed image output by another user in response to an instruction from the user. In this way, the control program for the user's terminal device, the user's terminal device, the control method for the user's terminal device, the control program for the server device, the server device, and the control method for the server device allow the user to change the output from the user's terminal device during the communication service, thereby increasing the user's motivation to continue using the communication service.

[0044] Furthermore, in conventional information systems, when a user communicates with other users through a character object, such as an avatar representing the user, the user's motion data is used to represent the character object's facial expressions, body gestures, etc. Conventional terminal devices are required to create animation of the user's character object using the user's motion data that is generated sequentially, which can increase the load on the user's terminal device's drawing process and slow down the drawing speed. Furthermore, in order to display the animation of the user's character object on the terminal devices of other users, the user's terminal device transmits the user's motion data to the other users' terminal devices via a server device. In such cases, the user's terminal device is required to transmit the user's motion data that is generated sequentially, which can increase the communication load between the terminal device and the server device, resulting in audio degradation and / or problems with the character object's drawing. In contrast, the control program of the user's terminal device, the user's terminal device, the control method of the user's terminal device, the control program of the server device, the server device, and the control method of the server device make it possible to perform drawing processing and communication processing using pre-prepared motion data of the character object's movements, without having to acquire the user's motions one by one and generate animation of the character object, thereby making it possible to reduce the processing load of the terminal device 2 and / or the communication load between the terminal device and the server device.

[0045] In addition, in conventional information systems, in order to change the facial expressions, body gestures, etc., of a user's character object, the user himself / herself creates facial expressions and moves his / her body to generate motion data. However, if a terminal device that acquires a user's gesture cannot change the gesture into the motion data intended by the user, the user must repeat the gesture. As such, in conventional information systems, it is sometimes difficult for a user to instantly and accurately cause a character object to perform the action he / she wants. In contrast, a control program for a user's terminal device, a user's terminal device, a control method for a user's terminal device, a control program for a server device, a server device, and a control method for a server device enable instantaneous and dynamic communication to be accurately realized by the user simply specifying the action, without the user having to perform the gesture to cause the character object to perform it, thereby improving usability.

[0046] Furthermore, in conventional information systems, an output image including a user's character object is only displayed within a display area set for each user, and information about each user cannot be displayed beyond that user's display area. Therefore, even if a user's character object expresses something to another user's character object, the expression is limited to that user's display area, making it difficult to convey the intent of that expression. In response to this, a control program for a user's terminal device, a user's terminal device, a control method for a user's terminal device, a control program for a server device, a server device, and a control method for a server device enable interactions between character objects beyond the display area of ​​each character object, enabling richer communication and expression even on terminal devices with limited display area.

[0047] In the example shown in Figure 1, the user and other users are users who belong to a predetermined communication group among multiple users who can use the communication service. The predetermined communication group is a group created by either the user or other users. In the example shown in Figure 1, character objects of two users are displayed, but character objects of three or more users who belong to the predetermined communication group may also be displayed, allowing three or more users to communicate.

[0048] In the example shown in FIG. 1, the user and other users may be users who have a predetermined relationship. Other users who have a predetermined relationship with a user are, for example, mutual followers of the user. When a user follows another user and the other user also follows the user (the user and the other user mutually follow each other), the user and the other user become mutual followers. Other users who have a predetermined relationship with a user may also be other users who have a mutual follow relationship with a specific user who mutually follows the user. Other users who have a predetermined relationship with a user may also be other users who are friends with the user, or other users whose specific information (such as a phone number, email address, or a specific ID (identification)) is stored in the user's terminal device. In addition, if the user's terminal device has a function to distribute user output images including the user's character object to one or more terminal devices of other users via a server device, other users who have a predetermined relationship with the user are other users who have viewed the user output images distributed by the user. In this case, when a user and another user who have a predetermined relationship communicate as shown in Figure 1, a predetermined communication group consisting of the user and the other user may be temporarily created by the user's or the other user's terminal device or server device.

[0049] The above description of Fig. 1 is merely provided to deepen understanding of the present invention. The present invention is specifically embodied in the following embodiments, and may be embodied in various modifications without substantially departing from the principles of the present invention. All such modifications are included within the scope of the present invention and the disclosure of this specification.

[0050] (Information Systems 1) FIG. 2 is a diagram illustrating an example of a schematic configuration of an information system 1. The information system 1 includes a terminal device 2 operated by each of multiple users, and a server device 3. The terminal device 2 and the server device 3 are connected to each other via a communication network, such as a base station 4, a mobile communication network 5, a gateway 6, and the Internet 7. Communication between the terminal device 2 and the server device 3 is performed based on a communication protocol such as Hypertext Transfer Protocol (HTTP). Alternatively, a connection may be established between the terminal device 2 and the server device 3 initially using HTTP communication, and thereafter communication may be performed based on WebSocket, which provides bidirectional communication with lower costs (lower communication loads and processing loads) than HTTP communication. Note that the communication method between the terminal device 2 and the server device 3 is not limited to the above-described method, and any communication method technology may be used between the terminal device 2 and the server device 3 as long as it can realize this embodiment.

[0051] The terminal device 2 is an information processing device such as a smartphone. The terminal device 2 may be a mobile phone, a notebook PC, a tablet terminal, a tablet PC, a head mounted display (HMD), an e-book reader, a wearable computer, or the like. The terminal device 2 may be a portable game device, a game console, or the like. The terminal device 2 may be any information processing device as long as it is capable of displaying and outputting character objects of the user and other users and outputting voices of the user and other users.

[0052] 2, one server device 3 is illustrated as a component of the information system 1, but the server device 3 may be a collection of multiple physically separate server devices 3. In this case, each of the multiple server devices 3 may have the same function, or may have the function of a single server device 3 in a distributed manner.

[0053] (Terminal device 2) 3 is a diagram illustrating an example of a schematic configuration of the terminal device 2. The terminal device 2 connects to the server device 3 via a communication network such as a base station 4, a mobile communication network 5, a gateway 6, and the Internet 7, and communicates with the server device 3. The terminal device 2 generates character video data including motion data according to various data (image data, etc.) input by a user, and transmits output information including the generated character video data and / or the user's voice data to the server device 3. The terminal device 2 also receives output information of other users transmitted from the server device 3, and displays character objects of other users and / or outputs voices of other users based on the received output information. To realize these functions, the terminal device 2 includes a terminal communication I / F 21, a terminal storage device 22, a display device 23, an input device 24, an image capture device 25, a microphone 26, and a terminal processing device 27.

[0054] The terminal communication I / F 21 is implemented as hardware, firmware, or communication software such as a TCP / IP (Transmission Control Protocol / Internet Protocol) driver or a PPP (Point-to-Point Protocol) driver, or a combination of these. The terminal device 2 can transmit data to and receive data from other devices such as the server device 3 via the terminal communication I / F 21.

[0055] The terminal storage device 22 is, for example, a semiconductor memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The terminal storage device 22 stores an operating system program, driver programs, control programs, data, etc. used for processing in the terminal processing device 27. The driver programs stored in the terminal storage device 22 include an output device driver program that controls the display device 23 and an input device driver program that controls the input device 24. The control programs stored in the terminal storage device 22 are, for example, application programs for realizing various functions related to the communication service. Note that the control programs may be those transmitted from the server device 3 or other devices.

[0056] The terminal storage device 22 stores, as data, identification information (for example, a user ID (identification)) for uniquely identifying a user operating the terminal device 2. The terminal storage device 22 also stores, as data, background data, model data, etc. The terminal storage device 22 also stores, as data, a user table T1, an object table T2, and a group table T3. The terminal storage device 22 may also temporarily store data related to a predetermined process.

[0057] Background data is asset data for constructing a virtual space in which a user's character object exists in a user output image. The background data includes data for drawing the background of the virtual space, data for drawing various objects included in the user output image, and data for drawing various other background objects displayed in the user output image. The background data may also include object position information indicating the positions of various background objects in the virtual space.

[0058] The display device 23 is a liquid crystal display. The display device 23 may be an organic EL (Electro-Luminescence) display or the like. The display device 23 displays, on a display screen, moving images corresponding to moving image data and / or still images corresponding to still image data supplied from the terminal processing device 27. The display device 23 does not have to be a component of the terminal device 2. In this case, the display device 23 may be a display of an HMD capable of communicating with the server device 3, a projection device for projection mapping, a retinal projection type projection device, or the like capable of communicating with the terminal device 2 via a wired or wireless connection.

[0059] The input device 24 is a pointing device such as a touch panel. When the input device 24 is a touch panel, the input device 24 can detect various touch operations by the user, such as tapping, double tapping, and dragging. The touch panel may be equipped with a capacitance-type proximity sensor and configured to be able to detect non-contact operations by the user. The input device 24 may also be an input key or the like. The user can use the input device 24 to input letters, numbers, symbols, or positions on the display screen of the display device 23. When operated by the user, the input device 24 generates a signal corresponding to the operation. The input device 24 then supplies the generated signal to the terminal processing device 27 as a user instruction.

[0060] The imaging device 25 is a camera having an imaging optical system, an imaging element, an image processing unit, etc. The imaging optical system is, for example, an optical lens, and focuses a light beam from a subject on the imaging surface of the imaging element. The imaging element is, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and outputs an image of the subject imaged on the imaging surface. The image processing unit creates moving image data in a predetermined file format from images continuously generated by the imaging element at predetermined intervals and outputs the moving image data as imaging data. Alternatively, the image processing unit creates still image data in a predetermined file format from images generated by the imaging element and outputs the moving image data as imaging data.

[0061] The microphone 26 is a sound collection device configured to acquire voice uttered by the user and convert it into voice data. The microphone 26 is configured to acquire voice input from the user, convert the user's voice input acquired by the microphone 26 into voice data, and output the voice data to the terminal processing device 27.

[0062] The terminal processing device 27 is a processor that loads the operating system program, driver program, and control program stored in the terminal storage device 22 into memory and executes instructions contained in the loaded programs. The terminal processing device 27 is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or a graphics processing unit (GPU), or a combination of various electronic circuits. The terminal processing device 27 may be realized by an integrated circuit such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or a micro controller unit (MCU). Although the terminal processing device 27 is illustrated in FIG. 3 as a single component, the terminal processing device 27 may also be a collection of multiple physically separate processors.

[0063] The terminal processing device 27 executes various commands included in the control program to function as a generation unit 271, a transmission unit 272, a reception unit 273, a display processing unit 274, and an audio output unit 275. The functions of the generation unit 271, the transmission unit 272, the reception unit 273, the display processing unit 274, and the audio output unit 275 will be described later.

[0064] (various tables) 4 is a diagram showing an example of the data structure of the user table T1, object table T2, and group table T3 stored in the terminal storage device 22. At least one of the user table T1, object table T2, and group table T3 may be stored in the server storage device 32 of the server device 3. In this case, the terminal device 2 may receive the latest table stored in the server storage device 32 at a predetermined timing and store it in the terminal storage device 22.

[0065] (User table T1) FIG. 4(a) is an example of the data structure of a user table T1 that manages users participating in a communication service. In the user table T1, for each user, a user ID, the user's name, the user's character object, the user's owned objects, and the user's used objects are stored in association with one another. In the user table T1, the user's owned coins (information indicating the total amount of virtual monetary value "coins" held by the user) and / or user points may be stored in association with the user's user ID. The user ID is an example of identification data for uniquely identifying each user. The name is an example of data indicating the name of each user.

[0066] For example, a character object is a model ID for identifying model data for generating animation of the character object. The model data is associated with the model ID and stored in the terminal storage device 22. The model data may be three-dimensional model data for generating three-dimensional animation, or two-dimensional model data for generating two-dimensional model data. The model data includes, for example, rig data (so-called "skeleton data") indicating the face and skeleton of the parts other than the face of the character object, and surface data indicating the shape and texture of the surface of the character object. The model data may include multiple different model data. Each of the multiple model data may have different rig data or the same rig data. Each of the multiple model data may have different surface data or the same surface data.

[0067] The owned object is an object ID indicating an object owned by each user. The owned object is an attachment object that can be associated with a specific part of a character object, a wallpaper object that is placed behind the user's character object in the user output image, etc. Details of attachment objects and wallpaper objects will be described later. The owned object is an object selected by a lottery game executed by the user's terminal device 2 or the server device 3 through the user's operation. The owned object may also be an object that the user has acquired by spending "coins" of virtual monetary value in the purchase function of the information system 1.

[0068] The used object is an object ID indicating an object used in the user output image currently being displayed on the user's terminal device 2. For example, if a user inputs a change instruction during a communication service, an attachment object held as one of the user's owned objects is attached to the user's character object included in the user output image currently being displayed. In this case, the object ID of the attachment object attached to the user's character object is stored in the user table T1 as a used object in association with the user ID. Also, for example, if a user inputs a change instruction during a communication service, a wallpaper object held as one of the user's owned objects is placed behind the user's character object in the user output image currently being displayed. In this case, the object ID of the wallpaper object placed in the user output image is stored in the user table T1 as a used object in association with the user ID.

[0069] (Object table T2) 4(b) shows an example of the data structure of an object table T2 that manages objects selected as additional objects. An additional object is, for example, a gift object that a user gives to another user. In the object table T2, the object ID, name, image information, rarity, placement location, etc. are stored for each additional object in association with one another.

[0070] The object ID is an example of identification data for uniquely identifying each additional object. The image information is one or more still images corresponding to each additional object. The image information may be one or more types of moving images corresponding to each additional object. The rarity is information indicating the rarity of each additional object. The placement location is information indicating the location where the image information of each additional object is displayed. Note that if the display position and display range of a character object are determined, information indicating the relative position of the character object may be stored as the placement location.

[0071] The additional objects are classified into a plurality of types (categories). For example, the additional objects include effect objects used as effect gifts, normal objects used as normal gifts, attachment objects used as attachment gifts, and message objects used as message gifts. The additional objects may also include wallpaper objects that are placed behind the character objects in the user output image. Information indicating the type (category) of the additional object may be stored in the object table T2 in association with the object ID of the additional object. Below, image information and placement locations will be described according to the type of additional object.

[0072] The effect object is an object that affects the overall impression of the user output image and / or other user output images. The effect object is, for example, an object that resembles confetti. When the effect object is an object that resembles confetti, an image showing multiple pieces of paper is stored as the image information.

[0073] Information indicating "space" as the placement location of the effect object is stored. For example, an object resembling confetti is displayed over the entire user output image and / or other user output images. Note that the effect object may be displayed over the entire screen (communication screen, described later) including the user output image and other user output images in response to a user instruction. The effect object may also be displayed only within an output image selected by the user from among the user output image and other user output images. The effect object may also be displayed so as to overlap with a character object included in the user output image and / or other user output image. Unlike wearable objects, described later, the effect object is displayed without being associated with a specific part of the character. In this way, by displaying an effect object in response to a user instruction, the terminal device 2 of the information system 1 can change the overall impression of the user output image and / or other user output image compared to before the effect object was displayed.

[0074] A normal object is, for example, an object that resembles a stuffed animal, a bouquet of flowers, an accessory, or an item suitable for a gift or present. Information indicating a "space" is stored as the placement location of the normal object. For example, information regarding a predetermined movement route is associated with the normal object, and the normal object is displayed moving along the predetermined movement route in the user output image or the other user output image. The information indicating the "space" stored as the placement location of the "normal object" may be classified into multiple types. The multiple types of "space" include, for example, a "center space" (the central area of ​​the image when the user output image or the other user output image is divided into three parts horizontally), a "left space" (the area on the left side of the image when the user output image or the other user output image is divided into three parts horizontally), and a "right space" (the area on the right side of the image when the user output image or the other user output image is divided into three parts horizontally). The multiple types of "spaces" may be a "center space" (a central region of the image when the user output image or the other user output image is vertically divided into thirds), an "upper space" (an upper region of the image when the user output image or the other user output image is vertically divided into thirds), and a "lower space" (a lower region of the image when the user output image or the other user output image is vertically divided into thirds). In this case, the movement route associated with the normal object is set within the "space" of the type associated with the position of the normal object. For example, if information indicating a "right space" is stored in a normal object, information regarding a movement route moving within the right region of the image when the user output image or the other user output image is horizontally divided into thirds is associated with the normal object. Furthermore, the normal object may be displayed moving according to a predetermined movement rule from a position set for each normal object in the user output image or the other user output image or a position automatically and randomly determined. The predetermined movement rule is, for example, a movement rule set so that the normal object moves as if it were free falling. The normal object may be displayed so as to overlap with a character object. In this case, the normal object is displayed without being associated with a specific part of the character object, unlike the attachment object described below.Furthermore, the normal object may be displayed so as to move in a manner that makes it bounce back when it comes into contact with the character object.

[0075] When a normal object is displayed overlapping with a character object, for example, the normal object may be displayed so as to overlap with "portions other than the 'head', including the 'face', of the character object. In this case, the normal object is displayed so as not to overlap with the "head' of the character object. Also, the normal object may be displayed so as to overlap with "portions other than the 'upper body', including the 'face', of the character object. In this case, the normal object is displayed so as not to overlap with the "upper body' of the character object.

[0076] An attachment object is an object that is displayed in a user output image or another user output image in association with a specific part (attachment location) of a character object. For example, the specific part may be the front left, front right, rear left, rear right, front center, rear center, left eye, right eye, left ear, right ear, or all of the hair on the character object's head. The specific part may also be the thumb, ring finger, wrist, elbow, shoulder, upper arm, entire hand, entire arm, etc.

[0077] An attachment object associated with a specific part of a character object may be displayed in the user output image or another user output image so as to be in contact with the specific part of the character object. Furthermore, an attachment object associated with a specific part of a character object may be displayed in the user output image or another user output image so as to cover part or all of the specific part of the character object. The specific part may be specified by three-dimensional position information indicating a position in a three-dimensional coordinate space, or may be associated with the position information in the three-dimensional coordinate space.

[0078] Image information of the wearing object is, for example, image information that resembles accessories (headbands, necklaces, earrings, etc.) worn by the character object, clothing (T-shirts, dresses, etc.), costumes, and other items that can be worn by the character object.

[0079] As the placement location of the attachment object, information indicating the attachment part, which indicates which part of the character object the attachment object is associated with, is stored. For example, if the attachment object is a "headband," information indicating the "head" of the character object is stored as the placement location of the attachment object. Also, if the attachment object is a "T-shirt," information indicating the "torso" of the character object is stored as the placement location of the attachment object.

[0080] Information indicating multiple attachment positions in a three-dimensional coordinate space may be stored as the attachment position of an attachment object. For example, if the attachment object is a "headband," information indicating two attachment positions of the attachment object, the "left rear of the head" and the "right rear of the head" of a character object, may be stored. As a result, an attachment object resembling a "headband" is displayed as being attached to both the "left rear of the head" and the "right rear of the head" of the character object.

[0081] When a character object is equipped with multiple types of attachment objects that are attached to the same attachment location, the multiple types of attachment objects are attached to the character object at different times. As a result, multiple types of attachment objects that are attached to the same attachment location are not attached to the character object at the same time. For example, if the "head" is stored as the attachment location for both an attachment object representing a "headband" and an attachment object representing a "hat," the "headband" attachment object and the "hat" attachment object will not be displayed on the character object's head at the same time.

[0082] In the object table T2, a display time according to the type of additional object may be stored in association with the object ID of the additional object. For example, the display time of an attached object may be stored as a time longer than the display time of an effect object and the display time of a normal object. For example, if "60 seconds" is stored as the display time of an attached object, "5 seconds" may be stored as the display time of an effect object, and "10 seconds" may be stored as the display time of a normal object.

[0083] (Group table T3) 4(c) is an example of the data structure of a group table T3 that manages groups to which a user belongs (such as a group of users who are friends, a group with other users who are mutual followers, or a group arbitrarily created by the user or other users). A group table T3 is provided for each user, and the group table T3 shown in FIG. 4(c) is the group table T3 of the user who operates the terminal device 2 that stores the group table T3.

[0084] In the group table T3, for each group, the group ID of the group, the group name, the users belonging to the group, etc. are stored in association with each other. The group ID is an example of identification data for uniquely identifying each group. The name is an example of data indicating the name of each group. The users belonging to the group are the user IDs of each user belonging to the group.

[0085] (Other information) The terminal storage device 22 stores motion data corresponding to the action corresponding to the change instruction. Details of the motion data will be described later. For example, the terminal storage device 22 may store motion data for each of a plurality of actions associated with an action ID for identifying the action. Furthermore, a plurality of motion data may be associated with an action ID. In this case, one of the plurality of motion data may be stored in association with each relative position on the display screen of a character object receiving the action relative to a character object performing the action. For example, the relative position may be "information indicating that, on the display screen, the character object receiving the action is located adjacent to the left of the character object performing the action," "information indicating that, on the display screen, the character object receiving the action is located adjacent to the top of the character object performing the action," or "information indicating that, on the display screen, the character object receiving the action is located adjacent to another character object below the character object performing the action." This motion data is data for generating an animation that automatically moves a character object to which the motion data is applied for a predetermined time (e.g., three seconds). Furthermore, the terminal storage device 22 may store motion-received data, associated with an action ID, for causing a character object of a user who receives an action corresponding to a change instruction to react for a predetermined time.

[0086] Returning to FIG. 3, the functions of the generation unit 271, transmission unit 272, reception unit 273, display processing unit 274, and audio output unit 275 will be described.

[0087] (Generation part 271) The generation unit 271 acquires imaging data continuously output by the imaging device 25 from the imaging device 25. For example, the imaging device 25 of the terminal device 2 held by the user is provided on the terminal device 2 so that the imaging direction is the face of the user looking at the display screen of the display device 23. The imaging device 25 continuously captures images of the user's face in the imaging direction, acquires imaging data of the user's face, and outputs the data to the generation unit 271 of the terminal processing device 27. The imaging device 25 may acquire imaging data by capturing an image of a part of the user other than the face, for example, the head, arms, hands (including fingers), chest, torso, legs, or other parts. The imaging device 25 may also be a 3D camera capable of detecting the depth of a person's face.

[0088] The generation unit 271 generates face motion data, which is a digital representation of the user's facial movements, as time passes, based on continuously acquired imaging data. The face motion data may be generated at predetermined sampling time intervals. In this way, the face motion data generated by the generation unit 271 can digitally represent the user's facial movements (changes in facial expressions) in time series.

[0089] The generation unit 271 may generate body motion data, which is a digital representation of the position and orientation of each part of the user (head, arms, hands (including fingers), chest, torso, legs, or other parts), together with or separately from the facial motion data.

[0090] The body motion data may be generated based on detection information from a known motion sensor worn by the user. In this case, the terminal communication I / F 21 of the terminal device 2 has a predetermined communication circuit for wirelessly acquiring the detection information from the motion sensor worn by the user. The generation unit 271 then generates the body motion data based on the detection information acquired by the terminal communication I / F 21. The body motion data may be generated at predetermined sampling time intervals. In this way, the body motion data generated by the generation unit 271 can digitally represent the user's body movements in time series.

[0091] The generation of body motion data based on detection information from a motion sensor worn by a user may be performed, for example, in a film studio. In this case, the film studio may be equipped with a base station, a tracking sensor, and a display. The base station may be, for example, a multi-axis laser emitter. The motion sensor worn by the user may be, for example, a Vive Tracker provided by HTC CORPORATION, and the base station may be, for example, a base station provided by HTC CORPORATION.

[0092] The supporter computer may also be installed in a separate room adjacent to the photography studio. The display of the photography studio may be configured to display information received from the supporter computer. The server device 3 may be installed in the same room as the supporter computer. The room where the supporter computer is installed may be separated from the photography studio by a glass window. In this case, the operator of the supporter computer can see the user. The supporter computer may be configured to change the settings of various devices provided in the photography studio in response to operations by the supporter. The supporter computer can, for example, set the scanning interval by the base station, set the tracking sensor, and change various settings of various other devices. The operator may input a message to the supporter computer, and the input message may be displayed on the display of the photography studio.

[0093] The generation unit 271 generates character moving image data including facial motion data and / or body motion data generated as needed, and outputs the generated character moving image data to the display processing unit 274. Hereinafter, facial motion data and body motion data may be collectively referred to as motion data. The generation unit 271 also generates output information including the generated character moving image data and a user ID stored in the terminal storage device 22, and outputs the generated output information to the transmission unit 272. When the generation unit 271 acquires user voice data output from the microphone 26, it generates output information including the generated character moving image data, the acquired voice data, and the user ID stored in the terminal storage device 22, and outputs the generated output information to the transmission unit 272.

[0094] (Transmitter 272) The transmitting unit 272 transmits the output information output from the generating unit 271 together with the destination information to the server device 3 via the terminal communication I / F 21. The destination information includes the user IDs of one or more other users participating in the communication service with the user. The user IDs included in the output information may also be used as the destination information. For example, the server device 3 stores the user IDs of one or more other users participating in the communication service with the user, and upon acquiring the output information, identifies the user IDs of one or more other users participating in the communication service with the user whose user ID is included in the output information, and transmits the received output information to the terminal device 2 of the other users indicated by the identified user IDs.

[0095] Furthermore, the transmitting unit 272 transmits information indicating various instructions input by the user by operating the input device 24, together with the user ID and destination information stored in the terminal storage device 22, to the server device 3 via the terminal communication I / F 21. Various instructions from the user include, for example, change instructions to change the user's character object and other user's output image. The change instruction is transmitted to the server device 3 via the terminal communication I / F 21 together with change identification information (for example, an action ID) for identifying the content of the change, the user ID of the other user whose output image is to be changed, and the user ID of the user who issued the change instruction. The change instruction may also be transmitted together with the relative position on the display screen of the character object receiving the action with respect to the character object performing the action.

[0096] (Receiving unit 273) The receiving unit 273 receives output information of other users and information indicating various instructions of other users, etc., transmitted from the server device 3, via the terminal communication I / F 21. The receiving unit 273 outputs the received information to the terminal processing device 27. The output information of other users transmitted from the server device 3 is generated by the generating unit 271 of the terminal device 2 of the other user and transmitted to the server device 3 by the transmitting unit 272 of the terminal device 2 of the other user. The output information of other users includes character video data of other users, voice data of other users, user IDs of other users, etc. The output information of other users may not include voice data of other users, and the output information of other users may not include character video data of other users.

[0097] (Display processing unit 274) The display processing unit 274 displays an image drawn based on the motion data (character moving image data) generated by the generation unit 271 as a user output image on the display device 23. Furthermore, the display processing unit 274 displays an image drawn based on the motion data (character moving image data) included in the output information of other users received by the reception unit 273 as a other user output image on the display device 23. The image drawing process based on the motion data will be described below. In the image drawing process based on the motion data of the user (the user operating the terminal device 2), the user ID stored in the terminal storage device 22 is used, and in the image drawing process based on the motion data included in the output information of other users received by the reception unit 273, the user ID included in the output information is used.

[0098] First, the display processing unit 274 extracts the model ID of the character object associated with the user ID and the object ID of the use object from the user table T1. Next, the display processing unit 274 reads from the terminal storage device 22 the model data associated with the extracted model ID and the image information and placement location associated with the extracted object ID of the use object. Next, the display processing unit 274 generates an animation of the character object in which the use object is attached to the placement location based on the read model data, the image information and placement location of the use object, and the motion data. If the motion data includes facial motion data, the display processing unit 274 generates an animation of the character object so that the facial expression of the character object changes based on the facial motion data. As a result, the display processing unit 274 can generate an animation of the character object that moves in synchronization with the user's facial expression and body movement based on the read model data and facial motion data. If the motion data includes facial motion data and body motion data, the display processing unit 274 generates an animation of the character object that moves in synchronization with the user's facial expression and body movement based on the read model data, facial motion data, and body motion data.

[0099] Next, the display processing unit 274 draws an image including an animation (moving image) of the generated character object and a background image showing a background generated using background data stored in the terminal storage device 22. The display processing unit 274 may use a wallpaper object previously associated with a communication group and stored in the terminal storage device 22 as the background image. In this case, the same wallpaper object may be applied as the background image for all users in the communication group. The display processing unit 274 then outputs drawing data showing the drawn image and terminates the image drawing process based on the motion data. An image drawn based on a character object associated with a user ID stored in the terminal storage device 22 is the user output image of the user. An image drawn based on a character object associated with a user ID included in the received output information of another user is the other user output image of the other user.

[0100] Furthermore, the display processing unit 274 may display on the display device 23 the user's character object that automatically moves in response to the user's change instruction, and other user's output image that automatically changes in response to the user's change instruction. For example, if the content of the change in the user's change instruction is an action of the user's character object hitting another user's character object, the display processing unit 274 first specifies the display positions of the user's character object and the other user's character object. Note that the other user's character object that receives the hitting action from the user's character object is the character object designated by the user.

[0101] The display processing unit 274 extracts motion data stored in the terminal storage device 22 based on the content of the change in the change instruction (a tapping motion) and the display positions of the user's character object and the other user's character object. Next, based on the extracted motion data, the display processing unit 274 generates an animation in which the arm of the user's character object extends toward the display position of the other user's character object and then returns to its original position. The display processing unit 274 then generates a user output image including the generated animation and a background image. In this case, the animation is automatically displayed for a predetermined time (e.g., three seconds) using pre-stored motion data, without using the user's motion data generated by the generation unit 271. Note that, when generating an animation related to the user's character object in response to the user's change instruction, a portion of the user's motion data generated by the generation unit 271 may be used. For example, if the content of the change in the user's change instruction is a motion related to the character object's arm, the facial motion data may continue to be used. This allows the information system 1 to allow the user's character object to express a variety of expressions rather than just using a uniform movement in response to the user's change instruction, thereby increasing the user's motivation to continue using the information system 1 for communication with the user.

[0102] Next, the display processing unit 274 generates an animation in which the arm of the user's character object extends from the direction of the display position of the user's character object, hits the other user's character object, returns to the original position, and then disappears. The display processing unit 274 then generates an other user output image including the generated animation and a background image. In this case, the animation is automatically displayed for a predetermined time (e.g., 3 seconds) using pre-stored motion data, without using the other user's motion data generated in the other user's terminal device 2. Even in this case, some of the other user's motion data generated by the other user's terminal device 2 may be used to generate an animation for the other user's character object in response to a user's change instruction. For example, the other user's facial motion data generated by the other user's terminal device 2 may continue to be used in a change (reaction) of the other user's character object in response to a user's change instruction. This allows the information system 1 to allow the other user's character object to express a variety of expressions, rather than just using uniform movements in response to the user's change instruction, thereby increasing the user's motivation to continue communicating with the information system 1.

[0103] Then, the display processing unit 274 displays the generated user output image and other user output image on the display device 23.

[0104] Furthermore, the display processing unit 274 displays a user output image including an additional object in response to an instruction from another user to add the additional object. The display processing unit 274 displays a user output image including an additional object in response to an instruction from a user to display the additional object in the other user output image.

[0105] When the object ID included in the instruction to add an additional object from another user is the object ID of one of the effect objects, the display processing unit 274 references the object table T2 and extracts from the terminal storage device 22 a still image or a moving image (image information) of the effect object associated with the object ID included in the instruction to add, and generates a user output image including the extracted still image or moving image. For example, when an instruction to add an effect object representing confetti or fireworks is given, the display processing unit 274 generates a user output image including a moving image of an effect object (not shown) that resembles confetti or fireworks. When the object ID included in the instruction to add an additional object from a user is the object ID of one of the effect objects, the display processing unit 274 similarly references the object table T2 and extracts from the terminal storage device 22 a still image or a moving image (image information) of the effect object associated with the object ID included in the instruction to add, and generates a user output image including the extracted still image or moving image.

[0106] When the object ID included in the instruction to add an additional object from another user is the object ID of a normal object, the display processing unit 274 refers to the object table T2 and extracts a still image or a moving image (image information) of the normal object associated with the object ID of the normal object and its placement location. Next, the display processing unit 274 generates a user output image including a still image or a moving image of the normal object placed at the position indicated by the placement location. Furthermore, when the object ID included in the instruction to add an additional object from the user is the object ID of a normal object, the display processing unit 274 similarly refers to the object table T2 and extracts a still image or a moving image (image information) of the normal object associated with the object ID of the normal object and its placement location, and generates an other user output image including a still image or a moving image of the normal object placed at the position indicated by the placement location.

[0107] The display processing unit 274 may generate a user output image and another user output image including a moving image of a normal object moving within the display area of ​​the user output image and the other user output image. Hereinafter, the user output image and the other user output image may be collectively referred to as output images. For example, the display processing unit 274 may generate an output image including a moving image of a normal object falling from top to bottom of the output image. In this case, the normal object may be displayed within the display area of ​​the output image from the start of its fall until it reaches the bottom of the output image, and may disappear from the output image after it reaches the bottom of the output image. Note that the movement route of the normal object may be from left to right, from right to left, from top left to bottom left, or a route in any other direction on the output image. It may also be a route along a linear trajectory, a circular trajectory, an elliptical trajectory, a spiral trajectory, or any other trajectory.

[0108] When the object ID included in an instruction to add an additional object from another user is the object ID of a worn object, the display processing unit 274 displays notification information regarding the wearing instruction in the user output image. In response to the wearing instruction from the user, the display processing unit 274 references the object table T2 and extracts image information and a placement location of the wearing object associated with the object ID included in the addition instruction. Next, based on the extracted image information and placement location, the display processing unit 274 generates a user output image including a character object wearing the wearing object at the wearing location indicated by the placement location.

[0109] (Audio output unit 275) When the output information of another user includes voice data of the other user, the voice output unit 275 outputs the voice of the other user generated based on the voice data from a speaker (not shown) while the display processing unit 274 displays the output image of the other user. As a result, the voice of the other user is output while the lips of the character object of the other user included in the output image of the other user are moving, so the user can see the output image of the other user including the character object of the other user that appears to be making a sound.

[0110] (Server device 3) 5 is a diagram showing an example of a schematic configuration of a server device 3. The server device 3, which provides communication services to the terminal devices 2 of multiple users, transmits output information and information indicating various instructions transmitted from a terminal device 2 of one user to a terminal device 2 of another user different from the one user. To this end, the server device 3 includes a server communication I / F 31, a server storage device 32, and a server processing device 33. For example, the terminal device 2 to which information is transmitted by the terminal device 2 of one user is the terminal device 2 of the user indicated by the user ID included in the received destination information.

[0111] The server communication I / F 31 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination of these. The server device 3 can transmit information to other devices and receive information from other devices via the server communication I / F 31.

[0112] The server storage device 32 is, for example, a semiconductor memory device such as a ROM or RAM. The server storage device 32 may be, for example, a magnetic disk, an optical disk, or any other storage device capable of storing data. The server storage device 32 stores an operating system program, a driver program, an application program, data, etc. used for processing in the server processing device 33. The server storage device 32 may also store a user table T1, an object table T2, and a group table T3 as data.

[0113] The server processing device 33 is a processor that loads the operating system program, driver program, and control program stored in the server storage device 32 into memory and executes instructions contained in the loaded programs. The server processing device 33 is, for example, an electronic circuit such as a CPU, MPU, DSP, or GPU, or a combination of various electronic circuits. The server processing device 33 may be realized by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. Although FIG. 5 illustrates the server processing device 33 as a single component, the server processing device 33 may also be a collection of multiple physically separate processors. The server processing device 33 functions as a server receiving unit 331 and a server transmitting unit 332 by executing various instructions contained in the control program.

[0114] (Server receiving unit 331) The server receiving unit 331 receives, via the server communication I / F 31, information indicating output information and various instructions transmitted from the terminal device 2 of any one of the multiple users. The server receiving unit 331 may also receive destination information along with the output information and information indicating various instructions transmitted from the terminal device 2.

[0115] (Server transmission unit 332) The server transmitting unit 332 transmits the output information and information indicating various instructions received by the server receiving unit 331 to the terminal device 2 of the other user indicated by the user ID included in the destination information received at the same time, via the server communication I / F 31. The server transmitting unit 332 may also transmit the information indicating various instructions received by the server receiving unit 331, via the server communication I / F 31, to one or more terminal devices 2 of the other users designated by the user who transmitted the information.

[0116] (Examples of various screens) 6 to 15, examples of various screens displayed on the display device 23 of the user's terminal device 2 will be described below. In FIGS. 6 to 15, multiple components denoted by the same reference numerals are components having equivalent functions.

[0117] 6(a) is a diagram showing an example of a group creation screen 600 displayed on the display device 23 of the user's terminal device 2. The group creation screen 600 includes an other user display area 601, a selection object 602, and a create button 603. For example, the group creation screen 600 is displayed when a group creation object or the like on a home screen (not shown) displayed in response to execution of a control program stored in the terminal storage device 22 is selected by the user operating the input device 24.

[0118] Other user information indicating each of the other users who have a predetermined relationship with the user is displayed in the other user display area 601. In the example shown in Fig. 6(a), the display processing unit 274 of the user's terminal device 2 displays thumbnail images of the character objects of the other users and the names of the other users as the other user information.

[0119] The selection object 602 is an operation object for selecting each other user indicated by each other user information. In the example shown in FIG. 6( a), the display processing unit 274 of the user's terminal device 2 displays a check box object corresponding to each other user information as the selection object 602. For example, when the selection object 602 is selected by the user's operation of the input device 24, the selection object 602 is displayed with a "check" mark. In this case, the other user in the other user information corresponding to the selection object 602 with a display with a "check" mark is selected. Furthermore, when the selection object 602 with a display with a "check" mark is selected by the user's operation of the input device 24, the selection object 602 is displayed without a "check" mark. In this case, the other user in the other user information corresponding to the selection object 602 with a display without a "check" mark is not selected. Each selection object 602 is associated with the user ID of the other user associated with the corresponding other user information.

[0120] The create button 603 is a button object or the like for the user to create a new communication group. When the user selects the create button 603 by operating the input device 24, a new group is created in which the user operating the terminal device 2 and the other user selected in the selection object 602 are the belonging users. For example, the display processing unit 274 of the user's terminal device 2 identifies the user ID of the user stored in the terminal storage device 22 and the user ID of the other user associated with the selection object 602 related to the other user selected in the selection object 602. Then, the display processing unit 274 associates the user ID of the identified user and the user ID of the other user as belonging users with the newly created group ID and stores them in the group table T3. At this time, the display processing unit 274 may automatically associate the name of the created group with the newly created group ID, or may associate the name of the group input by the user with the newly created group ID.

[0121] When a new group is created in response to selection of the create button 603, the display processing unit 274 displays an information exchange screen 700 for users belonging to the new group. Fig. 7(a) is a diagram showing an example of the information exchange screen 700 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 700 includes a display area 701 that displays information such as text or images entered by each of the users belonging to the new group, information 702 (such as a thumbnail image of a character object) indicating the users who entered the information displayed in the display area 701, an input object 703 that allows the user to enter text or images to be displayed in the new display area 701, and a start object 704 for starting the communication service.

[0122] When the information exchange screen 700 is displayed, the transmitting unit 272 of the user's terminal device 2 transmits an instruction to display the information exchange screen, together with information about the new group and destination information, to the server device 3 via the terminal communication I / F 21. The information about the new group includes the group ID and name of the new group, and the user IDs of each of the users belonging to the new group. The destination information includes the user IDs of each of the users belonging to the new group excluding the user operating the terminal device 2. The server receiving unit 331 of the server device 3 receives the instruction to display the information exchange screen, the information about the new group, and the destination information via the server communication I / F 31. The server receiving unit 331 may store the received destination information in the server storage device 32. The server transmitting unit 332 of the server device 3 transmits the instruction to display the information exchange screen and the information about the new group to the terminal device 2 of the other user identified by the user ID included in the destination information, via the server communication I / F 31. When the other user's terminal device 2 receives an instruction to display an information exchange screen sent from the user's terminal device 2 via the server device 3, it can display an information exchange screen 700 that allows information to be exchanged with users belonging to the new group.

[0123] Returning to Fig. 7(a), for example, when new information is input into input object 703 by an operation of one of the belonging users, a display area 701 displaying the new information is added to the information exchange screen 700 displayed by the terminal device 2 of the user who input the information. Furthermore, the transmission unit 272 of the terminal device 2 of the user who input the information transmits the input new information to the terminal devices 2 of the belonging users other than the user who input the information via the server device 3. As a result, a display area 701 displaying the new information is also added to the information exchange screen 700 displayed on the terminal devices 2 of the other belonging users other than the user who input the information.

[0124] The start object 704 is a button object or the like for starting a communication service in which each user belonging to the new group can participate. When the start object 704 is selected by the user operating the input device 24, a start process for a communication service is executed in which groups that can exchange information via the information exchange screen 700 are set as communication groups. An example of the start process in response to the user's selection of the start object 704 will be described below.

[0125] First, the display processing unit 274 of the user's terminal device 2 displays a communication screen 810 (FIG. 8(b)) that can be displayed by the terminal device 2 of each user belonging to the communication group. The display processing unit 274 also stores the group ID of the communication group in the terminal storage device 22 as the group ID of the group that has started the communication service. Furthermore, the display processing unit 274 stores the user's user ID in the terminal storage device 22 as the user ID of the user currently participating in the communication service. Note that if all of the other users belonging to the communication group have not yet participated in the communication service, only the user output information of the user is displayed on the communication screen 810.

[0126] Next, the transmitting unit 272 of the user's terminal device 2 transmits a communication service start instruction to the server device 3 via the terminal communication I / F 21 together with the user's user ID, information about the communication group, and / or destination information stored in the terminal storage device 22. The information about the communication group includes the group ID and name of the communication group, and the user IDs of each of the users belonging to the communication group. The destination information includes the user IDs of each of the users belonging to the communication group, excluding the user operating the terminal device 2. Note that if the destination information is stored in the server storage device 32, the destination information does not need to be transmitted. The server receiving unit 331 of the server device 3 receives the start instruction, the user ID, information about the communication group, and / or destination information via the server communication I / F 31. The server transmitting unit 332 of the server device 3 transmits the start instruction, the user ID, and information about the communication group to the terminal device 2 of the other user identified by the user ID included in the destination information via the server communication I / F 31.

[0127] The receiving unit 237 of the other user's terminal device 2 receives from the server device 3 the start instruction, the user ID, and information about the communication group transmitted from the user's terminal device 2. The receiving unit 237 stores the group ID from the information about the communication group in the terminal storage device 22 as the group ID of the group that has started the communication service. Furthermore, the display processing unit 274 stores the transmitted user ID in the terminal storage device 22 as the user ID of the user participating in the communication service. The display processing unit 274 of the other user's terminal device 2 displays a notification screen (not shown) based on the received information about the communication group. In response to the user's selection of a predetermined object in the notification screen, the display processing unit 274 displays a group selection screen 710 ( FIG. 7(b) ) for participating in the communication service on the display device 23. This completes the start processing. The display processing unit 274 of the other user's terminal device 2 may include a join button 805 (described later) in the notification screen. This allows other users to immediately participate in the communication service without displaying the group selection screen 710.

[0128] The start process may be executed when the user selects the create button 603 on the group creation screen 600 by operating the input device 24. That is, in response to the user selecting the create button 603, a start process for a communication service in which a new group created on the group creation screen 600 is used as a communication group may be executed. In this case, the information exchange screen 700 is not displayed.

[0129] Furthermore, the start process is not limited to start process of a communication service in which a new group created on the group creation screen 600 is used as a communication group. For example, start process of a communication service in which an already created group is used as a communication group may be executed. For example, FIG. 6(b) is a diagram showing an example of a group selection screen 610 for selecting a communication group created by any user. For example, the group selection screen 610 is displayed when a group selection object or the like on a home screen (not shown) displayed in response to execution of a control program stored in the terminal storage device 22 is selected by the user operating the input device 24.

[0130] The group selection screen 610 displayed on the user's terminal device 2 includes a group display area 611 for selecting one of multiple communication groups. The group display area 611 displays group information indicating each of the communication groups that have already been created and to which the user belongs. In the example shown in FIG. 6(b), the display processing unit 274 of the user's terminal device 2 displays, as group information, thumbnail images of each of the users who belong to the communication group and the name of the group. In the example shown in FIG. 6(b), thumbnail images of each of the other users among the belonging users are displayed, but thumbnail images of each of all the belonging users may also be displayed. Note that if a communication group includes only one other user in addition to the user, the name of the one other user may be displayed instead of the name of the group, as shown in FIG. 8(b).

[0131] When a piece of group information included in the group display area 611 is selected by the user operating the input device 24, an information exchange screen 700 by users belonging to the communication group corresponding to the selected group information is displayed. The information exchange screen 700 includes information such as text or images entered by each of the users belonging to the communication group. In this case, too, an instruction to display the information exchange screen and information about the communication group are transmitted via the server device 3 to the terminal devices 2 of the other users belonging to the communication group, and the information exchange screen 700 is also displayed on the terminal devices 2 of the other users.

[0132] Then, when the user selects the start object 704 on the information exchange screen 700, a process for starting a communication service by a communication group that will exchange information on the information exchange screen 700 is executed. When the start process is executed, the display processing unit 274 of the user's terminal device 2 displays a communication screen 810 (FIG. 8(b)) that can be displayed on the terminal device 2 of each user belonging to the communication group. Furthermore, the display processing unit 274 of the other user's terminal device 2 displays a notification screen (not shown), and displays a group selection screen 710 (FIG. 7(b)) for participating in the communication service on the display device 23 in response to an operation by the other user on the notification screen. Note that even in this case, the display processing unit 274 of the other user's terminal device 2 may include a join button 805, which will be described later, on the notification screen.

[0133] In this case, the start process may also be executed when a piece of group information is selected on the group selection screen 610 by the user operating the input device 24. That is, in response to the user's selection of a piece of group information, a start process for a communication service by the communication group corresponding to the selected group information may be executed. In this case, the information exchange screen 700 is not displayed.

[0134] Next, a method for a user who has not yet joined a communication service that has already started to join will be described. Fig. 7(b) is a diagram showing an example of a group selection screen 710 displayed on the display device 23 of the user's terminal device 2. For example, the group selection screen 710 is displayed in response to the selection of a predetermined object in the displayed notification screen. Alternatively, the group selection screen 710 may be displayed when a group selection object or the like on a home screen displayed in response to the execution of a control program stored in the terminal storage device 22 is selected by the user's operation of the input device 24.

[0135] In the group display area 711 of the group selection screen 710, group information indicating a communication group that has started a communication service is displayed so as to be distinguishable from other group information. In the example shown in FIG. 7(b), the group information displayed at the top indicates the communication group that has started a communication service. For example, information indicating the number of users who are participating in the communication service is displayed superimposed on or near each thumbnail image of the users who belong to the communication group. In addition, a mark image 712 indicating the communication service is displayed near the name of the communication group.

[0136] When a user operates the input device 24 to select group information indicating a communication group that has started a communication service from the group information included in the group display area 711, an information exchange screen 800 for users belonging to the communication group corresponding to the selected group information is displayed.

[0137] 8(a) is a diagram showing an example of an information exchange screen 800 displayed on the display device 23 of the user's terminal device 2. The information exchange screen 800 includes display areas 801a to 801b that display information such as text or images entered by each of the users belonging to the communication group that has started a communication service, information 802a to 802b (such as thumbnail images of character objects) indicating the users who entered the information displayed in each of the display areas 801a to 801b, an input object 803 that allows the user to enter text or images to be displayed in the new display area 801, a service display area 804 that indicates a communication service that has already started, and a join button 805 for joining a communication service that has already started.

[0138] Thumbnail images of character objects of three users participating in the communication service are displayed in the service display area 804 shown in FIG. 8(a). Information indicating that the communication service has started is displayed in the display area 801b shown in FIG. 8(a). The participation process is executed when the user selects the participation button 805. An example of the participation process will be described below.

[0139] When the user selects the join button 805 by operating the input device 24, first, the display processing unit 274 of the user's terminal device 2 stores the user ID of the user stored in the terminal storage device 22 in the terminal storage device 22 as the user ID of the user participating in the communication service related to the service display area 804. Next, the display processing unit 274 displays a communication screen 810 (FIG. 8(b)) including user output images of all users participating in the communication service.

[0140] Next, the transmitting unit 272 of the user's terminal device 2 transmits a participation instruction for the communication service to the server device 3 via the terminal communication I / F 21 together with the user's user ID, information about the communication group, and / or destination information stored in the terminal storage device 22. The information about the communication group includes the group ID and name of the communication group, and the user IDs of each of the users belonging to the communication group. The destination information includes the user IDs of each of the other users belonging to the communication group excluding the user operating the terminal device 2. The server receiving unit 331 of the server device 3 receives the participation instruction, the user ID, information about the communication group, and / or destination information via the server communication I / F 31. The server transmitting unit 332 of the server device 3 transmits the participation instruction, the user ID, and information about the communication group to the terminal device 2 of the other user identified by the user ID included in the destination information via the server communication I / F 31.

[0141] The receiving unit 237 of the other user's terminal device 2 receives from the server device 3 the participation instruction, user ID, and information about the communication group transmitted from the user's terminal device 2. The receiving unit 237 stores the received user ID in the terminal storage device 22 as the user ID of the user participating in the communication service. When the communication screen 810 is displayed on the terminal device, the display processing unit 274 of the other user's terminal device 2 displays the communication screen 810 including user output images of all users participating in the communication service. This completes the participation process.

[0142] The selection process may be executed when group information is selected on the group selection screen 710 by the user operating the input device 24. That is, in response to the user's selection of group information, a process of participating in a communication service in which the communication group corresponding to the group information can participate may be executed. In this case, the information exchange screen 800 is not displayed.

[0143] The embodiment of starting or joining a communication service is not limited to the above-described example. For example, other start processes different from the start process responsive to the user's selection of the start object 704 may be executed. The other start processes are executed when the user selects a predetermined display object for specifying the start of a communication service on a predetermined screen, such as a home screen. Note that the predetermined display object may be displayed when a thumbnail image of the user's character object is selected, or may be included in a menu item on the predetermined screen. The other start processes may be automatically executed when a predetermined start condition is satisfied. For example, the predetermined start conditions may include the current time reaching a predetermined time, a predetermined time having elapsed since the end of the previous communication service, or the number of mutual followers of the users exceeding a predetermined number. An example of the other start processes is described below.

[0144] First, the transmitting unit 272 of the user's terminal device 2 transmits an instruction to start a communication service to the server device 3 via the terminal communication I / F 21, together with the user's user ID and / or destination information stored in the terminal storage device 22. The transmitting unit 272 also stores the user's user ID in the terminal storage device 22 as the user ID of a user participating in the communication service. The destination information includes the user IDs of one or more other users who have a predetermined relationship with the user. For example, the destination information may include the user IDs of one or more other users who are mutual followers of the user. Furthermore, the destination information may include the user IDs of other users who have a mutual follow relationship with a predetermined user who is a mutual follower of the user, instead of or in addition to the user IDs of one or more other users who are mutual followers of the user.

[0145] The server receiving unit 331 of the server device 3 receives a start instruction, a user ID, and / or destination information via the server communication I / F 31. The server transmitting unit 332 of the server device 3 transmits the start instruction, the user ID, and the destination information to the terminal device 2 of the other user identified by the user ID included in the destination information via the server communication I / F 31. The receiving unit 237 of the terminal device 2 of the other user receives the start instruction, the user ID, and the destination information transmitted from the user's terminal device 2 from the server device 3. The receiving unit 237 stores the transmitted user ID in the terminal storage device 22 as the user ID of the user participating in the communication service. The display processing unit 274 of the terminal device 2 of the other user displays a notification screen (not shown) on the display device 23 based on the received user ID. The notification screen includes information encouraging the user identified by the received user ID to participate in the communication service, a participation button, and the like.

[0146] When the other user operates the input device 24 to select the join button in the notification screen, the display processing unit 274 of the other user's terminal device 2 stores the user ID of the other user stored in the terminal storage device 22 in the terminal storage device 22 as the user ID of the user participating in the communication service related to the service display area 804. Next, the display processing unit 274 displays a communication screen 810 (FIG. 8(b)) including user output images of all users participating in the communication service.

[0147] Next, the transmitting unit 272 of the other user's terminal device 2 transmits a participation instruction for the communication service to the server device 3 via the terminal communication I / F 21 together with the other user's user ID and / or destination information stored in the terminal storage device 22. The destination information is obtained by deleting the user ID of the other user of the terminal device 2 transmitting the destination information from the user ID included in the destination information previously received by the receiving unit 237 and adding the user ID received by the receiving unit 237 at the same time. The server receiving unit 331 of the server device 3 receives the participation instruction, the user ID, and / or the destination information via the server communication I / F 31. The server transmitting unit 332 of the server device 3 transmits the participation instruction, the user ID, and the destination information to the other user's terminal device 2 identified by the user ID included in the destination information via the server communication I / F 31.

[0148] The receiving unit 237 of the terminal device 2 of the user to whom the participation instruction is to be sent receives the participation instruction, user ID, and destination information sent from the user's terminal device 2 from the server device 3. The receiving unit 237 stores the received user ID in the terminal storage device 22 as the user ID of the user participating in the communication service. When the communication screen 810 is displayed on the terminal device 2, the display processing unit 274 of the terminal device 2 displays the communication screen 810 including user output images of all users participating in the communication service. This completes the series of participation processes.

[0149] Note that embodiments of starting or participating in a communication service with other users who have a predetermined relationship with the user are not limited to the above-described examples. For example, when a user selects a predetermined display object for specifying the start of a communication service, a selection screen including information indicating other users who are mutual followers of one or more users may be displayed. In this case, when the user selects information indicating one of the other users, another start process may be executed based on a start instruction to start communication with the selected other user. Furthermore, in this case, an information exchange screen with the selected other user may be displayed, and by selecting a start button on the information exchange screen, another start process may be executed based on a start instruction to start communication with the other user.

[0150] In conventional information systems, a user cannot use a simple interface to start and / or issue a command to join a communication with another user who has a predetermined relationship with the user, such as the user's mutual follower. In the information system 1 disclosed in this embodiment, the above-mentioned other start process is executed, thereby improving the user interface related to the start and / or issue of a communication command, and reducing the communication load between the server device 3 and the terminal device 2.

[0151] 8(b) is a diagram showing an example of a communication screen 810 displayed on the display device 23 of the terminal device 2. The communication screen 810 is a communication screen of a communication group that includes the user operating the terminal device 2 as a member user. Hereinafter, the user operating the terminal device 2 will be referred to as user A, and users other than user A who are participating in the communication service corresponding to the communication screen 810 will be referred to as user B1, user B2, user B3, .... Note that user B1, user B2, user B3, ... may be collectively referred to as user B.

[0152] The communication screen 810 includes at least a user output image 812a including a character object 811a of user A. In the example shown in FIG. 8(b), there are three users (user B1, user B2, and user B3) other than user A who are participating in the communication service. That is, the communication screen 810 includes user output images 812b1, 812b2, and 812b3 including character objects 811b1, 811b2, and 811b3 of user B1, user B2, and user B3, respectively. Hereinafter, the character objects 811b1, 811b2, and 811b3 may be collectively referred to as character object 811b, and the user output images 812b1, 812b2, and 812b3 may be collectively referred to as user output images 812b. Note that user B is another user from user A's perspective, and user output image 812b of user B displayed on user A's terminal device 2 is an another user output image.

[0153] When there are no users other than user A participating in the communication service, only user output image 812a is included in communication screen 810, and each time user B participates in the communication service, user output image 812b of user B is newly included in communication screen 810. There is no upper limit to the number of users who can participate in the communication service, and for example, when nine users B other than user A are participating in the communication service, user output images 812b1, 812b2, 812b3, ... 812b9 of each of the nine users B are included in communication screen 810, along with user output image 812a of user A.

[0154] The character object 811a is an animation (moving image) of the character object of user A generated by the display processing unit 274 based on the motion data of user A generated by the generation unit 271 and the character object and use object associated with the user ID of user A in the user table T1. The motion data of user A generated by the generation unit 271 is the same as the motion data included in the output information transmitted to the terminal device 2 of user B. The output information of user A includes the motion data of user A as well as the voice data of user A. When the motion data of user A generated by the generation unit 271 includes face motion data of user A, a user output image 812a is displayed that includes the character object 811a whose facial expression changes in synchronization with the movement of the facial expression of user A. For example, when user A is making a sound, a user output image 812a is displayed that includes the character object 811a whose lips are moving. In addition, if the motion data of user A generated by the generation unit 271 includes body motion data of user A, a user output image 812a including a character object 811a whose corresponding body part moves in synchronization with the body movement of user A is displayed.

[0155] The transmitting unit 272 of user A's terminal device 2 transmits user A's output information together with destination information (the user IDs of each user B (users B1 to B3) included in the communication group) to the server device 3 via the terminal communication I / F 21. Next, the server receiving unit 331 of the server device 3 receives user A's output information transmitted from user A's terminal device 2 via the server communication I / F 31. Next, the server transmitting unit 332 of the server device 3 references the received user IDs of users B (users B1 to B3) and transmits user A's output information to user B's (users B1 to B3) terminal device 2 via the server communication I / F 31. Then, when the receiving unit 273 of user B's terminal device 2 receives the output information via the terminal communication I / F 21, the display processing unit 274 of user B's terminal device 2 displays a user output image 812a including a character object 811a based on the motion data of user A identified by the user ID included in the output information.

[0156] Character object 811b is an animation (moving image) of user B's character object generated by the display processing unit 274 based on the motion data included in the output information of user B received by the receiving unit 273 and the character object associated with user B's user ID in the user table T1. If the motion data of user B received by the receiving unit 273 includes face motion data of user B, a user output image 812b is displayed that includes character object 811b whose facial expression changes in synchronization with the movement of user B's facial expression. For example, when user B utters a sound, a user output image 812b is displayed that includes character object 811b whose lips are moving. Furthermore, if the motion data of user B generated by the generation unit 271 includes body motion data of user B, a user output image 812b is displayed that includes character object 811b whose corresponding body parts are moving in synchronization with the movement of user B's body. If the output information of user B received by the receiving unit 273 includes user B's voice data, the display processing unit 274 displays the user output image 812b, and the voice of user B generated based on the voice data is output by the voice output unit 275.

[0157] In this way, by displaying the communication screen 810 including the user output image 812a and the user output image 812b, user A can hear the voice of user B while looking at the character object 811b of user B, which appears to be making a sound. Furthermore, the character object and voice data of user A are output by the terminal device 2 of user B. This allows user A and user B, who are participating in the communication service, to communicate via the character objects.

[0158] When user A wants to change the character object 811a and also the user output image 812 of user B, the user A performs an operation to select the user output image 812 of user B that the user A wants to change or the character object 811 of user B that the user A wants to change. The following describes an example in which the user output image that the user A wants to change is user output image 812b1. When user A operates the input device 24 to select character object 811b1, the display of the communication screen 810 ends and the selection screen 900 (FIG. 9(a)) is displayed. Note that the display of the selection screen 900 may be executed when user A operates the input device 24 to select the user output image 812b1.

[0159] 9(a) is a diagram showing an example of a selection screen 900 displayed on the display device 23 of the terminal device 2. The selection screen 900 is displayed when a character object 811b1 or a user output image 812b1 included in the communication screen 810 (FIG. 8(b)) is selected by user A operating the input device 24. Note that when user A operates the input device 24 to select the character object 811b1 or the user output image 812b1, the selection screen 900 may not be displayed, and a change candidate button 901, which will be described later, may be displayed on the user output image 812b1.

[0160] On the selection screen 900, a user output image 812b1 including a character object 811b1 of user B1 is displayed on the entire screen. Furthermore, the selection screen 900 includes a change candidate button 901 and a screen close button 902. Note that the change candidate button 901 does not have to be included on the selection screen 900 when it is displayed. In this case, when a specific operation by user A on the input device 24 (for example, a "long press" operation or a "tap" operation within the display area of ​​the selection screen 900) is detected by the input device 24, the display processing unit 274 may display the change candidate button 901.

[0161] The change candidate buttons 901 are selection objects for selecting an action to be performed on the character object 811b1 of user B1 or the user output image 812b1 of user A selected on the communication screen 810 (FIG. 8(b)). In the example shown in FIG. 9(a), three change candidate buttons 901 corresponding to three actions are included in the selection screen 900. The number of change candidate buttons 901 displayed on the selection screen 900 is not limited to three. When the display processing unit 274 of the terminal device 2 cannot include some of the images showing the multiple change candidate buttons 901 within the selection screen 900, the change candidate buttons 901 may be displayed in a scrollable manner. In this case, in response to a specific operation by user A (for example, a swipe operation in which a finger is moved from the bottom of the screen to the top or a flick operation in which a finger is moved in a flicking manner upward on the screen), the change candidate buttons 901 that are not currently displayed on the selection screen 900 are scrolled and displayed within the selection screen 900.

[0162] When user A operates the input device 24 to select one of the plurality of change candidate buttons 901, the input device 24 inputs a change instruction to the terminal processing device 27. Along with the change instruction, the input device 24 inputs to the terminal processing device 27 the user ID of user B1 corresponding to the character object displayed on the selection screen 900 (the character object 811b1 selected on the communication screen 810) and the action ID corresponding to the selected change candidate button 901. An example of change processing in response to a change instruction will be described below.

[0163] First, the display processing unit 274 of user A's terminal device 2 acquires the change instruction, user B1's user ID, and action ID input from the input device 24. Next, the display processing unit 274 specifies the relative position on the display screen of user B1's character object 811b1 with respect to user A's character object 811a on the communication screen 810. Next, the display processing unit 274 extracts from the terminal storage device 22 the motion data and the motioned data associated with the specified relative position from the motion data and the motioned data associated with the action ID.

[0164] The display processing unit 274 generates an animation (moving image) of user A's character object 911a based on the extracted motion data and the character object and used object associated with user A's user ID in user table T1. Then, the display processing unit 274 generates a user output image 912a including the generated animation (moving image) of user A's character object 911a. In this way, the display processing unit 274 automatically generates user A's changed character object 911a based on the positional relationship between character object 811a and character object 811b1 and the motion data stored in the terminal storage device 22.

[0165] The display processing unit 274 displays the generated user output image 912a on the display device 23, thereby displaying the character object 911a of user A moving so that its arm is stretched toward a predetermined position of the character object 911b1 of user B. In this case, the arm of the character object 911a of user A is drawn within the display area of ​​the user output image 812b of user A in FIG. 8(b).

[0166] Furthermore, the display processing unit 274 generates a user output image 912b1 including an animation (moving image) of an object 911a1 that is a newly appeared part of user A's character objects and an animation (moving image) of a changed character object 911b1 of user B1. For example, the animation (moving image) of object 911a1 that is a newly appeared part of user A's character objects is generated by the display processing unit 274 based on the extracted motion data and the character object and used object associated with user A's user ID in the user table T1. Furthermore, the animation (moving image) of user B1's character object 911b1 that moves in accordance with object 911a1 is generated by the display processing unit 274 based on the extracted motion-received data and the character object and used object associated with user B1's user ID in the user table T1. In this way, the movement of user B's character object 911b1 is based on the motion-received data associated with the movement ID, and is therefore automatically identified in response to a change instruction from user A.

[0167] Then, the display processing unit 274 displays a communication screen 910 (FIG. 9(b)) including the generated user output image 912a and user output image 912b1. Note that user output images 812b2 and 812b3 including character objects 811b2 and 811b3 of users B2 and B3, respectively, who are participating in the communication service, are the same as those in FIG. 8(b). Thus, on the communication screen 910 shown in FIG. 9(b), a user output image 912a including a character object 911a performing an action corresponding to a change instruction from user A is displayed. Furthermore, in accordance with the action of user A's character object 911a corresponding to the change instruction from user A, a user output image 912b1 including user B1's character object 911b1 performing an action related to the action of user A's character object 911a is displayed.

[0168] Next, when the transmitting unit 272 of the terminal device 2 of user A acquires the change instruction, the user ID of user B1, and the action ID input from the input device 24, the transmitting unit 272 transmits information indicating the change instruction, together with the user ID of user A stored in the terminal storage device 22, the acquired user ID and action ID of user B1, and destination information (user IDs of users B (users B1 to B3)), to the server device 3 via the terminal communication I / F 21. Next, the server receiving unit 331 of the server device 3 receives the information indicating the change instruction, the user ID of user A, the user ID of user B1, the action ID, and the destination information via the server communication I / F 31. Next, the server transmitting unit 332 of the server device 3 transmits the information indicating the change instruction, the user ID of user A, the user ID of user B1, and the action ID to the terminal device 2 of user B (users B1 to B3) via the server communication I / F 31.

[0169] Next, the receiving unit 273 of user B's terminal device 2 receives the information indicating the change instruction, the user ID of user A, the user ID of user B1, and the action ID via the terminal communication I / F 21. Next, the display processing unit 274 of user B's terminal device 2 identifies the relative position on the display screen of user B1's character object with respect to user A's character object in the communication screen displayed on user B's terminal device 2. Next, the display processing unit 274 extracts from the terminal storage device 22, from the motion data and motioned data associated with the action ID, the motion data and motioned data associated with the identified relative position.

[0170] The display processing unit 274 generates a user output image 1002a including an animation (moving image) of user A's character object 1001a based on the extracted motion data and the character object and used object associated with user A's user ID in the user table T1. Then, the display processing unit 274 generates a user output image 1002a including the animation (moving image) of the generated character object 1001a of user A. In this way, the display processing unit 274 of user B's terminal device 2 automatically generates a changed character object 1011a of user A based on the positional relationship between the displayed character object of user A and the character object of user B and the motion data stored in the terminal storage device 22.

[0171] Furthermore, the display processing unit 274 of user B's terminal device 2 generates a user output image 1002b1 including an animation (moving image) of object 1001a1, which is a newly appeared part of user A's character objects, and an animation (moving image) of user B1's changed character object 1001b1. The animation (moving image) of object 1001a1, which is a newly appeared part of user A's character objects, is generated by the display processing unit 274 based on the extracted motion data and the character object and used object associated with user A's user ID in the user table T1. The animation (moving image) of user B1's changed character object 1001b1 is generated by the display processing unit 274 based on the extracted motion data and the character object and used object associated with user B1's user ID in the user table T1.

[0172] Then, the display processing unit 274 of user B's terminal device 2 displays a communication screen 1000 (FIG. 10(a)) including the generated user output image 1002a and user output image 1002b1. Note that user output images 812b2 and 812b3 including character objects 811b2 and 811b3 of users B2 and B3, respectively, who are participating in the communication service, are the same as those in FIG. 8(b). This completes the change processing.

[0173] 9(b) and 10(a), the display positions on the screen of character object 911a and character object 911b1 displayed on user A's terminal device 2 are different from the display positions on the screen of character object 1001a and character object 1001b1 displayed on user B's terminal device 2. Therefore, even though the same change instruction is issued, the changes in user output image 912a and user output image 912b1 and the changes in user output image 1002a and user output image 1002b1 are different from each other.

[0174] As shown in FIG. 9(b), on the terminal device 2 of user A, a user output image 912a of user A and a user output image 912b1 of user B1 are displayed from left to right on the screen. On the other hand, as shown in FIG. 10(a), on the terminal device 2 of user B1, a user output image 1002b1 of user B1 and a user output image 1002a of user A are displayed from left to right on the screen. Therefore, if the change instruction by user A is an instruction for the character object of user A to hit the character object of user B1, the direction in which the arm of user A's character object extends is to the right on the screen on the terminal device 2 of user A and to the left on the screen on the terminal device of user B1. Note that on the terminal device of user A and the terminal device of user B1, the user output image of user A and the user output image of user B may be displayed side by side in the same order. That is, the same screen including the user output image of user A and the user output image of user B may be displayed on the terminal device 2 of user A and the terminal device 2 of user B1.

[0175] As described with reference to Figures 8(b) to 10(a), in response to a change instruction from user A, the character object of user A changes and the user output image of user B changes on the terminal devices 2 of users A, B1, B2, and B3 participating in the communication service. This prevents users A and B from getting bored during communication via the character objects and makes it possible to increase user A's motivation to continue communicating with user B.

[0176] The communication screen displayed on user A's terminal device 2, in which user A's character object has changed and user B's user output image has changed, is not limited to communication screen 910 in FIG. 9(b). For example, as shown in FIG. 10(b), the display processing unit 274 may exclude object 1011a1, which is a part of user A's character object, from being included in user B1's user output image 912b1. In this case, in the above-described change processing, processing is executed in which the arm of user A's character object 1011a extends beyond the display area of ​​user A's user output image 1012a in FIG. 10(b), as in the communication screen 1010 shown in FIG. 10(b). Therefore, the arm of user A's character object 1011a is displayed so as to be superimposed on user B's user output image 912b1. In this case as well, user A's terminal device 2 transmits information regarding the change instruction, the user IDs and action IDs of user A and user B1, along with destination information, to the server device 3, and the server device 3 transmits information regarding the change instruction, the user IDs and action IDs of user A and user B1 to the terminal devices 2 of user B (users B1, B2, and B3). As a result, each of user B's terminal devices 2 also executes change processing based on the received information regarding the change instruction, the user IDs and action IDs of user A and user B1, and a screen similar to communication screen 1010 is displayed.

[0177] 11(a), the display processing unit 274 may display a communication screen 1100 in which user A's character object 1101a appears in user B's user output image 1102b1 in response to a change instruction from user A. In this case, character object 811a that was included in user A's user output image 812a in FIG. 8(b) is not included in user A's user output image 1102a on the communication screen 1100. An example of change processing in this case will be described below.

[0178] First, the display processing unit 274 of the terminal device 2 of user A generates a user output image 1102a that does not include user A's character object 1101a in response to a change instruction by user A input from the input device 24. Next, the display processing unit 274 acquires the change instruction input from the input device 24, the user ID of user B1, and an action ID. Next, the display processing unit 274 identifies the relative position on the display screen of user B1's character object 811b1 with respect to user A's character object 811a on the communication screen 810. Next, the display processing unit 274 extracts, from the terminal storage device 22, the motion data and the motioned data associated with the identified relative position from the motion data and the motioned data associated with the action ID. Next, the display processing unit 274 generates an animation (moving image) of user A's character object 1101a based on the extracted motion data and the character object and used object associated with user A's user ID in the user table T1. Furthermore, the display processing unit 274 generates an animation (moving image) of user B's character object 1101b1 based on the extracted motion-targeted data and the character object and used object associated with user B1's user ID in user table T1. Next, the display processing unit 274 generates a user output image 1102b1 including an animation (moving image) of user A's newly appeared character object 1101a and an animation (moving image) of user B1's changed character object 1101b1. Then, the display processing unit 274 displays the communication screen 1100 including the generated user output image 1102a and user output image 1102b1, and the change processing ends.

[0179] In this case as well, user A's terminal device 2 transmits information regarding the change instruction, the user IDs and action IDs of user A and user B1, along with destination information, to server device 3, and server device 3 transmits information regarding the change instruction, the user IDs and action IDs of user A and user B1 to user B (users B1, B2, and B3). As a result, each terminal device of user B also executes change processing based on the received information regarding the change instruction, the user IDs and action IDs of user A and user B1, and a screen similar to communication screen 1100 is displayed.

[0180] 11(a), the display area of ​​user A's user output image 902a may be reduced or may disappear. A communication screen 1110 shown in FIG. 11(b) is an example in which the display area of ​​user A's user output image 1102a has been eliminated. In the example shown in FIG. 11(b), user B's user output image 1112b1, in which user A's character object 1111a has appeared, is displayed so as to include the display area of ​​the eliminated user output image 1102a. Furthermore, in the case in which the display area of ​​user A's user output image 1102a is reduced on the communication screen 1100 shown in FIG. 11(a), user B's user output image 1112b1 is displayed so as to include the area that was previously the display area of ​​user output image 1102a.

[0181] Furthermore, for example, the display processing unit 274 may display a communication screen 1200 including information 1201 related to the movement of user A's character object within a user output image 812b1 shown in FIG. 8(b). FIG. 12(a) is a diagram showing an example of the communication screen 1200. In this case, at least one of user A's character object 811a and user B's character object 811b1 may be displayed so as not to make any movement. For example, a user output image 812a including a character object 811a making a movement corresponding to a change instruction from user A is displayed, and a user output image 812b1 including information 1201 related to the movement of character object 811a is displayed in accordance with the movement of user A's character object 811a corresponding to the change instruction from user A. Furthermore, in this case, user A's character object 811 may be an image generated by the display processing unit 274 based on user A's motion data and the character object and used object associated with user A's user ID in the user table T1. Similarly, the character object 811b1 of user B1 may be an image generated by the display processing unit 274 based on the motion data contained in the output information transmitted from user B1's terminal device 2 via the server device 3 and the character object and used object associated with user B1's user ID in the user table T1.

[0182] In this case as well, user A's terminal device 2 transmits information regarding the change instruction, the user IDs and action IDs of user A and user B1, along with destination information, to server device 3, and server device 3 transmits information regarding the change instruction, the user IDs and action IDs of user A and user B1 to user B (users B1, B2, and B3). As a result, a screen similar to communication screen 1200 including information 1201 regarding the action indicated by the received action ID is displayed on each of user B's terminal devices.

[0183] Returning to FIG. 9(a), on the selection screen 900, the interface through which user A issues a change instruction is not limited to the change candidate button 901. For example, as shown in FIG. 12(b), user A may perform a predetermined operation on user B1's character object 811b1, and a change instruction corresponding to the predetermined operation may be input to the terminal processing device 27 by the input device 24. As shown in FIG. 12(b), when the input device 24 inputs information to the terminal processing device 27 indicating that the operation of swiping the screen area corresponding to the head of user B1's character object 811b1 left and right has continued a predetermined number of times, the display processing unit 274 determines that a change instruction has been input in which user A's character object 811a strokes the head of user B's character object 811b1. Then, the display processing unit 274 and the transmission unit 272 execute the change processing described with reference to FIGS. 8(b) to 10(a) based on the determined change instruction. As a result, a communication screen 1300 is displayed in which user A's character object 811a strokes the head of user B's character object 811b1, as shown in FIG. 13(a).

[0184] The predetermined operation on user B1's character object 811b1 is not limited to the example shown in Fig. 12(b). For example, an operation of tapping a screen area corresponding to the body of user B1's character object 811b1 may correspond to an action of user A's character object 811a poking user B's character object 811b1. For example, an operation of double-tapping a screen area corresponding to the abdomen of user B1's character object 811b1 may correspond to an action of user A's character object 811a tickling user B's character object 811b1.

[0185] Furthermore, for example, the display processing unit 274 and the transmission unit 272 may determine that a change instruction has been input by user A using continuously acquired imaging data without using the input device 24. First, the display processing unit 274 and the transmission unit 272 continuously monitor the continuously acquired imaging data and perform a determination process to determine whether a change instruction action has been performed by the user. In another example of the selection screen 900 shown in FIG. 13(b), the determination process determines whether user A has blinked a predetermined number of times or more within a predetermined determination period (e.g., 2 seconds). Note that, in the example shown in FIG. 13(b), information 1301 regarding the change instruction action is displayed on the selection screen 900, but the information 1301 regarding the change instruction action does not have to be displayed on the selection screen 900. For example, this determination process may use a trained discrimination model or the like that has learned about human blinking using known machine learning. Note that data to be determined in the determination process is not limited to continuously acquired imaging data. For example, motion data generated by the generation unit 271 may be used in the determination process.

[0186] The change instruction action and the action ID are stored in association with each other in the terminal storage device 22. When the display processing unit 274 and the transmission unit 272 determine that a change instruction action has been performed by a user, they identify the action ID corresponding to the change instruction action and the user ID of user B1 corresponding to the character object displayed on the selection screen 900. Then, the display processing unit 274 and the transmission unit 272 execute the change process using the identified action ID and user ID of user B1, as well as the user ID of user A stored in the terminal storage device 22.

[0187] In this way, during the communication service, the terminal device 2 can change the character object of user A and the user output image of user B based on the captured image data of user A. This prevents user A and user B from getting bored during communication via the character objects, and can increase user A's motivation to continue communicating with user B.

[0188] Furthermore, the interface screen for user A to issue a change instruction is not limited to the selection screen 900. For example, an object having the same function as the change candidate button 901 may be included in a user output image of a user who has already newly joined the communication service.

[0189] Fig. 14(a) is a diagram showing an example of a communication screen 1400 including user output images 812a, 812b1, and 812b2 of users A, B1, and B2 who are already participating in the communication service, respectively. Fig. 14(b) is a diagram showing an example of a communication screen 1410 when user B3 newly joins the communication service in which users A, B1, and B2 are participating.

[0190] 8(b) to which a change candidate button 1411 has been added. When user B3 newly joins the communication service, the display processing unit 274 displays a user output image 812b3 including a character object 811b3 of user B3 and the change candidate button 1411. Note that the display processing unit 274 may terminate the display of the change candidate button 1411 when a predetermined period of time (e.g., three minutes) has elapsed since the change candidate button 1411 was displayed.

[0191] The change candidate button 1411 has the same function as the change candidate button 901. For example, when user A operates the input device 24 to select one of the multiple change candidate buttons 1411, the input device 24 inputs a change instruction to the terminal processing device 27. The input device 24 inputs the user ID of newly participating user B3 and the action ID corresponding to the selected change candidate button 1411, along with the change instruction, to the terminal processing device 27. Then, the display processing unit 274 and the transmission unit 272 execute change processing according to the input change instruction.

[0192] In this way, user A can instantly perform an operation to change the user output image 812b3 of user B3 who has newly joined the communication service, so the information system 1 can prevent user A from getting bored during communication via the character object and can increase user A's motivation to continue communicating with user B. Furthermore, the display of the change candidate button 1411 makes it easy for users who are already participating in the communication service (user A, user B1, and user B2) to make changes to the character object of user B3 who has newly joined the communication service. In this way, the information system 1 can provide user B3 with an opportunity to communicate with users who are already participating, making it easier for him to participate in communication within a communication group that has joined the communication service.

[0193] Furthermore, a selection screen 1500 may be displayed simultaneously on the terminal devices 2 of multiple users participating in the communication service as an interface screen for user A to give change instructions. For example, when user A performs a predetermined operation (such as swiping from the bottom to the top of the screen) on the communication screen 810, the selection screen 1500 including multiple change candidate buttons may be displayed superimposed on the communication screen 810.

[0194] Fig. 15(a) is a diagram showing an example of a selection screen 1500. In the example shown in Fig. 15(a), three change candidate buttons corresponding to three actions are included in the selection screen 1500. The change instruction on the selection screen 1500 does not require selection of a user B (users B1, B2, and B3) who wants to change the user output image.

[0195] For example, when user A performs a predetermined operation on the communication screen 810 by operating the input device 24, the input device 24 inputs a display instruction for the selection screen 1500 to the terminal processing device 27. When the display processing unit 274 of user A's terminal device 2 acquires the display instruction for the selection screen 1500 input from the input device 24, it displays the selection screen 1500 on the display device 23. Furthermore, the transmission unit 272 of user A's terminal device 2 transmits information indicating the display instruction input from the input device 24 to the server device 3 via the terminal communication I / F 21, together with user A's user ID and destination information stored in the terminal storage device 22. Next, the server receiving unit 331 of the server device 3 receives the information indicating the display instruction, user A's user ID, and destination information via the server communication I / F 31. Next, the server transmitting unit 332 of the server device 3 transmits the information indicating the display instruction and user A's user ID to the terminal device 2 of user B (users B1 to B3) via the server communication I / F 31.

[0196] Next, when the receiving unit 273 of user B's terminal device 2 receives information indicating a change instruction and user A's user ID via the terminal communication I / F 21, the display processing unit 274 of user B's terminal device 2 displays the selection screen 1500 on the display device 23, similar to user A's terminal device 2. The display processing unit 274 may extract the name of user A associated with user A's user ID from the user table T1 and include the name of user A in the selection screen 1500. The selection screen 1500 may be displayed on each terminal device 2 (user A, user B1, user B2, and user B3) in response to a predetermined operation by any of users B (user B1, user B2, or user B3), not limited to a predetermined operation by user A.

[0197] In this way, the selection screen 1500 is displayed on each of the terminal devices 2 of user A and users B1 to B3. When user A and users B1 to B3 each operate the input device 24 to select one of the three change candidate buttons included in the selection screen 1500, the display processing unit 274 and the transmission unit 272 execute change processing on each of the terminal devices 2. An example of the change processing executed by user A's terminal device 2 in this case will be described below.

[0198] First, when the display processing unit 274 of user A's terminal device 2 acquires the change instruction input from the input device 24 and the action ID corresponding to the selected change candidate button, it extracts motion data associated with the action ID from the terminal storage device 22. The display processing unit 274 generates a user output image 1512a including an animation (moving image) of user A's character object 1511a based on the extracted motion data and the character object and use object associated with user A's user ID in user table T1.

[0199] Next, when the transmitting unit 272 of the terminal device 2 of user A acquires the change instruction and action ID input from the input device 24, it transmits information indicating the change instruction together with the user ID of user A stored in the terminal storage device 22, the acquired action ID, and the destination information to the server device 3 via the terminal communication I / F 21. Next, the server receiving unit 331 of the server device 3 receives the information indicating the change instruction, the user ID, action ID, and destination information of user A via the server communication I / F 31. Next, the server transmitting unit 332 of the server device 3 transmits the information indicating the change instruction, the user ID and action ID of user A to the terminal device 2 of user B (users B1 to B3) via the server communication I / F 31.

[0200] Next, when the receiving unit 273 of user B's terminal device 2 receives the information indicating the change instruction, user A's user ID, and action ID via the terminal communication I / F 21, the display processing unit 274 of user B's terminal device 2 extracts the motion data associated with the received action ID from the terminal storage device 22. Then, the display processing unit 274 generates a user output image 1512a including an animation (moving image) of user A's character object 1511a based on the extracted motion data and the character object and used object associated with user A's user ID in user table T1.

[0201] When the receiving unit 273 of the terminal device 2 of user A receives, via the server device 3, the information indicating the change instruction, the user ID of user B1, and the action ID of user B1, the display processing unit 274 extracts motion data associated with the action ID from the terminal storage device 22. Then, the display processing unit 274 generates a user output image 1512b1 including an animation (moving image) of user B's character object 1511b1 based on the extracted motion data and the character object and used object associated with user B1's user ID in the user table T1. When the information indicating the change instruction, the user IDs of users B1 and B2, and the action IDs of users B2 and B3 are received from the terminal devices 2 of users B2 and B3, a user output image 1512b2 including an animation (moving image) of user B2's character object 1511b2 and a user output image 1512b3 including an animation (moving image) of user B3's character object 1511b3 are similarly generated.

[0202] Then, the display processing unit 274 of the terminal device 2 of user A displays a communication screen 1510 ( FIG. 15(b) ) including the generated user output image 1512a, user output image 1512b1, user output image 1512b2, and user output image 1512b3. The display processing units 274 of the terminal devices 2 of users B1 to B3 also perform similar display processing. In the example shown in FIG. 15(b), the actions that each user can select are specific actions of showing "rock," "scissors," and "paper," which are in a so-called "sekisukuri" relationship. In this case, for example, a game related to "sekisukuri" (rock-paper-scissors) is realized in which each user (user A, user B1, user B2, and user B3) gives a voice signal to change the character object of each user to perform a specific action (e.g., "rock-paper-scissors"). In each user's terminal device 2, an action of each user's character object showing one of "rock," "scissors," and "paper" is displayed each time a change instruction is given by each user. This allows user A to enjoy a game related to "rock-paper-scissors" (e.g., "rock-paper-scissors") with user B, preventing user A and user B from getting bored during communication via character objects and increasing user A's motivation to continue communicating with user B.

[0203] Note that the terminal device 2 of user A may not display the user output images 1512b1-1512b3 of users B1-B3 corresponding to the change instructions transmitted from the terminal devices 2 of users B1-B3 within a predetermined period (e.g., 10 seconds) after the selection screen 1500 is displayed, and may instead display the user output image 1512a of user A and the user output images 1512b1-1512b3 of users B1-B3 after the predetermined period has elapsed. In this case, the terminal device 2 of user A may acquire the change instruction of user A input from the input device 24, and upon receiving change instructions from the terminal devices 2 of all users B1-B3, may display the user output image 1512a of user A and the user output images 1512b1-1512b3 of users B1-B3 even before the predetermined period has elapsed. Furthermore, after the predetermined period has elapsed, change instructions from each user may not be accepted. This makes it possible to prevent a user who has previously seen the action of another user's character object from deciding on their own action. The server device 3 may determine the result of the "game" based on the action ID transmitted from the terminal device 2 of user A together with the change instruction and the action ID transmitted from the terminal devices 2 of users B1 to B3 together with the change instruction. In this case, the server device 3 may transmit the result of the "game" to the terminal device 2 of each user, and the terminal device 2 of each user may display the result of the "game." Furthermore, the terminal device 2 of each user may display a grouping of users who performed the same action.

[0204] (Operation sequence of information system 1) 16 is a diagram showing an example of an operation sequence of the information system 1. This operation sequence is executed mainly by the terminal processing device 27 and the server processing device 33 in cooperation with the elements of the terminal device 2 and the server device 3, based on control programs stored in advance in the terminal storage device 22 and the server storage device 32. The following description will be given assuming that user A operates terminal device 2a, user B1 operates terminal device 2b1, and user B2 operates terminal device 2b2.

[0205] First, the transmitting unit 272 of the terminal device 2a transmits, together with destination information, output information including character moving image data including motion data of user A generated by the generating unit 271 based on imaging data continuously output by the imaging device 25, voice data of user A output from the microphone 26, and the user ID of user A, to the server device 3 via the terminal communication I / F 21 (step S101). Note that the destination information does not need to be included in the transmission process of the transmitting unit 272. The process of step S101 is executed continuously at predetermined time intervals (for example, every two seconds) until the distribution event ends, and therefore the processes of steps S101 to S110 are executed intermittently.

[0206] Next, the server transmitting unit 332 of the server device 3 refers to the destination information and transmits the output information received from the terminal device 2a to the terminal device 2b1 via the server communication I / F 31 (step S102) and also to the terminal device 2b2 (step S103).

[0207] Furthermore, transmitting unit 272 of terminal device 2b1 transmits output information including character video data of user B1, voice data of user B1, and user ID of user B1 together with destination information to server device 3 via terminal communication I / F 21 (step S104). Next, server transmitting unit 332 of server device 3 refers to the destination information and transmits the output information received from terminal device 2b1 to terminal device 2a via server communication I / F 31 (step S105), and also transmits the output information to terminal device 2b2.

[0208] Furthermore, transmitting unit 272 of terminal device 2b2 transmits output information including character video data of user B2, voice data of user B2, and user ID of user B2 together with destination information to server device 3 via terminal communication I / F 21 (step S106). Next, server transmitting unit 332 of server device 3 refers to the destination information and transmits the output information received from terminal device 2b2 to terminal device 2a via server communication I / F 31 (step S107), and also transmits the output information to terminal device 2b1.

[0209] The display processing unit 274 of the terminal device 2a displays and outputs a communication screen including a user output image including a character object of user A, a user output image including a character object of user B1, and a user output image including a character object of user B2 on the display device 23 of the terminal device 2a based on the output information of each of user A, user B1, and user B2, and outputs the voices of user B1 and user B2 (step S108).

[0210] As in step S108, the display processing unit 274 of terminal device 2b1 displays and outputs a communication screen including a user output image including a character object of user A, a user output image including a character object of user B1, and a user output image including a character object of user B2 based on the output information of each of user A, user B1, and user B2 on the display device 23 of terminal device 2b1, and outputs the voices of user A and user B2 (step S109).

[0211] As in step S108, the display processing unit 274 of terminal device 2b2 displays and outputs a communication screen including a user output image including a character object of user A, a user output image including a character object of user B1, and a user output image including a character object of user B2 on the display device 23 of terminal device 2b2 based on the output information of each of user A, user B1, and user B2, and outputs the voices of user A and user B1 (step S110).

[0212] Next, transmitting unit 272 of terminal device 2a transmits information indicating the change instruction, which has been input in response to user A's operation of input device 24, together with the destination information to server device 3 via terminal communication I / F 21 (step S111). Next, server transmitting unit 332 of server device 3 refers to the destination information and transmits the information indicating the change instruction received from terminal device 2a to terminal device 2b1 (step S112) and also to terminal device 2b2 (step S113) via server communication I / F 31.

[0213] The display processing unit 274 of the terminal device 2a changes the character object of user A in accordance with the change instruction input by user A in response to operation of the input device 24, and changes at least one of the user output image of user B1 and the user output image of user B2, and displays the image on the display device 23 of the terminal device 2a (step S114).

[0214] The display processing unit 274 of the terminal device 2b1 changes the character object of user A in accordance with the information indicating the change instruction transmitted from the terminal device 2a of user A, and changes at least one of the user output image of user B1 and the user output image of user B2, and displays the image on the display device 23 of the terminal device 2b1 (step S115).

[0215] The display processing unit 274 of the terminal device 2b2 changes the character object of user A in accordance with the information indicating the change instruction transmitted from the terminal device 2a of user A, and changes at least one of the user output image of user B1 and the user output image of user B2, and displays the image on the display device 23 of the terminal device 2b2 (step S116).

[0216] As described above in detail, the information system 1 of this embodiment displays a changed character object of user A and a changed user output image of at least one user B in response to an instruction from user A. In this way, the information system 1 of this embodiment allows the output from the terminal device 2 to be changed by user A or at least one user B during a communication service, thereby increasing the users' motivation to continue using the communication service.

[0217] (Variation 1) Note that the present invention is not limited to this embodiment. For example, as motion data corresponding to an action corresponding to a change instruction, among the motion-received data associated with an action ID, multiple types of motion-received data associated with the relative position on the display screen of the character object receiving the action relative to the character object performing the action may be stored. In this case, for example, when the user's terminal device 2 receives a change instruction from another user via the server device 3, a selection screen 1710 is displayed for selecting an action (hereinafter, sometimes referred to as a "reaction") that the user's character object will perform in response to the action of the other user's character object corresponding to the change instruction. This allows the user to select from multiple types of reactions for the character object receiving the action of the other user's character object, thereby improving each user's motivation to continue using the system for communication. Below, the reaction processing will be described with reference to FIG. 17.

[0218] 17(a) is a diagram showing another example of a communication screen 1700 displayed on the display device 23 of the terminal device 2 of user A after a change instruction is given by user B1. In the example shown in Fig. 17(a), first, the transmitting unit 272 of the terminal device 2 of user B1 transmits information about the change instruction corresponding to the change candidate button 901 selected on the selection screen by user B1 operating the input device 24, together with destination information, to the server device 3 via the terminal communication I / F 21. Note that, together with the information about the change instruction, the user ID of user B1, the user ID of user A indicated by the character object (character object receiving the action) displayed on the selection screen, and the action ID corresponding to the selected change candidate button 901 are transmitted.

[0219] Next, the server receiving unit 331 of the server device 3 receives the information indicating the change instruction, the user ID of user B1, the user ID of user A, the action ID, and the destination information via the server communication I / F 31. Next, the server transmitting unit 332 of the server device 3 transmits the information indicating the change instruction, the user ID of user B1, the user ID of user A, and the action ID to the terminal devices 2 of users A, B2, and B3, respectively, via the server communication I / F 31.

[0220] Next, when the receiving unit 273 of user A's terminal device 2 receives information indicating a change instruction, the user ID of user B1, the user ID of user A, and the action ID via the terminal communication I / F 21, the display processing unit 274 of user A's terminal device 2 displays information 1701 indicating that "user B performed the action indicated by the action ID on user A" within the display area of ​​user A's user output image 812a included in the communication screen 1700 (Figure 17(a)).

[0221] When user A operates the input device 24 to select information 1701 or a user output image 812a including the information 1701, the display processing unit 274 of user A's terminal device 2 displays a selection screen 1710. Note that when information 1701 or a user output image 812a including the information 1701 is selected, the display processing unit 274 of user A's terminal device 2 may display a reaction selection button 1711 (described later) on the communication screen 1700 without displaying the selection screen 1710. Furthermore, when the receiving unit 273 of user A's terminal device 2 receives information indicating a change instruction, the user ID of user B1, the user ID of user A, and the action ID, the display processing unit 274 of user A's terminal device 2 may automatically display the selection screen 1710 without displaying the information 1701 or after a predetermined time has elapsed since the information 1701 was displayed.

[0222] 17(b) is a diagram showing an example of a selection screen 1710 displayed on the display device 23 of the terminal device 2 of user A. On the selection screen 1710, a user output image 812a including user A's character object 811a is displayed on the entire screen. Furthermore, the selection screen 1710 includes information 1701 and a reaction selection button 1711. Note that the reaction selection button 1711 does not have to be included on the selection screen 1710 when it is displayed. In this case, when a specific operation by user A on the input device 24 (for example, a so-called "long press" operation within the display area of ​​the selection screen 1710) is detected by the input device 24, the display processing unit 274 may display the reaction selection button 1711.

[0223] Each of the multiple reaction selection buttons 1711 corresponds to each of multiple motion-subjected data associated with the action ID transmitted together with the change instruction by user B1 and associated with the relative position of user B1's character object 811b1 and user A's character object 811a on the communication screen 1700. In other words, each reaction selection button 1711 corresponds to a reaction of user A's character object 811a in response to the action of user B1's character object 811b1 corresponding to the change instruction.

[0224] When user A operates the input device 24 to select one of the multiple reaction selection buttons 1711, the input device 24 inputs information indicating the selected reaction selection button 1711 to the terminal processing device 27. Based on the information indicating the selected reaction selection button 1711, the display processing unit 274 of user A's terminal device 2 extracts the motion-received data corresponding to the selected reaction selection button 1711 from the terminal storage device 22. The display processing unit 274 also extracts the motion data associated with the action ID and relative position transmitted together with the change instruction by user B1.

[0225] The display processing unit 274 generates a user output image 812b1 including an animation (moving image) of the character object 811b1 of user B1 based on the extracted motion data and the character object and use object associated with the user ID of user B1 in the user table T1. The display processing unit 274 also generates a user output image 812a including an animation (moving image) of the character object 811a of user A based on the extracted motion-subjected data and the character object and use object associated with the user ID of user A in the user table T1. Note that the user output image 812a may include some objects of the character object 811b1 of user B1 generated based on the extracted motion data and the character object and use object associated with the user ID of user B1 in the user table T1.

[0226] The transmitting unit 272 of the terminal device 2 of user A transmits information indicating the reaction instruction, information indicating the selected reaction selection button 1711, and user A's user ID, along with the destination information, to the server device 3, and the server device 3 transmits the information indicating the reaction instruction, information indicating the selected reaction selection button 1711, and user A's user ID to each of the terminal devices 2 of users B1 to B3. If the destination information is stored in the server device 3, the destination information does not need to be transmitted by the terminal device 2 of user A. As a result, each of the terminal devices 2 of users B1 to B3 also executes reaction processing based on the received information indicating the reaction instruction, information indicating the selected reaction selection button 1711, and user A's user ID, and it becomes possible to display the action of user B1's character object and the reaction of user A's character object, just like on the terminal device 2 of user A.

[0227] In this way, a user can select from multiple reactions for the character object that receives an action by another user's character object, and the information system 1 enables diversification of communication between users. Furthermore, since the information system allows the user to select from multiple reactions rather than reacting based on the user's own actions, it enables the user to intuitively and instantly select and express a reaction, improving usability. In this way, the information system 1 can increase each user's motivation to continue using the system for communication. Note that if the reaction selection button 1711 is not selected within a predetermined period (e.g., 10 seconds) after the selection screen 1710 is displayed, the display of the selection screen 1710 may be terminated. In this case, the reaction of user A's character object may be displayed based on automatically selected motion data.

[0228] 17(b) may be displayed in advance before participating in the communication service, and one of the reaction selection buttons 1711 may be selected by user A. In this case, when the receiving unit 273 of user A's terminal device 2 receives information indicating a change instruction, the user ID of user B1, the user ID of user A, and the action ID, the display processing unit 274 of user A's terminal device 2 displays user A's character object performing the action related to the pre-selected reaction. The transmitting unit 272 of user A's terminal device 2 transmits the information indicating the reaction instruction, information indicating the pre-selected reaction selection button 1711, and user A's user ID, together with destination information, to the server device 3. The server device 3 transmits the information indicating the reaction instruction, information indicating the selected reaction selection button 1711, and user A's user ID received from user A's terminal device 2 to the terminal devices 2 of users B1 to B3, respectively. As a result, each of the terminal devices 2 of users B1 to B3 executes a reaction process based on the information indicating the received reaction instruction, the information indicating the selected reaction selection button 1711, and the user ID of user A, and, similar to user A's terminal device, it becomes possible to display the actions of user B1's character object and the reactions of user A's character object that have been selected in advance.

[0229] (Variation 2) User A's character object may be controlled to automatically perform an action in response to a change instruction from another user, user B1, and then perform an action selected by user A. In this case, the terminal storage device 22 stores a plurality of motion data and a plurality of first motioned data in association with an action ID. Each of the plurality of motion data associated with an action ID is associated with the relative position on the display screen of the character object receiving the action with respect to the character object performing the action indicated by the action ID. Furthermore, each of the plurality of first motioned data associated with an action ID is associated with the relative position on the display screen of the character object receiving the action with respect to the character object performing the action indicated by the action ID. Furthermore, the terminal storage device 22 stores a plurality of second motioned data associated with an action ID for each relative position on the display screen of the character object receiving the action with respect to the character object performing the action indicated by the action ID.

[0230] First, the display processing unit 274 of user A's terminal device 2 identifies the relative position on the display screen of user A's character object 811a relative to the displayed character object 811b1 of user B1. Next, the display processing unit 274 extracts, from the terminal storage device 22, motion data associated with the identified relative position from among multiple pieces of motion data associated with the action ID received together with user B1's change instruction. Next, the display processing unit 274 generates an animation (moving image) of user B1's character object 811b1 based on the extracted motion data and the character object and used object associated with user B1's user ID in user table T1. Then, the display processing unit 274 generates a user output image 812b1 including the generated animation (moving image) of user B1's character object 811b1.

[0231] Next, the display processing unit 274 extracts, from the terminal storage device 22, the first motion-received data associated with the identified relative position from among the multiple first motion-received data associated with the action ID received together with the change instruction from user B1. The display processing unit 274 generates an animation (moving image) of user A's character object 811a based on the extracted first motion-received data and the character object and use object associated with user A's user ID in user table T1. Then, the display processing unit 274 generates a user output image 812a including the generated animation (moving image) of user A's character object 811a. Then, the display processing unit 274 displays and outputs the generated user output image 812a and user output image 812b2.

[0232] Furthermore, the transmitting unit 272 of user A's terminal device 2 transmits information indicating the first reaction instruction, information indicating the first motioned data, and user A's user ID, along with destination information, to server device 3. Server device 3 transmits the information indicating the first reaction instruction, information indicating the first motioned data, and user A's user ID to each of users B1 to B3's terminal devices 2. As a result, each of users B1 to B3's terminal devices 2 can display the action of user B1's character object and a reaction corresponding to the first motioned data of user A's character object, just like user A's terminal device, based on the received information indicating the first reaction instruction, information indicating the first motioned data, and user A's user ID.

[0233] When user A operates the input device 24 to select one of the plurality of reaction selection buttons 1711 during a display period of a reaction corresponding to first motioned data in response to a change instruction from the terminal device 2 of another user, user B1, the input device 24 inputs information indicating the selected reaction selection button 1711 to the terminal processing device 27. Based on the information indicating the selected reaction selection button 1711, the display processing unit 274 of user A's terminal device 2 extracts second motioned data corresponding to the selected reaction selection button 1711 from the plurality of second motioned data associated with the action ID transmitted together with the change instruction from user B1 and associated with the identified relative position. Next, the display processing unit 274 generates a user output image 812a including an animation (moving image) of user A's character object 811a based on the extracted second motioned data and the character object and use object associated with user A's user ID in the user table T1.

[0234] Then, the display processing unit 274 displays a user output image 812a including an animation (moving image) of the character object 811a of the user A based on the second motion data, after displaying a user output image 812a including an animation (moving image) of the character object 811a of the user A based on the first motion data. This makes it possible for the character object of the user A to react corresponding to the first motion data and then react corresponding to the second motion data selected by the user A in accordance with the movement of the character object of the user B1.

[0235] Furthermore, transmitting unit 272 of user A's terminal device 2 transmits information indicating the second reaction instruction, information indicating the selected reaction selection button 1711, and user A's user ID, along with destination information, to server device 3. Server device 3 transmits the information indicating the second reaction instruction, information indicating the selected reaction selection button 1711, and user A's user ID to each of users B1 to B3. As a result, in each of users B1 to B3's terminal devices 2, similar to user A's terminal device, it becomes possible to display the action of user B1's character object and the reaction corresponding to the second motioned data of user A's character object after displaying the reaction corresponding to the first motioned data of user A's character object, based on the received information indicating the first reaction instruction, the second motioned data corresponding to the information indicating the selected reaction selection button 1711, and user A's user ID. If the terminal devices 2 of the users B1 to B3 do not receive a second reaction instruction within the predetermined display period, they may not display a reaction corresponding to the second motion-received data of the character object of the user A.

[0236] In this way, a user can select from multiple reactions for the character object in response to an action by another user's character object while the reaction that is automatically executed first is displayed. In this way, information system 1 makes it possible for the character object to react naturally, performing an action according to the user's selection while immediately responding to the action by the character object of another user. Therefore, information system 1 can increase each user's motivation to continue using the system for communication.

[0237] (Variation 3) When a change instruction is input by user A operating the input device 24, the display processing unit 274 may automatically change the user characters of other users (users B2 and B3) that are different from the character object of user A and the character object of user B1 that is the target of the action. For example, the display processing unit 274 of each user's terminal device 2 extracts predetermined motion data (e.g., motion data indicating a surprised action) from the terminal storage device 22, and automatically moves the character objects of users B2 and B3 according to the extracted predetermined motion data in response to the change instruction from user A. This allows not only the parties involved in the change instruction (users A and B1) but also users other than the parties involved to participate in the change instruction, thereby improving each user's motivation to continue using the communication service.

[0238] (Variation 4) If a predetermined activation condition is satisfied when a change instruction is input by the user, specific motion data associated with the action ID may be applied. Hereinafter, an example will be described with reference to FIG.

[0239] For example, the selection screen 900 shown in FIG. 9 is a screen for selecting an action of user A's character object when user A's character object acts on user B1's character object. For example, the predetermined activation condition includes the following: the number of times user A's character object performs action X on user B1's character object exceeds a first predetermined number within a predetermined period; the number of times user B2's character object performs action X on user B1's character object exceeds a first predetermined number; and the number of times user B3's character object performs action X on user B1's character object exceeds a first predetermined number. Note that action X is an action corresponding to the change instruction input by the current user. The predetermined activation condition also includes the following: the number of times user A's character object performs any action on user B1's character object exceeds a first predetermined number within a predetermined period; the number of times user B2's character object performs any action on user B1's character object exceeds a first predetermined number; and the number of times user B3's character object performs any action on user B1's character object exceeds a first predetermined number. Note that the "action" performed by user A's character object in the activation condition may be any action that user A's character object can perform. Similarly, the "action" performed by user B2's character object may be any action that user B2's character object can perform, and the "action" performed by user B3's character object may be any action that user B3's character object can perform. Furthermore, for example, the predetermined activation condition may include a condition in which the total number of times that user A's character object has performed action X on user B1's character object, the number of times that user B2's character object has performed action X on user B1's character object, and the number of times that user B3's character object has performed action X on user B1's character object, exceeds a second predetermined number within a predetermined period of time.Furthermore, for example, the predetermined activation condition may include a condition in which the total number of times that user A's character object performs any action on user B1's character object, the number of times that user B2's character object performs any action on user B1's character object, and the number of times that user B3's character object performs any action on user B1's character object, within a predetermined period of time, exceeds a second predetermined number of times. Note that the "action" performed by user A's character object in the activation condition may be any action that user A's character object can perform. Similarly, the "action" performed by user B2's character object may be any action that user B2's character object can perform, and the "action" performed by user B3's character object may be any action that user B3's character object can perform. Note that the predetermined period may be within one hour from the start time of the communication service in which user A is currently participating, or the period from 15 seconds before the current time to the current time, etc. Note that the first predetermined number of times and the second predetermined number of times may be the same or different.

[0240] Furthermore, for example, the predetermined activation condition may include the total number of reactions of user B1's character object to user A's character object performing action X, the total number of reactions of user B1's character object to user B2's character object performing action X, and the total number of reactions of user B1's character object to user B3's character object performing action X exceeding a third predetermined number within a predetermined period of time. Furthermore, for example, the predetermined activation condition may include the total number of reactions of user B1's character object to user A's character object performing action X exceeding a third predetermined number within a predetermined period of time, the total number of reactions of user B1's character object to user B2's character object performing action X exceeding a third predetermined number, or the total number of reactions of user B1's character object to user B3's character object performing action X exceeding a third predetermined number.

[0241] If a predetermined activation condition is met when user A inputs a change instruction, specific motion-received data from among the motion-received data associated with the action ID of the action currently selected by user A and associated with the relative position on the display screen of user A's character object 811a relative to user B1's character object 811b1 is used to change user B1's character object. Note that if the predetermined activation condition is not met when user A inputs a change instruction, normal motion-received data from among the motion-received data associated with the action ID of the action currently selected by user A and associated with the relative position on the display screen of user A's character object 811a relative to user B1's character object 811b1 is used to change user B1's character object.

[0242] The movements of a character object displayed using specific motion data may differ from those of a character object displayed using normal motion data in that the movements are larger and / or faster. A character object displayed using specific motion data may perform movements different from those of a character object displayed using normal motion data. A character object displayed using specific motion data may perform the movements of a character object displayed using normal motion data, as well as other movements.

[0243] The predetermined activation condition may be that the total number of times that only the character object of user A, who inputs the current change instruction, has performed the same action X on each of the character objects of users B1 to B3 within a predetermined period of time exceeds a fourth predetermined number. The predetermined activation condition may be that the total number of times that only the character object of user A, who inputs the current change instruction, has performed any action on each of the character objects of users B1 to B3 within a predetermined period of time exceeds a fourth predetermined number. Note that the "action" performed by user A's character object in the activation condition may be any action that user A's character object can perform. The predetermined activation condition is that each user performs a predetermined action on user B1's character object in a predetermined order within a predetermined period of time. For example, this includes a case where user A's character object performs action X on user B1's character object, then user B2's character object performs action X on user B1's character object, and then user B3's character object performs action X on user B1's character object.

[0244] The predetermined activation condition may be that the total number of times that only the character object of user A who inputs the current change instruction performs the same action X only on the character object user of user B1 who is the target of the action by the current change instruction within a predetermined period of time exceeds a fifth predetermined number of times.The predetermined activation condition may be that the total number of times that only the character object of user A who inputs the current change instruction performs any action only on the character object user of user B1 who is the target of the action by the current change instruction within a predetermined period of time exceeds a fifth predetermined number of times.The "action" performed by user A's character object in the activation condition may be any action that user A's character object can perform.

[0245] The predetermined activation condition may be that the number of users who have given a change instruction related to action X to a character object of user B1 exceeds a predetermined number within a predetermined period of time. The predetermined activation condition may also be that all users other than user B1 in the same communication group have given a change instruction related to action X to a character object of user B1 within a predetermined period of time.

[0246] The predetermined activation condition may also be that user B1, whose character object receives the action performed by user A's character object in response to the change instruction from user A, has previously permitted the character object of user B1 to receive the action of another character object.The predetermined activation condition may also be that user A, who inputs the current change instruction, has previously permitted the character object of user B1 to receive the action of another character object.

[0247] The predetermined activation condition may be that the number of other users who have a predetermined relationship with user A who inputs the change instruction this time exceeds a first predetermined number.The predetermined activation condition may be that the number of other users who have a predetermined relationship with user B1, whose character object receives the action of user A's character object in response to the change instruction from user A, exceeds a first predetermined number.

[0248] (Variation 5) The change candidate buttons 901 included in the selection screen 900 displayed on the terminal device 2 of user A may be changed when a predetermined display condition is satisfied. For example, when a condition regarding the relationship between user A and user B1, a character object displayed on the selection screen 900, is satisfied, the content of the action corresponding to the change candidate button 901 and the number of change candidate buttons 901 included on the selection screen 900 are changed. The condition regarding the relationship between user A and user B1 may be whether or not user A and user B1 are mutually following each other, whether or not the total amount of time both users have participated in the same communication service exceeds or does not exceed a predetermined amount of time, and / or whether or not the total amount of time user A has viewed user B1's distributed images and the total amount of time user B1 has viewed user A's distributed images exceeds or does not exceed a predetermined amount of time. Furthermore, for example, when a condition regarding information about user A is satisfied, the content of the action corresponding to the change candidate button 901 and the number of change candidate buttons 901 included on the selection screen 900 may be changed. Conditions for information about user A include, for example, whether the number of followers of user A exceeds or does not exceed a predetermined number, and / or whether the participation time of all participating users of user A exceeds or does not exceed a predetermined time.

[0249] (Variation 6) When a character object is generated based on three-dimensional model data for generating three-dimensional animation, the user output image for each user may be an image of the character object, which is a three-dimensional object (a three-dimensional object) placed in a virtual space defined for each user, viewed from a predetermined viewpoint in the virtual space. For example, the display processing unit 274 places a two-dimensional screen in the line of sight of the predetermined viewpoint, projects the three-dimensional coordinates of various three-dimensional objects placed in the virtual space onto the two-dimensional screen, and displays an image of the two-dimensional screen surface onto which the various objects are projected as the user output image. Note that the line of sight of the predetermined viewpoint is the direction toward the character object, and the two-dimensional screen moves and rotates in accordance with the movement of the predetermined viewpoint and the rotation of the line of sight.

[0250] Furthermore, the display processing unit 274 may control the movement of the predetermined viewpoint and / or the rotation of the line of sight of the predetermined viewpoint in response to a predetermined operation on the communication screen. Furthermore, the display processing unit 274 may include an automatic movement button on the communication screen for automatically moving the predetermined viewpoint. In this case, for example, when the automatic movement button is selected by user A operating the input device 24, the display processing unit 274 may automatically move the predetermined viewpoint in accordance with a predetermined movement rule.

[0251] (Variation 7) When a character object is generated based on three-dimensional model data for generating three-dimensional animation, the user output image of each user is an image viewed within the virtual space from a user viewpoint located at a predetermined position (for example, the eye position of the character object) of the character object, which is a solid object (three-dimensional object) placed within a virtual space common to each user. Note that the user viewpoint may move from the predetermined position of the character object in response to a user operation and / or automatically to a position behind, above, or in front of the character object.

[0252] An example of a method for using a communication service in a virtual space common to each user will be described. For example, as shown in FIG. 18, the display processing unit 274 causes a device object S, such as a virtual tablet PC, to appear in the virtual space within the field of view of a character object C of user A from the user's viewpoint. The various screens described with reference to FIGS. 6 to 15 and 17 are displayed on the display device portion of this device object S. In this case, the user output image of user A displayed on the display device portion of the device object S is a two-dimensional image of the virtual space viewed from a virtual camera provided at a predetermined position on the device object S. The voice data of user A is voice data acquired by the microphone 26 of the terminal device 2.

[0253] The transmission unit 272 of the terminal device 2 transmits the user output image of user A, the voice data of user A, and the user ID of user A to all of the terminal devices 2 of user B via the server device 3. Note that the transmission unit 272 may transmit the position and line of sight direction of the virtual camera in the virtual space, the position and body orientation of user A's character object in the virtual space, and motion data of user A, instead of the user output image of user A. Furthermore, the terminal device 2 of user A receives the user output image of each user B, the voice data of each user B, and the user ID of each user B from the terminal device 2 of each user B via the server device 3. Then, the display processing unit 274 of the terminal device 2 of user A displays the user output image of each user B on the display device portion of the device object S, and outputs the voice data of each user B from the terminal device 2. This makes it possible to realize a communication service using the virtual device object S.

[0254] (Variation 8) At least some of the functions implemented by the terminal processing device 27 of the terminal device 2 may be implemented by a processor other than the terminal processing device 27 of the terminal device 2. At least some of the functions implemented by the terminal processing device 27 may be implemented, for example, by the server processing device 33 of the server device 3. Specifically, some of the functions of the generation unit 271 and the display processing unit 274 may be executed in the server device 3. For example, the terminal device 2 transmits continuously acquired imaging data and audio data to the server device 3 along with the user ID of the user operating the terminal device 2. Then, the server device 3 generates character video data using the functions of the generation unit 271 and the display processing unit 274, and generates display information for displaying a communication screen including user output images of all users on the display device 23. Then, the server device 3 transmits the display information together with audio information (audio data) to each of the terminal devices 2 of all users. Each terminal device 2 outputs the information received from the server device 3 (so-called "server rendering").

[0255] (Variation 9) The terminal device 2 has the function of displaying communication screens and the like by executing various commands contained in a control program stored in the terminal device 2. To provide a communication service, the terminal device 2 may also invoke a browser function incorporated in a web application, use this browser function to receive a web page (a document written in a markup language such as HTML (HyperText Markup Language)) from a server device, and execute the control program incorporated in the web page (so-called "browser rendering"). For example, by using HTML 5 as the markup language, the terminal device 2 can easily execute new information processing. By employing such a web application to provide a communication service on the terminal device, a program creator can provide new information processing to the client side (terminal device side) simply by incorporating a new program into a web page transmitted by the server device, thereby significantly reducing the man-hours required to develop a new program. Furthermore, the client side can receive a new service simply by receiving a web page without downloading a new control program. This reduces the load on the communication network, communication costs, and / or communication time compared to downloading a control program, and simplifies the user interface.

[0256] (Variation 10) The generation unit 271 of user A's terminal device 2 may generate output information including facial motion data without using imaging data. For example, facial motion data corresponding to voice data may be stored in the terminal storage device 22 of the terminal device 2, and the generation unit 271 may extract facial motion data corresponding to the acquired voice data of user A from the terminal storage device 22 and generate output information including the extracted facial motion data, the voice data, and user A's user ID. Alternatively, the generation unit 271 of user A's terminal device 2 may generate output information including user A's voice data output from the microphone 26 and the user ID stored in the terminal storage device 22, without including facial motion data. In this case, the display processing unit 274 of user B's terminal device 2 may extract facial motion data corresponding to user A's voice data included in the output information of user A received via the server device 3, and generate animation of user A's character object. The process of generating facial motion data corresponding to user A's voice data may be a process using a known lip synchronization algorithm.

[0257] In this way, the terminal device 2 of user A, which uses an HMD as the display device 23, can communicate with user B via a character object without acquiring image data of user A.

[0258] (Variation 11) A record button for instructing screen capture may be displayed on communication screen 810 (FIG. 8(b)). For example, when user A operates input device 24 to select the record button while communication screen 810 is displayed on user A's terminal device 2, display processing unit 274 displays one composite output image including character object 811a, character object 811b1, character object 811b2, and character object 811b3. Each character object in the composite output image may be controlled to automatically strike a predetermined pose. Then, transmission unit 272 of user A's terminal device 2 transmits the composite output image to server device 3 for transmission to each of users B1 to B3's terminal devices 2, and server device 3 transmits the composite output image received from user A's terminal device 2 to each of users B1 to B3's terminal devices 2. This allows the composite output image to be viewed on each of users B1 to B3's terminal devices 2.

[0259] The composite output image displayed on user A's terminal device 2 may be, for example, a composite output image displayed across the entire screen. In this case, in response to user A's selection of the record button, the boundaries between the user output images may disappear, and each posed character object may appear. Alternatively, the composite output image displayed on user A's terminal device 2 may be displayed within the display area of ​​the user output image 812a of user A who selected the record button. In this case, the display processing unit 274 may cause each character object of users B1 to B3 to move in a direction toward the display area of ​​user A's user output image 812a. In this case, the composite output image displayed within the display area of ​​user A's user output image 812a may be displayed full-screen after a predetermined time has elapsed.

[0260] Furthermore, the display of the composite output image does not have to be performed in response to the selection of the record button, but may be performed automatically at a predetermined timing (e.g., 10 minutes after the start of communication). Furthermore, the display of the composite output image may be performed when a predetermined composition condition is satisfied. The predetermined composition condition may be, for example, when the number of people participating in the same communication service reaches a predetermined number, when the number of change instructions by a specific user or the total number of change instructions by all users participating in the communication service reaches a predetermined number, etc.

[0261] The displayed composite output image may be assigned to user A (stored in the terminal storage device 22 of user A's terminal device 2) in response to an operation by user A. Also, the composite output image received by the terminal device 2 of user B (users B1 to B3) may be assigned to user B (stored in the terminal storage device 22 of user B's terminal device 2) in response to an operation by user B. In this way, the information system 1 can assign a composite output image including the character object of each user who has participated in the communication service to each user, which allows each user to upload the composite output image to a linked SNS or the like. In this way, the information system 1 can increase users' motivation to use the communication service. [Explanation of symbols]

[0262] 1. Information Systems 2(2a, 2b1, 2b2) Terminal equipment 21 Terminal communication I / F 22 Terminal storage device 23 Display device 24 Input Devices 25 Imaging device 26. Mike 27 Terminal Processing Device 271 Generation part 272 Transmitter 273 Receiving Unit 274 Display processing section 275 Audio Output Unit 3. Server equipment 31 Server communication I / F 32 Server storage device 33 Server processing unit 331 Server Receiving Unit 332 Server Distribution Department 4 base station 5. Mobile communication networks 6 Gateway 7. Internet

Claims

1. A control program for a terminal device of a first user, In the terminal device of the first user, displaying, in a first display area, a first image including a first object representing the first user, the first image changing in response to first input information being a movement or a voice of the first user; receiving second input information, which is a movement or a voice of a second user different from the first user, on a terminal device of the second user, and displaying a second image in a second display area, the second image including a second object representing the second user, which changes in accordance with the second input information; displaying a third image including the first object and the second object in a third display area in response to a first instruction from the first user; A control program for executing the above.

2. The control program described in Claim 1, characterized in that the third image is a video.

3. A control program as described in claim 1, characterized in that the first image does not include the second object, and the second image does not include the first object.

4. 4. The control program according to claim 1, wherein the third display area includes the first display area or the second display area.

5. A control program according to any one of claims 1 to 4, wherein the third image displayed in the third display area is an image of the first object and the second object arranged in a virtual space projected from a predetermined viewpoint.

6. The control program according to claim 5, further comprising causing the terminal device of the first user to control the movement of the specified viewpoint and / or the rotation of the line of sight direction of the specified viewpoint in accordance with a second instruction by the first user.

7. The control program according to claim 6 , wherein the predetermined viewpoint is a viewpoint that can be automatically moved.

8. In displaying the third image, the terminal device of the first user displaying the first object performing a first action corresponding to the first input information; displaying the second object performing a second action related to the first action in response to the first action of the first object; The control program according to any one of claims 1 to 7, wherein the control program executes the following.

9. A control program for a server device capable of communicating with a terminal device of a first user and a terminal device of a second user different from the first user, The server device receiving, from a terminal device of a first user, information relating to display of a first image including a first object representing the first user, which changes in response to first input information being a movement or voice of the first user; receiving, from a terminal device of a second user different from the first user, second input information by the second user to the terminal device of the second user; and receiving, from the terminal device of the second user, information relating to display of a second image including a second object representing the second user, which changes in response to the second input information; transmitting information for displaying the first image in a first display area to at least the terminal device of the second user, and transmitting information for displaying the second image in a second display area to at least the terminal device of the first user; transmitting, in response to a first instruction by the first user or a first instruction by the second user, information regarding display of a third image including the first object and the second object to a terminal device of the second user or a terminal device of the first user; A control program characterized by causing the program to execute the above.

10. A terminal device of a first user, displaying, in a first display area, a first image including a first object representing the first user, the first image changing in response to first input information being a movement or a voice of the first user; receiving second input information, which is a movement or a voice of a second user different from the first user, on a terminal device of the second user, and displaying a second image in a second display area, the second image including a second object representing the second user, which changes in accordance with the second input information; a processing unit that displays a third image including the first object and the second object in a third display area in response to a first instruction by the first user. A terminal device comprising:

11. A server device capable of communicating with a terminal device of a first user and a terminal device of a second user different from the first user, receiving, from a terminal device of a first user, information relating to display of a first image including a first object representing the first user, which changes in response to first input information being a movement or voice of the first user; receiving, for a terminal device of a second user different from the first user, second input information being a movement or voice of the second user; and receiving, from the terminal device of the second user, information relating to display of a second image including a second object representing the second user, which changes in response to the second input information; transmitting information for displaying the first image in a first display area to at least the terminal device of the second user, and transmitting information for displaying the second image in a second display area to at least the terminal device of the first user; a processing unit that transmits information regarding a display of a third image including the first object and the second object to a terminal device of the second user or a terminal device of the first user in response to a first instruction by the first user or a first instruction by the second user. A server device comprising:

12. A method for controlling a terminal device of a first user, comprising: The terminal device of the first user, displaying, in a first display area, a first image including a first object representing the first user, the first image changing in response to first input information being a movement or a voice of the first user; receiving second input information, which is a movement or a voice of a second user different from the first user, on a terminal device of the second user, and displaying a second image in a second display area, the second image including a second object representing the second user, which changes in accordance with the second input information; displaying a third image including the first object and the second object in a third display area in response to a first instruction from the first user; A control method comprising:

13. A method for controlling a server device capable of communicating with a terminal device of a first user and a terminal device of a second user different from the first user, the method comprising: The server device: receiving, from a terminal device of a first user, information relating to display of a first image including a first object representing the first user, which changes in response to first input information being a movement or voice of the first user; receiving, for a terminal device of a second user different from the first user, second input information being a movement or voice of the second user; and receiving, from the terminal device of the second user, information relating to display of a second image including a second object representing the second user, which changes in response to the second input information; transmitting information for displaying the first image in a first display area to at least the terminal device of the second user, and transmitting information for displaying the second image in a second display area to at least the terminal device of the first user; transmitting, in response to a first instruction by the first user or a first instruction by the second user, information regarding display of a third image including the first object and the second object to a terminal device of the second user or a terminal device of the first user; A control method comprising:

Images