Information processing system, information processing method, and computer program

Abstract

To activate communication between users.SOLUTION: An information processing system comprises one or more computer processors. The one or more computer processors comprise: a reception unit capable of receiving information for generating video, including information related to a movement of a user, information related to sound, and information related to a character object, sent from a user terminal of the user; an execution unit to execute a video chat between multiple users using character objects, on the basis of the information for generating the video received by the reception unit; an identification unit to identify a state of the user terminal; and a control unit to change a display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by the identification unit.SELECTED DRAWING: Figure 25

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a computer program.

Background Art

[0002] There is known an information processing system that generates an animation of a character object based on the movement of an actor and distributes a video including the animation of the character object (see, for example, Patent Document 1).

[0003] Also, technologies for executing video chat with a user via a virtual space have been developed (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technical field related to such video distribution and video chat, activation of communication between participating users is one of the important issues.

[0006] An object of the present disclosure is to provide a technical improvement that solves or alleviates at least a part of the problems of the above-described conventional technologies. One more specific object of the present disclosure is to provide an information processing system, an information processing method, and a computer program for activating communication between users.

Means for Solving the Problems

[0007] The information processing system in this disclosure is an information processing system comprising one or more computer processors, wherein one or more computer processors include a receiving unit capable of receiving information for generating a video, including information about the user's movements, information about voice, and information about character objects transmitted from a user's user terminal; an execution unit that causes a video chat between multiple users using character objects based on the information for generating a video received by the receiving unit; an identification unit that identifies the state of the user terminal; and a change in the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by the identification unit.

[0008] The identification unit can identify the user terminal as being in the first state if the receiving unit has not received any information regarding the user's movements from the user terminal, or if the information regarding the user's movements received by the receiving unit satisfies the first condition.

[0009] The first condition is that the receiving unit continues to receive information about the same movement for a predetermined period of time, or does not receive information about the amount of change in movement, which is transmitted only when the movement changes, for a predetermined period of time.

[0010] When the specific unit determines that the user terminal is in a first state, the control unit can, as a change in the display mode of the character object, attach a first specific object to the character object and / or apply a first specific action to the character object.

[0011] The first specific object is an object that represents the state in which the character object is not looking at the video chat screen, and the first specific action can be an action that represents the state in which the character object is not looking at the video chat screen.

[0012] The identification unit can determine that the user terminal is in the second state if the receiving unit has not received any audio information from the user terminal, or if the audio information received by the receiving unit satisfies the second condition.

[0013] If the specific unit determines that the user terminal is in a second state, the control unit may, as a change in the display mode of the character object, attach a second specific object to the character object and / or apply a second specific action to the character object.

[0014] The second specific object is an object that represents the state in which the character object is not speaking, and the second specific action can be an action that represents the state in which the character object is not speaking.

[0015] The identification unit can determine that the user terminal is in a third state if the receiving unit has received information indicating that a specific application is running or displayed on the user terminal.

[0016] The identification unit can determine that the user terminal is in a third state if the receiving unit has received information indicating that a music playback application is running on the user terminal.

[0017] If the specific unit determines that the user terminal is in a third state, the control unit may, as a change in the display mode of the character object, attach a third specific object to the character object and / or apply a third specific action to the character object.

[0018] A third specific object can be an object that represents the character object listening to music, and a third specific action can be an action that represents the character object listening to music.

[0019] When the volume of other voices other than the user's voice included in the information regarding the voice received by the receiving unit is equal to or greater than a first value, the specifying unit can specify that the user terminal is in a fourth state.

[0020] When the specifying unit specifies that the user terminal is in a fourth state, the control unit can attach a fourth specific object to the character object and / or apply a fourth specific operation to the character object as a change in the display mode of the character object.

[0021] The fourth specific object is an object for representing the state in which the character object feels it difficult to hear the voice of the video chat, and the fourth specific operation can be an operation for representing the state in which the character object feels it difficult to hear the voice of the video chat.

[0022] When the specifying unit specifies that the user terminal is in a fourth state, the control unit can generate a video without including information regarding the voice when the volume of the other voice is equal to or greater than a second value.

[0023] When the receiving unit has not received information regarding the user's movement from the user terminal but has received information regarding the voice, the specifying unit can specify that the user terminal is in a fifth state.

[0024] When the specifying unit specifies that the user terminal is in a fifth state, the control unit can apply a fifth operation to the character object as a change in the display mode of the character object.

[0025] The fifth operation can be an operation for moving the mouth of the character object according to the information regarding the voice.

[0026] The receiving unit further receives the position information of the user terminal transmitted from the user terminal, and when the position information satisfies a predetermined condition, the specifying unit can specify that the user terminal is in a sixth state.

[0027] When the specifying unit specifies that the user terminal is in the sixth state, the control unit can attach a sixth specific object to the character object and / or apply a sixth specific action to the character object as a change in the display mode of the character object according to the moving speed based on the position information.

[0028] The sixth specific object is an object for representing the state in which the character object is moving, and the sixth specific action can be an action for representing the state in which the character object is moving.

[0029] The receiving unit further receives the instruction information transmitted from the user terminal, and the control unit can change the display mode of the character object according to the instruction.

[0030] As a change in the display mode of the character object, the control unit can attach a seventh specific object to the character object and / or apply a seventh specific action to the character object.

[0031] The seventh specific object is an object on which predetermined text is displayed, and the seventh specific action can be an action of moving at least a part of the character object at a predetermined interval.

[0032] When the volume of the user's voice included in the information related to the voice received by the receiving unit satisfies a predetermined condition, the specifying unit can specify that the user terminal is in the eighth state.

[0033] When the specifying unit specifies that the user terminal is in the eighth state, the control unit can further attach an eighth specific object to the character object and / or display the eighth specific object in the video according to the volume of the voice.

[0034] The information processing method in this disclosure is characterized by causing one or more computer processors to perform a receiving step capable of receiving information for generating a video, including information about the user's movements, information about voice, and information about character objects transmitted from a user's user terminal; an execution step that causes a video chat between multiple users using character objects to be performed based on the information for generating a video received in the receiving step; an identification step that identifies the state of the user terminal; and a control step that changes the display mode of the character object corresponding to the user terminal according to the state of the information processing device identified in the identification step.

[0035] The information processing method in this disclosure is characterized by causing one or more computer processors in the information processing device to perform a transmission step that enables the transmission of information for generating a video relating to a user, including information about the user's movements, information about voice, and information about character objects, to a server device; a reception step that enables the transmission of information for generating a video relating to other users, including information about the movements, voice, and character objects of other users, from the server device; an execution step that causes a video chat between multiple users using character objects to be performed based on the information for generating a video relating to a user and the information for generating a video relating to other users; a specification step that identifies the state of the information processing device; and a control step that changes the display mode of the character object corresponding to the user terminal according to the state of the information processing device identified in the specification step.

[0036] The computer program in this disclosure is characterized in that one or more computer processors in the information processing device implement a transmission function capable of sending information for generating a video relating to a user, including information about the user's movements, information about voice, and information about character objects, to a server device; a reception function capable of receiving information for generating a video relating to other users, including information about the movements, voice, and character objects of other users, from the server device; an execution function that causes a video chat between multiple users using character objects based on the information for generating a video relating to a user and the information for generating a video relating to other users; an identification function that identifies the state of the information processing device; and a control function that changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified in the identification function. [Effects of the Invention]

[0037] This disclosure provides technical improvements that solve or mitigate at least some of the problems of the prior art described above. Specifically, this disclosure provides technology that can revitalize communication among users. [Brief explanation of the drawing]

[0038] [Figure 1] This is a system configuration diagram showing an example of the information processing system described in this disclosure. [Figure 2] This is a system configuration diagram showing an example of the information processing system described in this disclosure. [Figure 3] This is a system configuration diagram showing an example of the information processing system described in this disclosure. [Figure 4] This diagram shows an example of the hardware configuration of the server device, distribution user terminal, and viewing user terminal in this disclosure. [Figure 5] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 6]This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 7] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 8] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 9] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 10] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 11] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 12] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 13] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 14] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 15] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 16] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 17] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 18] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 19] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 20] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 21] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 22] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 23] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 24]This diagram shows an example of the functional configuration of the server device in this disclosure. [Figure 25] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 26] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 27] This is a conceptual diagram showing an image of the screen displayed on the user's terminal. [Figure 28] This is a conceptual diagram illustrating the object or action that will be applied to the character object. [Figure 29] This is a conceptual diagram illustrating the object or action that will be applied to the character object. [Figure 30] This is a conceptual diagram illustrating the object or action that will be applied to the character object. [Figure 31] This is a conceptual diagram illustrating the object or action that will be applied to the character object. [Figure 32] This is a conceptual diagram illustrating the object or action that will be applied to the character object. [Figure 33] This flowchart shows an example of the information processing method described in this disclosure. [Figure 34] This is a circuit diagram showing an example of a circuit configuration for implementing the computer program described in this disclosure. [Figure 35] This diagram shows an example of the functional configuration of a user terminal in this disclosure. [Figure 36] This flowchart shows an example of the information processing method at the user terminal in this disclosure. [Figure 37] This is a circuit diagram showing an example of a circuit configuration for implementing a computer program executed on a user terminal in this disclosure. [Modes for carrying out the invention]

[0039] First, an overview of the information processing system according to the embodiment of this disclosure will be described with reference to the drawings.

[0040] The information processing system in this disclosure is an information processing system comprising one or more client devices and server devices, and comprising one or more computer processors.

[0041] The video displayed on each client device is described as including, but is not limited to, animations of 3D or 2D character objects generated based on the movements of the streaming user. It may also include animations of character objects generated in response to the streaming user's actions, or it may include an image of the streaming user themselves. Furthermore, these character objects and the streaming user may not be displayed, and only the streaming user's voice may be included.

[0042] Here, "distributing user" means the user who transmits video and / or audio information. For example, a distributing user can be a user who hosts or organizes a solo video stream, a collaborative stream with multiple participants, a video chat or voice chat with multiple participants and / or viewers, or an event (such as a party) in a virtual space with multiple participants and / or viewers—in other words, a user who primarily performs these activities. Therefore, "distributing user" in this disclosure can also be rephrased as "host user," "organizer user," or "event organizer user."

[0043] On the other hand, a viewing user refers to a user who receives information related to video and / or audio. However, a viewing user may also be a user who can react to the above information. For example, a viewing user may be a user who watches video streams or collaborative streams, or a user who participates in and / or watches video chats, voice chats, or events. Therefore, in this disclosure, a viewing user can also be referred to as a guest user, participating user, listener, spectator, or supporter.

[0044] Furthermore, the information processing system in the embodiments of this disclosure can be used to provide the next internet space (metaverse), a digital world in which many people can simultaneously participate in free activities such as interaction, work, and play through character objects (avatars) at a level close to that of real society, enabling social activities that transcend the gap between reality and virtuality.

[0045] In this kind of space, users' avatars can freely roam the world and communicate with each other.

[0046] Furthermore, the system may be configured so that one avatar (character object) among multiple avatars in such a virtual space can distribute video as the character object of the streaming user. In other words, it may be possible to perform one-to-many video distribution in a many-to-many metaverse virtual space.

[0047] In such a space, there can be no particular distinction between broadcasting users and viewing users.

[0048] Furthermore, the space represented in the video may be a virtual space, a real space, or an augmented reality space that combines these two. Also, the video may be something like a karaoke video or a game commentary video that plays at least predetermined video and the voice of the streaming user, or it may be a video in which character objects or the actual image of the streaming user are superimposed on these.

[0049] Furthermore, if the streaming user is present in the real world, a character object generated based on the streaming user's movements may be superimposed onto the actual image of the streaming user. Additionally, animations such as gift objects may be superimposed onto the captured image of the real world.

[0050] <System Configuration> As shown in Figure 1 as an example, the information processing system 1000 relating to this disclosure may include one or more viewing user terminals 1100, and an information processing device (support computer) 1300 located in a video distribution studio or the like, which is connected to these viewing user terminals 1100 via a network 1200.

[0051] Furthermore, the information processing device 1300 may be connected to a predetermined server device via the Internet, and some or all of the processing that would normally be performed by the information processing device 1300 may be performed by the server device. This server device may be the information processing device 2400 shown in Figure 2.

[0052] In this specification, the distribution by such information processing system 1000 shall be referred to as studio distribution.

[0053] In studio broadcasts, the full-body movements of the broadcasting user (actor) will be captured by cameras installed in the studio using markers attached to the broadcasting user, and reflected in the character in real time using known motion capture technology.

[0054] Furthermore, the information processing system 1000 of the present invention can be linked with another information processing system 2000, which is shown as an example in Figure 2. The information processing system 2000 shown in Figure 2 may include a distribution user terminal 2100, one or more viewing user terminals 2200, and an information processing device (server device) 2400 connected to these distribution user terminals 2100 and viewing user terminals 2200 via a network 2300.

[0055] In the above example, the distribution user terminal 2100 can be an information processing terminal such as a smartphone. In this specification, distribution by such information processing system 2000 will be referred to as mobile distribution.

[0056] In mobile streaming, the facial movements of the streaming user will be captured by the camera on the streaming user terminal 2100, and reflected in the character's face in real time using known face tracking technology.

[0057] Furthermore, there is no particular distinction between broadcasters and viewers in mobile streaming; viewers can broadcast from their mobile devices at any time, and broadcasters can become viewers when watching videos from other broadcasters.

[0058] The videos generated by the above-mentioned information processing systems 1000 and 2000 can, for example, be distributed to viewers from a video distribution platform.

[0059] Furthermore, in any distribution, the processes of reflecting motion onto the character to generate animation and the processes of displaying gifts (described later) may be shared among the distribution user terminal, the viewing user terminal, the information processing device, and other devices.

[0060] In other words, "distribution" as used here refers to transmitting information that enables viewing of a video on the viewing user's terminal, and video rendering may be performed on the information processing devices 1300 and 2400, or on the distribution user terminal 2100 and viewing user terminals 1100 and 2200.

[0061] Specifically, the terminal or device that generates (renders) the character object's animation receives face motion data and voice data from the distribution user's terminal or information processing device. In addition to face motion, body motion may also be transmitted.

[0062] In this disclosure, the process of generating animations is described as being performed on both the distribution user terminal and the viewing user terminal, but it is not limited to this.

[0063] Furthermore, the information processing system in this disclosure is applicable to either of the examples shown in Figures 1 and 2. Therefore, the information processing system 3000 in the embodiment of this disclosure will be described as comprising a distribution user terminal 100, a viewing user terminal 200, and a server device 400 that can be connected to these distribution user terminals 100 and viewing user terminals 200 via a network 300, as shown in Figure 3.

[0064] The distribution user terminal 100 and the viewing user terminal 200 are interconnected with the server device 400 via, for example, a base station, a mobile communication network, a gateway, and the internet. Communication between the distribution user terminal 100 and the viewing user terminal 200 and the server device 400 is conducted based on a communication protocol such as the Hypertext Transfer Protocol (HTTP). Alternatively, a connection may be initially established between the distribution user terminal 100 and the viewing user terminal 200 and the server device 400 via HTTP communication, and thereafter, communication may be conducted based on WebSocket, which provides bidirectional communication at a lower cost (lower communication load and processing load) than HTTP communication. The communication method between the distribution user terminal 100 and the viewing user terminal 200 and the server device 400 is not limited to the method described above, and any communication technology that can realize this embodiment may be used.

[0065] The distribution user terminal 100 functions as at least the information processing device 1300 or distribution user terminal 2100 described above, the viewing user terminal 200 functions as at least one or more viewing user terminals 1100, 2200 described above, and the server device 400 functions as at least the predetermined server device or information processing device 2400 described above.

[0066] In this disclosure, the distribution user terminal 100 and the viewing user terminal 200 can be smartphones (multifunction telephone terminals), tablet terminals, personal computers, console game machines, head-mounted displays (HMDs), wearable computers such as glasses-type wearable terminals (AR glasses, etc.), and other information processing devices capable of playing videos. Furthermore, these terminals may be standalone devices that operate independently, or they may consist of multiple devices connected to each other in a manner that enables them to send and receive various types of data.

[0067] <Hardware Configuration> Here, the hardware configuration of the distribution user terminal 100 will be described using Figure 4. The distribution user terminal 100 includes a processor 101, memory 102, storage 103, input / output interface (I / F) 104, and communication interface (communication I / F) 105. Each component is interconnected via bus B.

[0068] The distribution user terminal 100 can realize the functions and methods described in this embodiment through the cooperation of the processor 101, memory 102, storage 103, input / output I / F 104, and communication I / F 105.

[0069] The processor 101 executes functions and / or methods realized by code or instructions contained in a program stored in the storage 103. The processor 101 may include, for example, a central processing unit (CPU), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a processor core, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc., and may realize each process disclosed in each embodiment by logic circuits (hardware) or dedicated circuits formed on an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), etc. Furthermore, these circuits may be realized by one or more integrated circuits, and the multiple processes shown in each embodiment may be realized by a single integrated circuit. In addition, LSIs may be referred to as VLSI, Super LSI, Ultra LSI, etc., depending on the degree of integration.

[0070] Memory 102 temporarily stores the program loaded from storage 103 and provides a workspace for the processor 101. Various data generated while the processor 101 is executing the program are also temporarily stored in memory 102. Memory 102 includes, for example, RAM (Random Access Memory) and ROM (Read Only Memory).

[0071] Storage 103 stores programs. Storage 103 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), flash memory, etc.

[0072] The communication interface 105 is implemented as hardware such as a network adapter, communication software, or a combination thereof, and transmits and receives various types of data via the network 300. This communication may be performed via wired or wireless connection, and any communication protocol may be used as long as communication between the devices is possible. The communication interface 105 communicates with other information processing devices via the network 300. The communication interface 105 transmits various types of data to other information processing devices according to instructions from the processor 101. The communication interface 105 also receives various types of data transmitted from other information processing devices and transmits them to the processor 101.

[0073] The input / output interface 104 includes an input device for inputting various operations to the distribution user terminal 100, and an output device for outputting processing results processed by the distribution user terminal 100. The input / output interface 104 may have the input device and output device integrated, or they may be separated into an input device and an output device.

[0074] The input device is implemented by any or a combination of any type of device capable of receiving input from a user and transmitting the information related to said input to the processor 101. The input device includes, for example, hardware keys such as touch panels, touch displays, and keyboards, pointing devices such as mice, cameras (for image-based operation input), and microphones (for voice-based operation input).

[0075] The input device may include a sensor unit. The sensor unit is one or more sensors that detect face motion, which indicates changes in the user's facial expression, and body motion, which indicates changes in the relative position of the user's body to the sensor unit. Face motion includes movements such as blinking and opening and closing the mouth. Known sensor units can be used. An example of a sensor unit includes a ToF sensor that measures and detects the time of flight (Tof) for light directed towards the user to reflect back from the user's face, a camera that photographs the user's face, and an image processing unit that processes the data captured by the camera. The sensor unit may also include an RGB camera that captures visible light and a near-infrared camera that captures near-infrared light. The RGB camera and near-infrared camera can be, for example, the True Depth sensor in "iPhone X" or the Lider sensor in "iPad Pro" or other ToF sensors installed in smartphones. Specifically, this camera projects tens of thousands of invisible dots onto the user's face using a dot projector. Then, by detecting and analyzing the reflected light of the dot pattern to form a depth map of the face and capturing infrared images of the face, accurate facial data is captured. The processing unit of the sensor unit generates various information based on the depth map and infrared images, and compares this information with registered reference data to calculate the depth of each point on the face (the distance between each point and the near-infrared camera) and positional deviations other than depth.

[0076] Furthermore, the sensor unit may have a function to track not only the user's face but also their hands (hand tracking). The sensor unit may also include sensors other than those mentioned above, such as an accelerometer or a gyroscope. The sensor unit may have a spatial mapping function that recognizes objects in the real space where the user is located based on the detection results of the ToF sensor and other known sensors, and maps the recognized objects onto a spatial map. Hereinafter, when face motion detection data and body motion detection data are not specifically distinguished, they will simply be referred to as "tracking data". The image processing unit of the sensor unit may be provided by a control unit that can be equipped with the information processing system of the present invention.

[0077] The control unit, as an input device, can be one that is appropriate for the type of user terminal. Examples of control units include a touch panel integrated with a display, operation buttons provided on the user terminal's casing, a keyboard, a mouse, or a controller operated by the user's hand. The controller may incorporate various known sensors, such as acceleration sensors and inertial measurement units (IMUs) like gyroscopes. Another example of a control unit is a tracking device that identifies the user's hand movements, eye movements, head movements, and gaze direction. In this embodiment, for example, the system can determine the user's instructions based on the user's hand movements and perform various operations such as starting or ending video streaming, evaluating messages and videos, and requesting the display of a predetermined object (e.g., a gift, as described later). If the sensor unit also has input interface functions such as hand tracking, the control unit can be omitted.

[0078] The output device outputs the processing results processed by the processor 101. The output device includes, for example, a touch panel, a speaker, etc.

[0079] Furthermore, the viewing user terminal 200 and server device 400 in this disclosure can also be configured with the same hardware configuration as in Figure 4, unless otherwise noted.

[0080] Next, various functions that can be executed on a user terminal that has launched an application realized by the information processing system in the embodiment of this disclosure, as well as the transitions of the displayed screens, will be described with reference to the drawings.

[0081] Figure 5 shows the top screen T10 that appears on the user's terminal (it is not yet determined whether the user will actually watch or stream) when a video streaming / viewing application is launched.

[0082] As shown in Figure 5, on the top screen T10, a user can watch a video played on a particular streaming channel by selecting one streaming channel (also called a streaming slot, streaming program, or streaming video) from the thumbnail images of one or more recommended streaming channels T12 displayed in the recommended tab T11.

[0083] Alternatively, users can view videos played on a specific streaming channel by accessing a permalink to that channel. Such permalinks can be obtained through notifications from streaming users they follow, or through shared notifications sent by other users.

[0084] Thus, a user who watches a video becomes a viewing user, and the device used to watch the video becomes viewing user device 200.

[0085] Furthermore, as shown in Figure 5, the top screen T10 may also display a notification area T13 for campaigns, events, and other announcements. This notification area T13 can be switched to display other announcements by sliding it.

[0086] From the top screen T10, the following tabs T14, the game tab T15 for displaying game categories, the collaboration waiting tab T16 for displaying streaming channels awaiting collaboration, and the beginner tab T17 for displaying beginner streaming channels are displayed. By selecting one of these (switching tabs), you will transition from the top screen T10 to a different screen.

[0087] Furthermore, the service name display T18 and the search button T19 in the upper frame of the top screen T10 may remain fixed and displayed on the destination screen.

[0088] Similarly, the home button T20, message button T21, streaming preparation button T22, gacha button T23, and profile button T24 in the lower frame of the top screen T10 may remain fixed and displayed on the destination screen.

[0089] In the top screen T10 shown in Figure 5, a user who selects the displayed thumbnail image T12 becomes a viewer who watches the video, as described above, while a user who selects the distribution preparation button T22 may become a distribution user who distributes the video.

[0090] For example, selecting the distribution preparation button T22 on the top screen T10 shown in Figure 5 will transition to the avatar settings screen D10 shown in Figure 6. Then, selecting the distribution button D11 on the avatar settings screen D10 will transition to the distribution settings screen D20 shown in Figure 7. Finally, selecting the distribution start button D25 on the distribution settings screen D20 will transition to the avatar distribution screen D30 shown in Figure 8.

[0091] Next, we will explain the details of the process leading up to the start of video distribution.

[0092] One or more computer processors in this disclosure may include a distribution start request receiving unit, a distribution setting unit, and a distribution start unit.

[0093] The distribution start request receiving unit receives a request from the distribution user's distribution user terminal to start distribution of the first video, which includes the animation of character objects.

[0094] Here, the first video refers to a video that includes the animation of a character object. In this specification, a character object may sometimes be referred to as an "avatar."

[0095] Furthermore, the above distribution start request can be transmitted from the user terminal to the information processing device 400 by selecting the distribution button D11 located on the avatar settings screen D10, etc., which is accessed from the top screen displayed on the user terminal (which will later become the distribution user terminal) that has launched a dedicated application (video distribution / viewing application) for accessing the above video distribution platform.

[0096] Figure 6 shows an example of the avatar settings screen D10. The avatar settings screen D10 can display the character object CO, distribution button D11, gacha button D12, outfit change button D13, photo button D14, etc.

[0097] When the user selects the change of clothes button D13, a closet screen opens where the user can select various avatar parts for the character object CO, such as eyes, nose, mouth, hair, accessories, clothes, and background.

[0098] When the user selects the gacha button D12, a lottery screen for obtaining the above avatar parts will appear.

[0099] When the user selects the photo button D14, a shooting screen for taking a still image of the character object will appear.

[0100] Then, when the user selects the distribution button D11, a distribution start request is sent to the information processing device 400.

[0101] The distribution settings unit, in response to the distribution start request for the first video received by the distribution start request reception unit, sets the distribution settings for the first video based on the specifications from the distribution user terminal 100.

[0102] For example, when the above distribution button D11 is selected, the screen displayed on the distribution user terminal 100 transitions from the avatar settings screen D10 shown in Figure 6 to the distribution settings screen D20 shown in Figure 7.

[0103] The distribution settings may include at least one of the following: settings related to the title of the first video, settings related to whether other users can appear in the first video, settings related to the number of people who can appear in the first video, and settings related to the password.

[0104] These distribution settings can be configured in the title setting field D21, collaboration permission setting field D22, number of participants setting field D23, and password setting field D24 in Figure 7. Additionally, Figure 7 also displays the "Anyone Can Collaborate" setting field D26 and the "SNS Posting Permission" field D27.

[0105] The title of the first video can be freely determined by the broadcaster, within the allowed character limit. If no title is entered by the broadcaster, a pre-set title including the name of the character object (broadcaster), such as "This is a broadcast from XXX♪", may be automatically determined.

[0106] The broadcaster can freely decide whether or not other users can appear in the first video. If other users can appear, they can apply to appear in the broadcaster's video; if other users cannot appear, they cannot. In this specification, the state in which other users appear in the broadcaster's video may be referred to as a "collaboration." Details about collaborations will be described later.

[0107] The number of people who can appear in the first video can only be set if other users are allowed to appear in the first video as mentioned above, and the broadcaster can freely decide the number of people up to the maximum allowed limit.

[0108] The password can be set optionally only if other users are allowed to appear in the first video mentioned above, and the broadcaster can freely choose a number of digits. When other users apply to appear in the first video, they will be required to enter this password. Alternatively, the password setting field D24 may be configured to be active only when the "Anyone Can Collaborate" setting field D26 is OFF.

[0109] The distribution initiation unit distributes information about the first video to the viewer user's terminal 200 based on the conditions set by the distribution setting unit.

[0110] The instruction to start such distribution is sent by selecting the distribution start button D25 shown in Figure 7.

[0111] As an example, the distribution start unit distributes information about a video (the first video) that includes the animation of the distribution user's character object to the viewing user's viewing terminal 200 (avatar distribution).

[0112] The information related to the first video includes, for example, motion information showing the movement of character objects, audio information of the broadcasting user, and gift object information showing gifts sent by other viewers. The gift object information includes at least gift object identification information that identifies the type of gift object, and location information that indicates where the gift object is displayed.

[0113] The distribution starter can then live stream the video via the aforementioned video distribution platform.

[0114] Figure 8 shows the avatar distribution screen D30 displayed on the distribution user terminal 100.

[0115] The avatar streaming screen D30 displays the character object CO, as well as a comment input button D31 for the streaming user to enter comments, a photo button D32 for saving a still image of the screen, a play start button D33 for playing the game described later, an external service link button D34 for watching videos provided by external services, and a gacha button D12 for obtaining avatar parts.

[0116] Additionally, the top of the avatar streaming screen D30 can display the total number of viewers D35, the total number of likes D36, the number of collaborators D37, a share button to external social media D38, a guest details button D39, a ranking display button D40, a settings button D41, and an audio toggle button D42 to switch the audio on / off. An end button D43 to end the stream can also be displayed.

[0117] While a detailed explanation of these displays and buttons is omitted, selecting the settings button D41 allows you to change the distribution settings configured on the distribution settings screen D20.

[0118] Figure 8 shows an example where, in the distribution settings screen D20, other users are allowed to appear in the first video, and the number of people who can appear in the first video is set to 3. Therefore, the character object CO is displayed shifted to the lower left. Up to 3 other users' character objects can appear in the empty space.

[0119] The above is an explanation of the screen transitions when distributing avatars according to the invention in this disclosure.

[0120] Next, we will explain the screen transitions that occur when a streamer plays a game during a live stream.

[0121] One or more computer processors in this disclosure may include a game request receiving unit, a game video distribution unit, and a game display processing unit.

[0122] During avatar streaming, as shown in Figure 8, the streaming user can request to start playing the game by selecting the "Start Play" button D33.

[0123] Furthermore, the game displayed by selecting the play start button D33 is a dedicated game implemented in the application realized by the information processing system in this disclosure, and may be different from a general-purpose game provided by an external service. Therefore, the game distribution in this disclosure may be distinguished from the distribution of gameplay videos of a general-purpose game provided by an external service, along with the commentary of the distribution user.

[0124] Alternatively, the request to start playing may be transmitted from the streaming user terminal 100 to the information processing device 400 by selecting a "start playing" button located on a predetermined screen displayed on the streaming user terminal 100.

[0125] Figure 9 shows an example of screen G10, which is the predetermined screen described above, with the play start button G11 located on it. Screen G10 shown in Figure 9 is the screen that has been accessed by selecting the game tab T15 from the top screen T10 (Figure 5) displayed on the user terminal that has launched the application realized by the information processing system in this disclosure.

[0126] Screen G10 displays at least a play start button G11 that can send a request to start playing a given game.

[0127] Then, when the game request reception unit receives a request to start playing a designated game, the game video distribution unit distributes information about the second video to the viewing user terminal 200.

[0128] Here, the second video is a gameplay video of a specified game. In this specification, the act of distributing such a video so that it is displayed on the screen of the viewing user terminal 200 is referred to as "game distribution".

[0129] Furthermore, after launching the application realized by the invention of this disclosure, the user can send a request to the information processing device 2400 to start distributing a second video, as a distributing user, by selecting a play start object placed on the game list screen and the game details screen.

[0130] These game list screens or game details screens are the first screens, which will be explained in detail below.

[0131] In other words, the game display processing unit performs display processing on a first screen that includes a distribution start object capable of sending a distribution start request, a play start object capable of sending a play start request for a predetermined game, and a thumbnail image of the video currently being streamed for the gameplay of the predetermined game.

[0132] Screen G10 shown in Figure 9 corresponds to the game list screen among the first screens. The first screen, which is the game list screen, is the screen that the user has transitioned to from the top screen T10 by selecting the game tab T15.

[0133] This first screen includes a streaming preparation button T22 as the streaming start object, a play start button G11 as the play start object, and a thumbnail image indicating the video streaming channel.

[0134] The first screen displays, for each of the multiple playable games, a play start button G11, a game icon G12, the game name G13, the total number of viewers for the game's streaming channel G14, and a streaming list G15 that includes thumbnail images of the streaming channels currently broadcasting the game.

[0135] The order of thumbnail images displayed in the streaming list G15 can be customized for each user. For example, the order is prioritized by the number of times the user follows the stream and the number of views by that user, followed by the total number of viewers, and then by the oldest streaming start date. The thumbnail images in this streaming list G15 can be scrolled horizontally to change the displayed area.

[0136] Furthermore, the games displayed on this game list screen will be the top 10 titles according to the following priority order. For example, priority is determined by: games released within the last 48 hours, games played by viewers within the last 30 days (newest first), games with the highest priority period ID, and games with the lowest priority period ID (descending order).

[0137] This distribution list G15 will be updated when you return from another tab's screen and when you perform a refresh operation (pull-to-refresh).

[0138] Figure 10 corresponds to the game details screen of the first screen. The first screen, which is the game details screen, is G20, which is the screen accessed by selecting the game icon G12 or game name G13 displayed on the game list screen shown in Figure 9.

[0139] This first screen includes a streaming start object, the streaming preparation button T22, a play start object, the play start button G21, and a thumbnail image indicating the video's streaming channel.

[0140] The first screen also displays the game icon G22, the game name G23, the total number of viewers for the game's streaming channel G24, and a streaming list G25 that includes thumbnail images of the streaming channels currently broadcasting the game.

[0141] The order of thumbnail images displayed in the streaming list G25 shown here can be customized for each user. For example, the order is prioritized by the number of times the user follows the stream and the number of views by that user, followed by the total number of viewers, and then by the oldest streaming start date. The thumbnail images in this streaming list G25 can be scrolled vertically to change the displayed area.

[0142] This distribution list G25 will be updated when you return from another tab's screen and when you perform a refresh operation (pull-to-refresh).

[0143] As described above, a user who selects a streaming start object (streaming preparation button T22) or a play start object (play start button G11, 21) becomes a streaming user who requests to start streaming or play.

[0144] Additionally, users who select a thumbnail image will become viewers who watch the second video.

[0145] Furthermore, the first screen includes a first area where scrolling is not possible and a second area where scrolling is possible.

[0146] The first screen referred to here is the first screen shown in Figure 10. The first screen includes a first region R1 and a second region R2. Specifically, the first region R1 displays the game title, and the second region R2 displays the aforementioned play start button G21, game icon G22, game name G23, number of viewers G24, and streaming list G25.

[0147] The first region, R1, is a fixed portion of the display screen that cannot be scrolled, while the second region, R2, is a portion that can be scrolled by the user. By scrolling the second region, R2, the user can view thumbnail images that are hidden off-screen.

[0148] However, because scrolling in the second area may cause the play start button G21 to be hidden off-screen, the display processing unit in this disclosure can display the play start object (play start button G21) in the first area R1 depending on the display state of the play start object (play start button G21) displayed in the second area R2.

[0149] For example, in Figure 10, the play start button G21 is displayed in the second area R2, but in Figure 11, it is displayed in the first area R1. In other words, when part or all of the play start button G21 is no longer displayed in the second area R2, the play start button G21 will reappear in the first area.

[0150] Furthermore, the game display processing unit may display the play start object in the first area R1 in stages according to the display status of the play start object displayed in the second area R2.

[0151] This effect can be achieved by changing the transparency of the play-start object in accordance with the scroll amount of the second region, R2.

[0152] As an example, the scroll amount (in pixels) from 0 to 50 can be associated with the button's transparency from 0.0 (completely transparent) to 1.0 (completely opaque). In this case, the object is completely transparent and invisible in the initial display state, and becomes fully visible when scrolled 50 pixels or more. It is preferable to change the object's transparency linearly during that period (0 to 50). Note that the unit of the scroll amount is logical pixels and may differ from the actual pixels on the display.

[0153] Furthermore, the game request reception unit will be able to receive a request from the distribution user terminal 100 to end the game after the distribution of information regarding the second video by the game video distribution unit.

[0154] A request to end the game can be sent by selecting the end button located on the game screen.

[0155] Then, when the game request reception unit receives a request to end the play of a specified game, the distribution start unit can terminate the distribution of information regarding the second video and begin distributing information regarding the first video.

[0156] In other words, what is being streamed here is not just a part of the information from the first video, but all of the information from the first video.

[0157] Furthermore, when the distribution start unit finishes distributing information about the second video and begins distributing information about the first video, the first video will be displayed on the viewing user terminal 200.

[0158] Next, we will explain the steps to begin watching the video.

[0159] One or more processors in this disclosure may further include a viewing reception unit.

[0160] The viewing reception department receives video viewing requests from users.

[0161] The above-mentioned distribution initiation unit distributes video and audio information as video information to the user's information processing terminal in response to a viewing request.

[0162] Figure 12 shows an example of the viewing screen V10 of an avatar video displayed on the viewing user terminal 200.

[0163] Viewers can post comments by entering text in the comment input field V11 and pressing the submit button V12.

[0164] Additionally, by pressing the gift button V13, the viewer will see a list of gifts (screen V30 in Figure 13), and can send a request to display the selected gift.

[0165] In this case, one or more processors in this disclosure may include a determination unit. The determination unit determines whether or not a gift display request has been received from the viewing user terminal 200.

[0166] A display request may include gift object information. Gift object information includes at least gift object identification information that identifies the type of gift object, and location information that indicates where the gift object will be displayed.

[0167] Furthermore, as shown in Figure 13, gifts can be displayed separately by category (free (paid) gifts, accessories, cheering goods, appeal items, variety items, etc.).

[0168] Here, paid gifts are gifts that can be purchased by viewers using their purchased My Coins (coin gifts), while free gifts are gifts that viewers can obtain by using or without using My Points that they have acquired for free (point gifts).

[0169] Furthermore, the term "gift" as used in this application has the same meaning as the term "token." Therefore, it is possible to understand the technology described in this application by replacing the term "gift" with the term "token."

[0170] Additionally, viewers can post positive feedback by pressing the Like button V14. It is also possible to display buttons for posting negative feedback or other emotions in addition to / besides the Like button V14.

[0171] Additionally, if a streamer has enabled the option to allow other users to appear in their stream settings, they can submit a request to appear in the video by selecting the collaboration request button V15.

[0172] Additionally, when a viewer watches a video streamed by a broadcaster they do not yet follow, a follow button (V16) will appear on the screen allowing the viewer to follow the broadcaster. This follow button functions as an unfollow button when the viewer watches a video streamed by a broadcaster they already follow.

[0173] Furthermore, this "following" may occur from one viewer to another, from one broadcaster to another, or from one broadcaster to another. However, following will be managed as a one-way association only, and the reverse association will be managed separately as "followers."

[0174] Additionally, a photo button V25 for saving still images of the screen can also be displayed.

[0175] Additionally, the viewing screen V10 also displays the support ranking display button V17, the share button V18, and the ranking display button V19.

[0176] The support ranking displays a ranking of viewers who are supporting a particular streamer, and the ranking can be calculated based on the amount of gifts (points / coins), etc.

[0177] Furthermore, regarding video sharing, viewers can press the share button V18 to see a list of shareable SNS (Social Networking Services), and by selecting one, they can send a permalink to a specified location on the specified SNS.

[0178] Additionally, by pressing the collaboration request button V15, you can request a collaborative stream from the streaming user. A collaborative stream means that the viewer's character object will appear in the streaming user's video.

[0179] The top of the viewing screen V10 can display the streamer's icon V21, the streamer's name (character object name) V22, the total number of viewers V23, and the total number of likes V24.

[0180] Additionally, selecting the "End Viewing" button V20 will display a screen for ending viewing, allowing you to send a request to end viewing.

[0181] The screen used to end such viewing will be explained in detail. This screen is called "mini-window audio streaming" and is for viewing in a manner that plays only the audio without displaying the video.

[0182] The selection of the above-mentioned End Viewing button V20 is received by the viewing reception unit as a request to end video viewing.

[0183] At this time, the distribution start unit terminates the distribution of video information in response to a request to end viewing, but does not terminate the distribution of audio information.

[0184] Furthermore, when video and audio information is being distributed to the user's terminal, the video is displayed on the main screen of the user's terminal. When only audio information is being distributed, the video is not displayed on the user's terminal, and a secondary screen indicating that video is being watched is displayed.

[0185] Figure 14 shows an image of screen V50 with the sub-screen V51 displayed.

[0186] When this secondary screen V51 is displayed, the main screen shown in the background will revert to the screen before watching the video. For example, if you move from the Recommended tab to the viewing area, it will return to the Recommended tab display, and if you move from the Follow tab to the viewing area, it will revert to the Follow tab.

[0187] When this secondary screen V51 is displayed, you can operate the main screen and navigate to other screens.

[0188] The secondary screen V51 displays the profile picture, name, title, and an audio icon that visually indicates that audio is playing.

[0189] Then, by selecting the end icon V52 displayed on the secondary screen V51, you can completely end the viewing experience.

[0190] Regarding the termination of video display, the server device may continue to transmit information but not display it on the terminal side, or the server device may stop transmitting information altogether.

[0191] With this configuration, users can search for other streams or enjoy chatting with other users while listening to audio only.

[0192] Next, we will explain "collaboration," where other users appear in a video created by a streaming user.

[0193] As described above, viewers can submit a request to participate in the above video by pressing the collaboration request button V15 shown in Figure 12, which will display a confirmation screen for the collaboration streaming participation request.

[0194] The collaborative avatar display unit of one or more computer processors in this disclosure displays a character object in the video that is generated based on the movements of the viewer who made the participation request, in response to the participation request that was received.

[0195] Figure 15 shows, as an example, a viewing or streaming screen when a guest user's character object, Avatar CO4, participates in a video displaying the host user's character object, Avatar CO3. Note that other objects besides avatars are omitted from the display in Figure 15.

[0196] Furthermore, as shown in Figure 16, a third avatar CO1, which is a character object generated based on the movements of another viewer, may also participate in the video. In Figure 16, the third avatar CO1 is positioned behind the first avatars CO3 and CO4 (avatar A2), but the three may be positioned in a single row. The placement of the avatars may also be specified by the streaming user.

[0197] Figure 17 shows the list screen T30 of users with a mutual following relationship, which is displayed when the Follow tab T14 is selected on the top screen shown in Figure 5. Mutual following refers to a relationship where each user is a follower of the other.

[0198] This list screen T30 displays the profile pictures and names of users who have a mutual following relationship.

[0199] As shown in Figure 17, the list screen T30 displays a first object T31 for each user with a mutual following relationship. Alternatively, a chat object T32 may be displayed along with the first object T31. Selecting this chat object allows the user to transition to a separate chat screen with the second user.

[0200] The first object T31 sends a predetermined notification to the terminal of the user associated with the first object T31.

[0201] The prescribed notification may, for example, be a summons.

[0202] Next, we will describe in detail the flow for performing a video chat in the embodiments of this disclosure.

[0203] For example, users can initiate a video chat from an individual chat screen or a group chat screen.

[0204] These chat screens can be accessed, for example, from the chat list screen C10 (Figure 18), which is expanded by selecting the message button T21 on the top screen T10 (Figure 5).

[0205] The chat list screen C10 shown in Figure 18 displays icons of users (character objects) or groups with whom messages (chat) have been sent or received in the past, along with their names or titles. Note that the group icon may include the icons of the users (character objects) who are members of the group.

[0206] Then, the user can start a video chat by selecting a user or group on the chat list screen C10, opening the individual chat screen C20 (Figure 19) or the group chat screen, and selecting the video chat button C21.

[0207] Furthermore, by selecting the edit button C11 on the chat list screen C10, and then selecting the chat creation button C12 or group creation button C13 (Figure 20), it is possible to create a chat screen for a user or group that is not displayed on the chat list screen C10.

[0208] Figure 21 shows the user selection screen C30 that unfolds when the chat creation button C12 is selected. A chat screen is displayed / generated with a user found using the recommended user or the search field C31. The structure of the generated chat screen is the same as the chat screen C20 shown in Figure 19, and a video chat can be started by selecting the video chat button C21.

[0209] Similarly, Figure 22 shows the group creation screen C40, which expands when the group creation button C13 is selected. Users can add other users as group members by selecting the add user button C41. The number of group members that can be added is limited to seven, as an example. The group name can also be set on this screen.

[0210] Once the group is created, the group's chat screen C50 is displayed (Figure 23). Video chat can also be started on the group's chat screen C50 by selecting the video chat button C51.

[0211] Note that the above chat screen C20 can also be accessed from the chat icon T32 on the follower list screen T30 (Figure 17).

[0212] Additionally, chat icons can be placed on other users' profile screens, allowing users to transition to the chat screen from various pages and start video chats.

[0213] When a video chat begins, a notification will be sent to the other party, who can join the chat by responding to this notification. Users can configure whether or not they receive such notifications.

[0214] Alternatively, the system may be configured to allow video chats only with users who are mutually following each other. In this case, an icon indicating that a mutually following user is currently video chatting with another user may be displayed on the follower list screen, and the user may be able to join such an ongoing video chat by selecting this icon.

[0215] In this disclosure, the video chat function can be described as a feature that allows only specific users to view the aforementioned collaborative broadcast. Here, "specific users" refers to users participating in the video chat.

[0216] Next, various functions that can be performed in the information processing system 3000 according to the embodiment of this disclosure will be described with reference to the drawings.

[0217] In this embodiment, the distribution user terminal 100 provided by the information processing system 3000 is described as being the user terminal of a user participating in the video chat, but there is no particular distinction between the distribution user terminal 100 and the viewing user terminal 200 when executing the video chat.

[0218] In other words, the video chat according to the embodiment of this disclosure may be part of a function incorporated into a video distribution system as described above, or it can be implemented as an independent system specifically for video chat using avatars.

[0219] The information processing system 3000 according to the embodiment of this disclosure comprises one or more computer processors, as shown in Figure 24, a receiving unit 410, an execution unit 420, a specific unit 430, and a control unit 440.

[0220] The receiving unit 410 is capable of receiving information for generating a video, including information about the user's movements, audio, and character objects, transmitted from the user's user terminal.

[0221] In the above explanation regarding avatar distribution, it was stated that the video information (information for generating the video) included motion information showing the character object's movements, the broadcasting user's voice information, and gift object information showing gifts sent by other viewers. However, here, it will be assumed that at least the video information includes information about the user's movements, voice information, and character object information.

[0222] Information regarding user movements may include, for example, information regarding the movement of at least the user's face, captured by a camera installed in or connected to the user's terminal.

[0223] Audio information may include, for example, audio corresponding to user speech picked up by a microphone on or connected to the user terminal, and / or information about other audio other than the user's speech.

[0224] Other sounds include, for example, the voices of other users and ambient sounds. Whether a voice is made by the user or another user can be identified using publicly available speech recognition technology. Ambient sounds include the sounds of television, intercoms, telephones, animal noises, station announcements, trains, cars, motorcycles, and the voices of multiple people talking.

[0225] The execution unit 420 executes a video chat between multiple users using character objects, based on the information for generating the video received by the receiving unit 410.

[0226] Figure 25 shows an example of the VC10 video chat screen during a video chat.

[0227] Figure 25 shows an example of four users participating in a video chat using character objects CO1, CO2, CO3, and CO4. The video chat screen VC10 may be divided into display frames according to the number of participants, or it may be configured so that multiple people are displayed together on a single screen.

[0228] In the example shown in Figure 25, the video chat screen VC10 is divided into four display frames. The number, shape, size, etc., of these display frames are not limited to those shown in the figure and may change in real time according to the status of the user's terminal, as described later.

[0229] Furthermore, users participating in a video chat can consist of an initiating user and participating users joining the initiated video chat. In the example shown in Figure 25, the character object CO1 corresponding to the initiating user is displayed in the upper left, but the display location of these users is not limited to that shown in the figure and may change in real time according to the status of the user's terminal, as described later.

[0230] The user terminal of the initiating user will send information to the server device to generate the aforementioned video when the video chat starts. The user terminals of participating users will also send information to the server device to generate the aforementioned video when they respond to the notification of the start of the video chat and join the video chat.

[0231] The user terminal has camera imaging and microphone sound recording capabilities, and the image / audio data captured / recorded by these functions is transmitted to the server device via the network. Whether or not this data is transmitted to the server device can be switched by the user's operation by selecting the video object VC12 and microphone object VC11 displayed on the video chat screen VC10.

[0232] Alternatively, the video object VC12 and microphone object VC11 displayed on the video chat screen VC10 may be selected by the user, thereby allowing these functions on / off on the user's terminal.

[0233] Additionally, the exit object VC13 is used when leaving a video chat.

[0234] In this embodiment, the explanation will be given using the expression that the camera can be switched on / off, and / or the microphone on / off, including both the case of whether or not transmission to the server device described above is possible and the case of turning the function on / off on the user terminal.

[0235] Furthermore, the camera and / or microphone can be automatically switched on / off, not only when the video object VC12 and / or microphone object VC11 are selected by user operation, but also independently of user operation. For example, the system can be configured to automatically switch the camera and / or microphone on / off, independently of user operation, depending on whether the screen displayed on the user terminal is the video chat screen VC10 or another screen.

[0236] Figure 26 shows an example of the display when the microphone is turned off on a user terminal corresponding to character object CO1. The microphone object VC11 changes to object VC14 with a diagonal line through it, and an icon VC15 indicating that the microphone is off is displayed in the position related to character object CO1. Alternatively, an icon VC16 indicating that the microphone is on may be displayed in the positions related to character objects CO2, CO3, and CO4, where the microphone is on.

[0237] Figure 27 shows a typical example of the display when the camera is turned off on the user terminal corresponding to the character object CO1. In this case, the video object VC12 changes to object VC17, which has diagonal lines added to it.

[0238] In typical cases, if the camera is turned off on the user's device, a different image, such as the user's (character object's) profile icon, will be displayed. Furthermore, this different image is usually the same regardless of whether the microphone is on or off.

[0239] However, this type of display makes it difficult to understand the actual state of the user. The actual state of a user can vary greatly, such as whether they are simply watching the video chat with their camera (and microphone) turned off, or whether they have actually left their device. Displaying only a single profile icon as described above may hinder proper conversation and disrupt communication among users participating in the video chat.

[0240] Furthermore, a problem unique to video chats using character objects is that character objects can continue to be displayed even without information about the user's movements. In this case, conversations may not be able to proceed properly, and communication between users participating in the video chat may be hindered.

[0241] Such disruptions to communication among users participating in video chats can reduce users' willingness to participate, and are therefore one of the issues that need to be improved.

[0242] Therefore, the identification unit 430 in this embodiment identifies the state of the user terminal.

[0243] Here, the state of the user terminal includes the state of the user operating the user terminal.

[0244] For example, a user's state can be classified primarily into categories such as whether they can see the video chat screen or not, whether they can hear the audio or not, and whether they can speak or not.

[0245] Such situations include cases where a user is running an application different from the video chat application according to this embodiment on their terminal, and is video chatting while playing a game, playing music, or playing a video.

[0246] Furthermore, the above-mentioned state may also include cases where a video chat is being conducted while other screens within the video chat application according to this embodiment are open. Other screens include, for example, a closet screen for changing the clothing of character objects, a game screen, a menu screen, a streaming video viewing screen, etc.

[0247] Furthermore, the above conditions may also include situations where the user's surroundings are distracting them. Surrounding sounds include other users' voices and ambient noises.

[0248] In embodiments of this disclosure, these states can be estimated by identifying the state of the user terminal.

[0249] The user terminal status will be explained by illustrating the first to eighth statuses in the embodiments described later, but the user terminal status is not limited to these statuses, and the display mode can be changed according to the various statuses that may occur, as described later.

[0250] Furthermore, the control unit 440 in this disclosure changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by the identification unit 430.

[0251] Changing the display mode includes displaying other objects superimposed or combined on character objects, and applying pre-defined specific actions to character objects instead of user motion tracking.

[0252] Changes to the display mode according to the state of the identified user terminal include, for example, changing a character object to a character object equipped with an object that represents a state in which the user cannot see the video chat screen, cannot hear the audio, or cannot speak, and changing a character object to which the user's motion is applied to a character object to which actions that represent a state in which the user cannot see the video chat screen, cannot hear the audio, or cannot speak are applied.

[0253] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0254] The identification unit 430 can identify that the user terminal is in a first state if the receiving unit 410 has not received any information regarding the user's movements from the user terminal, or if the information regarding the user's movements received by the receiving unit 410 satisfies the first condition.

[0255] If the receiving unit 410 has not received information about the user's movements from the user terminal, this includes cases where the video is turned off, or where the video is on but the receiving unit 410 has not received information about the movements due to communication issues or other reasons.

[0256] Furthermore, if the information regarding the user's movements received by the receiving unit 410 satisfies the first condition, one example of this is when the camera on the user terminal is on, and although information regarding the user's movements is received, it is determined that there is no movement.

[0257] For example, the first condition is that the receiving unit 410 continues to receive information about the same movement for a predetermined period of time, or does not receive information about the amount of change in movement, which is transmitted only when the movement changes, for a predetermined period of time.

[0258] "Same movement" refers to movement that is considered to be no movement at all. The degree of such movement may be determined by image analysis or by quantifying the movement. Furthermore, the predetermined time here could be, for example, about 5 minutes, but is not limited to this and may be set by the user.

[0259] Furthermore, if the user terminal transmits the amount of change in movement only when the user's movement changes, the first condition is that information regarding such change has not been received. Also, the predetermined time referred to here can be, for example, about 5 minutes, but is not limited to this and may be set by the user.

[0260] When the identification unit 430 identifies that the user terminal is in a first state, the control unit 440 can change the display mode of the character object by attaching a first specific object to the character object and / or applying a first specific operation to the character object.

[0261] The first specific object could, for example, be an object representing a character object that is not looking at the video chat screen.

[0262] Objects used to indicate that a character object is not looking at the video chat screen include, for example, objects that cover the character object's eyes, such as a mask or sunglasses, as shown in Figure 28.

[0263] These attachment objects can be displayed in association with specific parts of a character object.

[0264] Such a specific part can be a part related to the state of the user terminal (here, the first state), for example, when the character object shows a state of not looking at the video chat screen, it can be a part related to the action of "looking" (such as eyes or face).

[0265] The first specific action is an action for the character object to represent a state of not looking at the video chat screen.

[0266] Examples of actions for the character object to represent a state of not looking at the video chat screen include actions such as hiding the eyes or face with hands.

[0267] According to the above configuration, it is possible to provide a technical improvement that solves or alleviates at least some of the problems of the above-described prior art. Specifically, by clearly displaying the situation of the user during the video chat to other users via the character object, communication misunderstandings can be prevented, and communication between users can be activated.

[0268] Also, according to the above configuration, there is an advantage that an avatar with low load and rich variety can be displayed compared to the case of displaying a character object that moves its eyes according to face tracking.

[0269] Here, in the case of a configuration in which the server device 400 includes the above specific part 430, the server device 400 can generate display change information for changing the display mode of the character object, and transmit the display change information to the user terminals of all users participating in the video chat. And it can be assumed that the display mode of the character object related to one user is changed based on such display change information in the user terminal.

[0270] Alternatively, in a configuration where the user terminal is equipped with the specified unit 430, one user's user terminal can generate display change information to change the display manner of a character object, transmit the display change information to the server device 400, and the server device 400 can transmit the display change information to the user terminals of other users participating in the video chat. Then, one user's user terminal can change the display manner of a character object relating to that user based on the display change information it possesses, and other users' user terminals can change the display manner of a character object relating to that user based on the display change information received from the server device 400.

[0271] The identification unit 430 can determine that the user terminal is in the second state if the receiving unit 410 has not received any voice information from the user terminal, or if the voice information received by the receiving unit 410 satisfies the second condition.

[0272] If the receiving unit 410 has not received voice information from the user terminal, this includes cases where the microphone is turned off, or where the microphone is on but the receiving unit 410 has not received voice information due to communication or other reasons.

[0273] Furthermore, if the audio information received by the receiving unit 410 satisfies the second condition, one example is when the microphone on the user terminal is turned on and audio information is received, but it is determined that the user has not made any sounds for a predetermined period of time.

[0274] The determination of whether or not the user is making a sound may be made through voice analysis or by quantifying the voice. Furthermore, the predetermined time referred to here may be, for example, about 5 minutes, but is not limited to this and may be set by the user.

[0275] If the identification unit 430 determines that the user terminal is in a second state, the control unit 440 may, as a change in the display mode of the character object, attach a second specific object to the character object and / or apply a second specific operation to the character object.

[0276] A second specific object can be an object that represents the state in which a character object is not speaking.

[0277] Objects used to represent a character object not speaking include, for example, objects that cover the character object's mouth, such as a mouth zipper object or mask object, as shown in Figure 29.

[0278] These attachment objects can be displayed in association with specific parts of a character object.

[0279] Such specific parts can be parts related to the state of the user terminal (in this case, the second state), for example, if the character object is not speaking, it can be parts related to the act of "speaking" (for example, the mouth).

[0280] A second specific action can be one that represents the character object not speaking.

[0281] Actions that indicate a character object is not speaking include, for example, covering the mouth with a hand.

[0282] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0283] Furthermore, the above configuration offers the advantage of displaying a variety of avatars with lower overhead compared to displaying character objects whose mouths move in response to face tracking.

[0284] The identification unit 430 can identify the user terminal as being in a third state if the receiving unit 410 has received information indicating that a specific application is running or displayed on the user terminal.

[0285] "A specific application is running on the user's terminal" means that the specific application is running in the background or foreground on the user's terminal, and "a specific application is displayed on the user's terminal" means that the specific application is running in the foreground on the user's terminal.

[0286] Examples of specific applications include music playback apps, video playback apps, game apps, and calling apps.

[0287] In a configuration where the video is automatically switched off upon the launch or display of the specific application mentioned above, the third state may be included in the first state, but in this example, the change in the display mode described later will be performed preferentially or additionally.

[0288] Similarly, in a configuration where the microphone is automatically switched off upon launching or displaying the specific application described above, the third state may be included in the second state, but in this example, the change in the display mode described later will be performed preferentially or additionally.

[0289] When the specific unit 430 determines that the user terminal is in the third state, the control unit 440 can, as a change in the display mode of the character object, attach a third specific object to the character object and / or apply a third specific action to the character object.

[0290] As an example, the third specific object can be at least one of an object for representing a state where the character object is not viewing the video chat screen, an object for representing a state where the character object is not making a sound, and an object for representing a state where the character object is not listening to the video chat audio.

[0291] As described above for the object for representing a state where the character object is not viewing the video chat screen and the object for representing a state where the character object is not making a sound.

[0292] As an example, the object for representing a state where the character object is not listening to the video chat audio includes at least an object that covers the ears of the character object, such as an earphone object as shown in FIG. 30.

[0293] Such a worn object can be displayed in association with a specific part of the character object.

[0294] Such a specific part can be a part related to the state of the user terminal (here, the third state), for example, when indicating a state where the character object is not listening to the video chat audio, it can be a part related to the action of "listening" (such as the ears).

[0295] A third specific action could be, for example, at least one of the following actions: an action to indicate that the character object is not looking at the video chat screen, an action to indicate that the character object is not speaking, and an action to indicate that the character object is not listening to the audio of the video chat.

[0296] The actions to indicate that a character object is not looking at the video chat screen and actions to indicate that a character object is not speaking are as described above.

[0297] Actions that indicate a character object is not hearing the audio of a video chat include, for example, covering their ears with their hands.

[0298] Furthermore, the control unit 440 may change the display mode of character objects depending on the type of specific application launched or displayed on the user terminal.

[0299] For example, if a particular application is a music playback application, the display mode is changed so that the character object appears to be listening to music.

[0300] Specifically, a third object could be attached to a character object, such as an earphone or headphone object, or a musical note object could be displayed near the character object. A third action could be applied to the character object, such as making it move to the rhythm of the music. These applications are not limited to just one; multiple objects or actions can be applied to the character object.

[0301] Alternatively, if a particular application is a video playback application, the display mode can be changed so that the character object appears to be watching a video (such as a movie).

[0302] Specifically, a third object could be a character object to which popcorn or drink objects are attached, or a screen object could be displayed near the character object. A third action could be applied to the character object, such as watching a movie. These applications are not limited to just one; multiple objects or actions can be applied to the character object.

[0303] Alternatively, if a particular application is a game application, the display mode is changed so that the character object appears as if it is playing a game.

[0304] Specifically, a third object could be a controller object attached to a character object, or a game console object or monitor object could be displayed near the character object. A third action could be a game-playing action applied to the character object. These applications are not limited to one; multiple objects or actions can be applied to the character object.

[0305] Alternatively, if a particular application is a calling application, the display behavior can be changed so that the character object appears to be making a call.

[0306] Specifically, a third object could be a telephone receiver object or a smartphone object attached to a character object, or a telephone object could be displayed near a character object. A third action could be applied to the character object, such as making a phone call. These applications are not limited to just one; multiple objects or actions can be applied to a character object.

[0307] With the above configuration, it becomes possible to display the user's status in a more specific and easily understandable way to other users.

[0308] The identification unit 430 can determine that the user terminal is in the fourth state if the volume of other sounds other than the user's voice included in the audio information received by the receiving unit 410 is equal to or greater than the first value.

[0309] Other sounds include, for example, the voices of other users and ambient sounds. Whether a voice is made by the user or another user can be identified using publicly available speech recognition technology. Ambient sounds include the sounds of television, intercoms, telephones, animal noises, station announcements, trains, cars, motorcycles, and the voices of multiple people talking.

[0310] The first value can be equal to or greater than the volume of the user's voice.

[0311] In this case, one or more computer processors in this disclosure may further include a voice determination unit. The voice determination unit determines whether the information regarding the voice received by the receiving unit 410 includes other voices, and whether the volume of any other voices other than those uttered by the user is greater than or equal to a first value. The voice determination unit may also analyze the type of the other voices.

[0312] The fourth state is a state that may be included in the first, second, and third states, but in this example, the changes in the display manner described later may be applied preferentially or additionally.

[0313] If the identification unit 430 determines that the user terminal is in a fourth state, the control unit 440 may, as a change in the display mode of the character object, attach a fourth specific object to the character object and / or apply a fourth specific operation to the character object.

[0314] A fourth specific object can be an object that represents the character object having difficulty hearing the audio in the video chat.

[0315] Objects that can represent a character object having difficulty hearing the audio during a video chat include, for example, objects that cover the character object's ears, such as earplugs.

[0316] A fourth specific action could be one that represents a character object having difficulty hearing the audio in a video chat.

[0317] Actions that indicate a character object is having difficulty hearing the audio during a video chat include, for example, covering their ears with their hands.

[0318] Furthermore, the control unit 440 may change the display mode of the character object according to the type of other sound analyzed by the sound determination unit. This change in display mode may be applied regardless of the volume of the other sound.

[0319] For example, if other sounds are noise from the train station, the display will be changed so that the character object appears to be bothered by the noise, or that it cannot hear the video chat audio.

[0320] Specifically, as a fourth object, a noise object can be attached to a character object, or a noise object can be displayed near a character object. As a fourth action, the character object can be made to cover its ears. These applications are not limited to one; multiple objects or actions can be applied to a character object. Furthermore, the noise object displayed here can be determined according to the type analyzed from other sounds. For example, if the other sound is the sound of a television, the noise object will be a television object, representing the source of the noise.

[0321] Additionally, if another sound is the sound of an intercom, the display will be changed so that the character object appears to be going to retrieve a package.

[0322] Specifically, as a fourth object, you can attach a luggage object to a character object, display a luggage object near a character object, or apply a fourth action to the character object, such as retrieving the luggage. These applications are not limited to just one; multiple objects and actions can be applied to a character object.

[0323] Alternatively, if another sound is the sound of an animal (pet), the display behavior of the character object will be changed to show that it is concerned about or feeding the pet.

[0324] Specifically, the fourth object can be a pet object attached to a character object, or a pet object displayed near a character object. The fourth action can be an action applied to the character object, such as petting or feeding the pet. These applications are not limited to one; multiple objects and actions can be applied to the character object. Furthermore, the pet object and food object displayed here can be determined according to the type of animal analyzed from its vocalizations.

[0325] Alternatively, if another voice is a child's voice, the display mode of the character object is changed to indicate that it is concerned about the child.

[0326] Specifically, the fourth object could be a child object attached to a character object, or a child object displayed near a character object. The fourth action could be an action that soothes a child applied to the character object. These applications are not limited to one; multiple objects or actions can be applied to the character object. Furthermore, the child object displayed here can be determined based on the gender and / or age analyzed from the voice.

[0327] With the above configuration, it becomes possible to display the user's status in a more specific and easily understandable way to other users.

[0328] If the identification unit 430 determines that the user terminal is in the fourth state, the control unit 440 can generate a video without including audio information if the volume of other audio is equal to or greater than the second value.

[0329] The second value is a value greater than the first value mentioned above, and represents a volume level at which the user's voice is drowned out and inaudible. This second value may be changed relatively according to the volume of the user's voice, or it may be a predetermined absolute value.

[0330] At this time, the voice determination unit described above further determines whether the volume of any voice other than the user's voice included in the voice information received by the receiving unit 410 is equal to or greater than the first value or the second value.

[0331] It is also possible to generate a video by omitting only certain audio information, rather than omitting all audio information. This can be achieved by applying known noise cancellation techniques.

[0332] With the above configuration, it is possible to make the user's voice easier to hear during video chat.

[0333] The identification unit 430 can determine that the user terminal is in the fifth state if the receiving unit 410 has not received information about the user's movements from the user terminal but has received information about voice.

[0334] Such a fifth state may be included in the first state, but in this embodiment, the changes in the display mode described later may be applied preferentially or additionally.

[0335] In addition, if the receiving unit 410 has not received information about the user's movements from the user terminal but has received information about audio, this includes, for example, cases where the user is speaking in a video chat with the video off and the microphone on, or cases where the user is not moving in a video chat with both the video and microphone on.

[0336] If the identification unit 430 determines that the user terminal is in the fifth state, the control unit 440 can apply the fifth operation to the character object as a change in the display mode of the character object.

[0337] The fifth action can be to make the character object's mouth move in accordance with information about the sound.

[0338] Since information about user movement includes information about the user's mouth movements, the mouth movements of a character object are usually captured from the user's mouth movements. However, in this embodiment, since information about user movement is not available, the character object's mouth is synchronized with the voice based on information about the user's voice (lip-sync). Known techniques can be applied for such lip-syncing.

[0339] In this case, one or more computer processors in this disclosure may include a voice analysis unit.

[0340] With the above configuration, it becomes possible to properly control the mouth movements of a character object even when information about the user's movements is not available.

[0341] The receiving unit 410 can also receive location information of the user terminal transmitted from the user terminal. At this time, the identification unit 430 identifies the user terminal as being in the sixth state if the location information satisfies predetermined conditions.

[0342] The specified conditions related to location information may be based on the speed of movement calculated based on the location information. Location information meeting the specified conditions may be defined as the speed of movement being equal to or greater than a specified value. The specified value may be, but is not limited to, the speed at which a person is considered to be running.

[0343] Such a state may be included in the first to fifth states described above, but in this example, the change in the display method described later may be given priority.

[0344] When the identification unit 430 determines that the user terminal is in the sixth state, the control unit 440 can, as a change in the display mode of the character object, attach a sixth specific object to the character object and / or apply a sixth specific action to the character object according to the movement speed based on the position information.

[0345] A sixth specific object can be an object that represents the movement of a character object.

[0346] Objects used to represent a character object in motion include, for example, an airplane object, a train object, or a car object, as shown in Figure 31, which indicate that the character object is riding in a vehicle.

[0347] These attachment objects can be displayed in association with specific parts of a character object.

[0348] Such specific parts can be parts related to the state of the user terminal (in this case, the sixth state), for example, if it is showing a character object moving, it can be parts related to the act of "movement" (for example, the feet or waist).

[0349] The sixth specific action can be an action that represents the movement of a character object.

[0350] Actions used to represent the movement of a character object include, for example, running, as shown in Figure 31.

[0351] With the above configuration, it becomes possible to display the user's status in a more specific and easily understandable way to other users.

[0352] The receiving unit 410 can also receive instruction information transmitted from the user terminal. At this time, the control unit 440 shall change the display mode of the character object according to the instruction contained in the instruction information.

[0353] Specifically, the above instruction information may be transmitted by selecting an instruction object that is additionally displayed when the camera function and / or microphone function are turned off by user operation on the video object VC12 and / or microphone object VC11 displayed on the user terminal.

[0354] Such a state may be included in the first to sixth states described above, but in this example, the change in the display method described later may be given priority.

[0355] The control unit 440 can change the display mode of the character object by attaching a seventh specific object to the character object and / or applying a seventh specific operation to the character object.

[0356] The seventh specific object can be an object on which a predetermined text is displayed.

[0357] Objects displaying the specified text include, for example, placard objects and sign objects. Placard objects may display text that indicates the user's status. Examples of user status text include, but are not limited to, "Away from desk," "Playing a game," and "Moving."

[0358] The seventh specific action can be an action that moves at least a portion of a character object at predetermined intervals.

[0359] Actions that move at least a portion of a character object at predetermined intervals include, for example, the character object blinking, nodding, or smiling.

[0360] The attachment of these seventh specific objects and / or the application of the seventh specific action may be selected by the user through the operation of the instruction object described above.

[0361] In this case, the seventh specific object may include all of the first, second, third, fourth, fifth, and sixth objects mentioned above. The user can then select a desired object from among several instruction objects corresponding to each of these objects and attach it to the character object.

[0362] Similarly, the seventh specific action may include all of the first, second, third, fourth, fifth, and sixth specific actions described above. The user can then select the desired action from among multiple instruction objects corresponding to each of these actions and apply it to the character object.

[0363] With the above configuration, it becomes possible to display the user's own situation more concretely and clearly to other users through character objects.

[0364] The identification unit 430 can identify that the user terminal is in the eighth state if the volume of the user's voice included in the audio information received by the receiving unit 410 satisfies predetermined conditions.

[0365] In this case, the voice determination unit described above determines whether the volume of the user's voice included in the voice information received by the receiving unit 410 is outside a predetermined range.

[0366] In this context, "outside the specified volume range" refers to a volume level outside the appropriate range for a user's voice in a video chat. For example, if the user's voice is too loud or too quiet for the video chat audio, this falls outside the appropriate range. Such volume values ​​may be defined by specific numerical values, or they may be determined relatively based on the volume of other users' voices and / or the volume of other sounds other than the user's voice.

[0367] Then, when the identification unit 430 determines that the user terminal is in the eighth state, the control unit 440 can attach the eighth specific object to the character object and / or display the eighth specific object in the video, depending on the volume of the voice output.

[0368] The eighth specific type of object includes, for example, an object to represent the volume of a voice.

[0369] Objects used to represent the volume of a sound include, but are not limited to, microphone objects, megaphone objects (Figure 32), and volume meter objects.

[0370] Furthermore, the size of the microphone object and megaphone object may be displayed such that the size increases as the volume of the voice is louder, and the volume meter object may change according to the volume of the voice.

[0371] These attachment objects may be displayed in association with a specific part of the character object (for example, the mouth), or they may be displayed around the character object.

[0372] The eighth specific action includes, for example, actions to indicate the volume of speech.

[0373] Actions used to indicate the volume of a sound include, specifically, placing a hand over the mouth like a megaphone, or placing an index finger over the mouth as if whispering a secret.

[0374] Furthermore, the volume meter object described above may be displayed on the screen even when the user terminal is not in state eight.

[0375] With the above configuration, the volume level of a user's voice in a video chat can be clearly displayed to other users via a character object.

[0376] Next, the information processing method in the embodiment of this disclosure will be described.

[0377] The information processing method in the embodiments of this disclosure can be executed in an information processing system 3000 comprising one or more user terminals and server devices 400.

[0378] The information processing method in this disclosure, as shown as an example in Figure 33, causes one or more computer processors in the information processing system 3000 to execute a receiving step S410, an execution step S420, a identification step S430, and a control step S440.

[0379] In the receiving step S410, information for generating a video can be received from the user's user terminal, including information about the user's movements, information about voice, and information about character objects. This receiving step S410 can be performed by the receiving unit 410 described above.

[0380] The receiving step S410 may be performed on the server side (server device 400).

[0381] In the execution step S420, a video chat between multiple users using character objects is performed based on the information for generating the video received in the receiving step S410. This execution step S420 can be performed by the execution unit 420 described above.

[0382] Execution step S420 may be performed on the server side (server device 400) or on the client side (user terminal).

[0383] In the specific step S430, the state of the user terminal is identified. This specific step S430 can be performed by the identification unit 430 described above.

[0384] The specific step S430 may be executed on the server side (server device 400) or on the client side (user terminal).

[0385] In control step S440, the display mode of the character object corresponding to the user terminal is changed according to the state of the user terminal identified in identification step S430. This control step S440 can be executed by the control unit 440 described above.

[0386] The control step S440 may be executed on the server side (server device 400) or on the client side (user terminal).

[0387] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0388] Next, a computer program in an embodiment of this disclosure will be described.

[0389] The computer program in the embodiment of this disclosure can be executed in an information processing system 3000 comprising one or more user terminals and server devices 400.

[0390] The computer program in this disclosure enables one or more computer processors of the information processing system 3000 to implement receiving functions, execution functions, identification functions, and control functions.

[0391] The receiving function can receive information for generating a video, including information about the user's movements, audio, and character objects, transmitted from the user's user terminal.

[0392] The execution function, based on the information received by the receiving function for generating video, executes a video chat between multiple users using character objects.

[0393] In certain functions, the status of the user's terminal is identified.

[0394] The control function changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by a specific function.

[0395] The above functions can be realized by the receiving circuit 1410, execution circuit 1420, specific circuit, and control circuit 1430 shown in Figure 34. The receiving circuit 1410, execution circuit 1420, specific circuit 1430, and control circuit 1440 are realized by the receiving unit 410, execution unit 420, specific unit 430, and control unit 440 described above, respectively. Details of each unit are as described above.

[0396] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0397] Next, the information processing device in the embodiment of this disclosure will be described. The information processing device corresponds to the user terminal in the information processing system 3000 described above.

[0398] The information processing device is an information processing device comprising one or more computer processors, the one or more computer processors comprising a transmission unit 110, a reception unit 120, an execution unit 130, a specification unit 140, and a control unit 150, as shown in Figure 35.

[0399] The transmission unit 110 is capable of transmitting information for generating a video related to the user to the server device, including information about the user's movements, information about voice, and information about character objects. The information for generating the video is as described above.

[0400] The receiving unit 120 is capable of receiving information from the server device 400 for generating videos related to other users, including information about the movements of other users, information about voices, and information about character objects.

[0401] The execution unit 130 executes a video chat between multiple users using character objects, based on information for generating a video related to a user and information for generating a video related to other users. The execution unit 130 can have the same configuration as the execution unit 420 described above.

[0402] The identification unit 140 identifies the state of the information processing device. The identification unit 140 can have the same configuration as the identification unit 430 described above.

[0403] The control unit 150 changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by the identification unit 140. The control unit 150 can have the same configuration as the control unit 440 described above.

[0404] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0405] Next, an example of an information processing method in an embodiment of this disclosure will be described. This information processing method is an information processing method that is executed in the information processing device (user terminal) described above.

[0406] The information processing method in this disclosure is characterized by causing one or more computer processors provided in the information processing device to execute a transmission step S110, a reception step S120, an execution step S130, a identification step S140, and a control step S150, as shown as an example in Figure 36.

[0407] In the transmission step S110, information for generating a video related to the user, including information about the user's movements, information about voice, and information about character objects, can be sent to the server device. This transmission step S110 can be performed by the transmission unit 110 described above.

[0408] In the receiving step S120, information for generating a video related to another user can be received from the server device, including information about the other user's movements, information about their voice, and information about their character objects. This receiving step S120 can be performed by the receiving unit 120 described above.

[0409] In execution step S130, a video chat between multiple users using character objects is performed based on information for generating a video related to a user and information for generating videos related to other users. This execution step S130 can be performed by the execution unit 130 described above.

[0410] In the specific step S140, the state of the information processing device is identified. This specific step S140 can be performed by the identification unit 140 described above.

[0411] In control step S150, the display mode of the character object corresponding to the user terminal is changed according to the state of the user terminal identified in a specific step. This control step S160 can be executed by the control unit 150 described above.

[0412] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0413] Finally, the computer program in the embodiment of this disclosure will be described. This computer program is executed in the information processing device (user terminal 100) described above.

[0414] The computer program in this disclosure is characterized by enabling one or more processors in an information processing device to implement transmission, reception, execution, identification, and control functions.

[0415] The transmission function is capable of sending information to a server device for generating user-related videos, including information about the user's movements, audio, and character objects.

[0416] The receiving function is capable of receiving information from the server device for generating videos related to other users, including information about other users' movements, audio, and character objects.

[0417] The execution function uses character objects to perform video chats between multiple users, based on information for generating videos related to the user and information for generating videos related to other users.

[0418] The specific function identifies the state of the information processing device.

[0419] The control function changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified in a specific function.

[0420] The above functions can be realized by the transmitting circuit 1110, receiving circuit 1120, execution circuit 1130, specific circuit 1140, and control circuit 1150 shown in Figure 37. The transmitting circuit 1110, receiving circuit 1120, execution circuit 1130, specific circuit 1140, and control circuit 1150 are realized by the transmitting unit 110, receiving unit 120, execution unit 130, specific unit 140, and control unit 150 described above, respectively. Details of each unit are as described above.

[0421] The above configuration provides a technical improvement that solves or mitigates at least some of the problems of the conventional technology described above. Specifically, by clearly displaying the status of a user during a video chat to other users via a character object, misunderstandings in communication can be prevented, and communication among users can be revitalized.

[0422] Furthermore, a computer or mobile phone or other information processing device can be suitably used to function as a server or terminal device according to the above-described embodiment. Such an information processing device can be realized by storing a program describing the processing content that realizes each function of the server or terminal device according to the embodiment in the storage unit of the information processing device, and having the CPU of the information processing device read and execute the program.

[0423] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0424] Furthermore, the methods described in the embodiments can be distributed by storing the program, which can be executed by a computer, on recording media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, MOs, etc.), and semiconductor memory (ROMs, RAMs, flash memory, etc.), and by transmitting them via communication media. The program stored on the media also includes a configuration program that configures the software means (including not only the executable program but also tables and data structures) to be executed by the computer. The computer implementing this device reads the program recorded on the recording media and, if necessary, constructs the software means using the configuration program, and executes the above-described process by controlling its operation with this software means. The recording media referred to in this specification are not limited to those for distribution, but also include storage media such as magnetic disks and semiconductor memory provided inside the computer or in devices connected via a network. The storage unit may function as, for example, main memory, auxiliary memory, or cache memory. [Additional Notes] The invention described in the claims of the original application immediately prior to its division is listed below. [1] An information processing system comprising one or more computer processors, The one or more computer processors mentioned above are: A receiving unit capable of receiving information for generating a video, including information about the user's movements, audio, and character objects transmitted from the user's user terminal, An execution unit, based on the information for generating the video received by the receiving unit, causes a video chat between multiple users using character objects to be performed. A unit for identifying the status of the user terminal, A control unit that changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by the specified unit, and An information processing system equipped with the following features. [2] The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in a first state when the receiving unit has not received information regarding the user's movements from the user terminal, or when the information regarding the user's movements received by the receiving unit satisfies a first condition. [3] The information processing system according to claim 2, characterized in that the first condition is that the receiving unit continues to receive information about the same movement for a predetermined period of time, or does not receive information about the amount of change in movement that is transmitted only when the movement changes for a predetermined period of time. [4] When the identifying unit identifies the user terminal as being in the first state, The information processing system according to claim 2 or 3, characterized in that the control unit causes the character object to attach a first specific object and / or apply a first specific operation to the character object as a change in the display manner of the character object. [5] The aforementioned first specific object is an object that represents the state in which the character object is not looking at the video chat screen, The information processing system according to claim 4, wherein the first specific action is an action to represent that the character object is not looking at the video chat screen. [6] The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in a second state if the receiving unit has not received the audio information from the user terminal, or if the audio information received by the receiving unit satisfies a second condition. [7] When the identifying unit identifies that the user terminal is in a second state, The information processing system according to claim 6, characterized in that the control unit causes the character object to attach a second specific object and / or apply a second specific operation to the character object as a change in the display manner of the character object. [8] The aforementioned second specific object is an object that represents the state in which the character object is not speaking, The information processing system according to claim 7, characterized in that the second specific action is an action to represent the state in which the character object is not speaking. [9] The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in a third state when the receiving unit has received information indicating that a specific application is running or displayed on the user terminal.

[10] The information processing system according to claim 9, characterized in that the identifying unit identifies the user terminal as being in a third state when the receiving unit has received information indicating that a music playback application is running on the user terminal.

[11] When the identifying unit identifies that the user terminal is in a third state, The information processing system according to claim 10, characterized in that the control unit causes the character object to attach a third specific object to the character object and / or to apply a third specific operation to the character object as a change in the display manner of the character object.

[12] The aforementioned third specific object is an object that represents the character object listening to music, The information processing system according to claim 11, characterized in that the third specific action is an action to represent the character object listening to music.

[13] The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in a fourth state when the volume of other sounds other than the user's voice included in the information about the voice received by the receiving unit is equal to or greater than a first value.

[14] When the identifying unit identifies the user terminal as being in the fourth state, The information processing system according to claim 13, characterized in that the control unit causes the character object to attach a fourth specific object and / or apply a fourth specific operation to the character object as a change in the display manner of the character object.

[15] The fourth specific object is an object that represents the character object having difficulty hearing the audio of the video chat, The information processing system according to claim 14, characterized in that the fourth specific action is an action to represent the character object having difficulty hearing the audio of the video chat.

[16] When the identifying unit identifies the user terminal as being in the fourth state, The information processing system according to claim 13, 14, or 15, characterized in that the control unit generates the video without including information about the audio when the volume of the other audio is equal to or greater than a second value.

[17] The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in a fifth state when the receiving unit has not received information regarding the user's movements from the user terminal but has received information regarding the voice.

[18] When the identifying unit identifies the user terminal as being in the fifth state, The information processing system according to claim 17, characterized in that the control unit causes the character object to apply a fifth operation as a change in the display manner of the character object.

[19] The information processing system according to claim 18, characterized in that the fifth operation is to move the mouth of the character object in accordance with the information regarding the sound.

[20] The receiving unit further receives the location information of the user terminal transmitted from the user terminal, The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in a sixth state when the location information satisfies predetermined conditions. [twenty one] When the identifying unit identifies the user terminal as being in the sixth state, The information processing system according to claim 20, characterized in that the control unit causes the character object to attach a sixth specific object to the character object and / or apply a sixth specific operation to the character object in accordance with the movement speed based on the position information, as a change in the display manner of the character object. [twenty two] The sixth specific object is an object that represents the movement of the character object, The information processing system according to claim 21, characterized in that the sixth specific action is an action for representing the movement of the character object. [twenty three] The receiving unit further receives instruction information transmitted from the user terminal, The information processing system according to claim 1, characterized in that the control unit changes the display mode of the character object in accordance with the instruction. [twenty four] The information processing system according to claim 23, characterized in that the control unit causes the character object to attach a seventh specific object and / or apply a seventh specific operation to the character object as a change in the display manner of the character object. [twenty five] The seventh specific object is an object on which a predetermined text is displayed, The information processing system according to claim 24, characterized in that the seventh specific operation is an operation to move at least a portion of the character object at predetermined intervals.

[26] The information processing system according to claim 1, characterized in that the identifying unit identifies the user terminal as being in the eighth state when the volume of the user's utterance included in the audio information received by the receiving unit satisfies predetermined conditions.

[27] When the identifying unit identifies the user terminal as being in the eighth state, The information processing system according to claim 26, further characterized in that the control unit attaches an eighth specific object to the character object and / or displays the eighth specific object in the video in accordance with the volume of the voice output.

[28] One or more computer processors, A receiving step that can receive information for generating a video, including information about the user's movements, audio information, and character object information transmitted from the user's user terminal, An execution step which, based on the information for generating the video received in the receiving step, causes a video chat between multiple users using character objects to be performed, A selection step to identify the state of the user terminal, A control step which changes the display mode of the character object corresponding to the user terminal according to the state of the user terminal identified in the specified step, and An information processing method that enables execution of [the specified action].

[29] One or more computer processors provided in the information processing device A transmission step that enables the transmission of information for generating a video relating to the user, including information about the user's movements, information about voice, and information about character objects, to a server device, A receiving step in which information for generating a video relating to the other user is received from a server device, including information about the other user's movements, information about their voice, and information about character objects, An execution step of performing a video chat between multiple users using character objects, based on information for generating a video relating to the aforementioned user and information for generating a video relating to the aforementioned other users, A selection step to identify the state of the information processing device, A control step that changes the display mode of a character object corresponding to the information processing device according to the state of the information processing device identified in the aforementioned specific step. An information processing method that enables execution of [the specified action].

[30] One or more computer processors provided in the information processing device A transmission function capable of sending information to a server device for generating a video related to the user, including information about the user's movements, information about voice, and information about character objects, A receiving function that can receive information from a server device for generating a video related to the other user, including information about the other user's movements, information about their voice, and information about their character objects. An execution function that performs a video chat between multiple users using character objects, based on information for generating a video related to the aforementioned user and information for generating a video related to the aforementioned other users, A specific function for identifying the state of the information processing device, A control function that changes the display mode of a character object corresponding to the information processing device according to the state of the information processing device identified in the aforementioned specific function, and A computer program that makes this possible. [Explanation of symbols]

[0425] 100 distribution user terminals (user terminals) 200 Viewing user terminals (user terminals) 300 Networks 400 Server Devices 410 Receiver 420 Execution Unit 430 Specific part 440 Control Unit

Claims

1. An information processing system including one or more computer processors, the one or more computer processors: a receiving unit capable of receiving information for generating a moving image, the information including information on the user's movement, information on the user's voice, and information on the character object, transmitted from a user terminal of the user; an execution unit that executes a video chat between a plurality of users using the character object based on the information for generating the moving image received by the receiving unit; An identification unit that identifies a state of the user terminal; a control unit that changes a display mode of the character object corresponding to the user terminal according to the state of the user terminal identified by the identification unit; An information processing system comprising:

2. The information processing system described in Claim 1, characterized in that the control unit changes the display mode of the character object corresponding to the user terminal from one that reflects the user's movements to one that does not reflect the user's movements, depending on the state of the user terminal identified by the identification unit.

3. The one or more computer processors further include a voice determination unit that determines whether the information about the voice received by the receiving unit includes other voices, The voice determination unit further analyzes the type of the other voice, 2. The information processing system according to claim 1, wherein the control unit changes a display mode of the character object in accordance with the type of other voice analyzed by the voice determination unit.

4. The information processing system described in Claim 1, characterized in that the control unit, as a change in the display mode of the character object, displays a fourth specific object in association with the character object, displays the fourth specific object in the vicinity of the character object, or applies a fourth action to the character object.

5. When the voice determination unit analyzes that the other voice is an animal's cry, The information processing system according to claim 3 , wherein the control unit displays an animal object corresponding to the animal in association with the character object, or displays the animal object by arranging it in the vicinity of the character object.

6. When the voice determination unit analyzes that the other voice is the sound of an intercom, 4. The information processing system according to claim 3, wherein the control unit displays the character object in association with a luggage object, or applies an action of the character object going to pick up luggage.

7. The information processing system according to claim 1, characterized in that the identification unit identifies the user terminal as being in a first state when the receiving unit has not received information regarding the user's movement from the user terminal, or when the information regarding the user's movement received by the receiving unit satisfies a first condition.

8. The information processing system described in claim 7, characterized in that the first condition is that the receiving unit continues to receive information regarding the same movement for a predetermined period of time, or does not receive information regarding the amount of change in movement, which is transmitted only when the movement changes, for a predetermined period of time.

9. The information processing system described in claim 1, characterized in that the identification unit identifies the user terminal as being in a second state when the receiving unit has not received information regarding the voice from the user terminal or when the information regarding the voice received by the receiving unit satisfies a second condition.

10. The information processing system according to claim 1, characterized in that the identification unit identifies the user terminal as being in a third state when the receiving unit receives information indicating that a specific application is launched or displayed on the user terminal.

11. The information processing system according to claim 1, characterized in that the identification unit identifies the user terminal as being in a fourth state when the volume of a voice other than the voice uttered by the user included in the information regarding the voice received by the receiving unit is equal to or greater than a first value.

12. The information processing system according to claim 1, characterized in that the identification unit identifies the user terminal as being in a fifth state when the receiving unit does not receive information regarding the user's movement from the user terminal but receives information regarding the voice.

13. The receiving unit further receives location information of the user terminal transmitted from the user terminal; The information processing system according to claim 1 , wherein the identification unit is configured to identify that the user terminal is in a sixth state when the location information satisfies a predetermined condition.

14. The receiving unit further receives instruction information transmitted from the user terminal; the control unit changes a display mode of the character object in response to the instruction information; The information processing system according to claim 1, characterized in that the control unit changes the display mode of the character object by attaching a seventh specific object to the character object and / or applying a seventh specific action to the character object.

15. The information processing system according to claim 1, characterized in that the identification unit identifies the user terminal as being in an eighth state when the volume of the user's speech contained in the information regarding the voice received by the receiving unit satisfies a predetermined condition.

16. one or more computer processors, a receiving step capable of receiving information for generating a moving image, the information including information on the user's movement, information on the user's voice, and information on the character object, transmitted from a user terminal of the user; an execution step of executing a video chat between a plurality of users using the character object based on the information for generating the moving image received in the receiving step; A determination step of determining a state of the user terminal; a control step of changing a display mode of the character object corresponding to the user terminal in accordance with the state of the user terminal identified in the identification step; An information processing method for executing the above.

17. One or more computer processors included in the information processing device, a transmitting step capable of transmitting information for generating a moving image related to the user, the information including information on the user's movement, information on the voice, and information on the character object, to a server device; a receiving step capable of receiving information for generating a video relating to another user from a server device, the information including information relating to the other user's movements, information relating to the voice, and information relating to the character object; an execution step of executing a video chat between a plurality of users using the character object based on information for generating a moving image related to the user and information for generating a moving image related to the other user; A step of identifying a state of the information processing device; a control step of changing a display mode of the character object corresponding to the information processing device according to the state of the information processing device specified in the specifying step; An information processing method for executing the above.

18. One or more computer processors included in the information processing device, a transmission function capable of transmitting information for generating a moving image related to the user, the information including information related to the user's movement, information related to the user's voice, and information related to the character object, to a server device; a receiving function capable of receiving information for generating a video relating to another user, the information including information relating to the other user's movements, information relating to the voice, and information relating to the character object, from a server device; an execution function of executing a video chat between a plurality of users using the character object based on information for generating a moving image related to the user and information for generating a moving image related to the other user; A specification function for specifying a state of the information processing device; a control function of changing a display mode of the character object corresponding to the information processing device according to the state of the information processing device specified in the specifying function; A computer program that achieves this.

Images