Program, information processing method, and information processing apparatus

Abstract

This reduces the processing load for rendering while increasing opportunities for second users watching videos to participate in the video stream. [Solution] The user device 12 performs a first output control process that outputs a video to the display that includes a first avatar object to which tracking data representing the actions of the streaming user has been applied. If the action of the second user watching the video corresponds to a predetermined standard action, the second output control process outputs a video to the display 28 that includes a second avatar object to which the actions of the second user have not been applied, and the first avatar object.

Description

Technical Field

[0006] , , ,

[0005] , , ,

[0001] The present invention relates to a program, an information processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, an information processing system that distributes a video including animation of an avatar object based on the movement of a user has been known (see, for example, Patent Document 1). In this system, a video including two characters is generated in real time. Also, the voice of a voice actor is synthesized into the video in real time and distributed through a network.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in an information processing system that distributes a video, in order to increase the number of viewing users, or to increase the cumulative viewing time or the number of viewing times obtained by accumulating the viewing time, etc., increasing the opportunity for viewing users to participate in the video being distributed has been considered.

[0005] However, even when a plurality of users participate in the distribution of a video as in the above system, the number of participants is limited, and the opportunity for viewing users to participate has not increased.

Means for Solving the Problems

[0006] In one embodiment, a program is provided that is executed by an information processing device. The program causes one or more computers to execute a first output control process that outputs a video to a display that includes a first avatar object to which tracking data representing the actions of a first user has been applied. The second output control process, when an action taken by a second user on the video corresponds to a predetermined standard action, displays in the video, in addition to the first avatar object, a second avatar object which has a lower rendering processing load than the first avatar object and corresponds to the second user, and information corresponding to the standard action, for a display period shorter than the display period of the first avatar object.

[0007] In another embodiment, an information processing method is provided. The information processing method includes: a first output control process in which one or more computers output a video to a display that includes a first avatar object to which tracking data representing the actions of a first user has been applied; and a second output control process in which, when an action taken by a second user on the video corresponds to a predetermined standard action, a second avatar object that has a lower rendering processing load than the first avatar object and corresponds to the second user, and information corresponding to the standard action are displayed on the video for a display period shorter than the display period of the first avatar object.

[0008] In another embodiment, an information processing device is provided. The information processing device includes: a first processing unit that causes a video including a first avatar object to which tracking data representing the actions of a first user is applied to be output to the display of a second user device used by a second user based on a request to view the video from the second user device; a second processing unit that obtains from the second user device information corresponding to a predetermined standard action which is an action taken by the second user on the video; a third processing unit that obtains from the second user device data for drawing a second avatar object which has a lower rendering processing load than the first avatar object and corresponds to the second user; and a fourth processing unit that, when the action taken by the second user on the video corresponds to the standard action, causes the second avatar object and the information corresponding to the standard action to be displayed in the video in addition to the first avatar object for a display period shorter than the display period of the first avatar object.

[0009] In another embodiment, an information processing method is provided. The information processing method includes: a first process in which one or more computers cause a video including a first avatar object to which tracking data representing the actions of a first user is applied to be output to the display of a second user device used by a second user based on a request to view the video; a second process in which information corresponding to a predetermined standard action, which is an action of the second user on the video, is obtained from the second user device; a third process in which data for drawing a second avatar object that has a smaller rendering processing load than the first avatar object and corresponds to the second user is obtained from the second user device; and a fourth process in which, if the action of the second user on the video corresponds to the standard action, the second avatar object and the information corresponding to the standard action are displayed in the video in addition to the first avatar object for a display period shorter than the display period of the first avatar object. [Effects of the Invention]

[0010] According to the present invention, it is possible to reduce the processing load for rendering while increasing the opportunities for viewers to participate in video streaming. [Brief explanation of the drawing]

[0011] [Figure 1] This is a schematic diagram showing one embodiment of the information processing system in the first embodiment. [Figure 2] This figure shows the user management data for the same embodiment. [Figure 3] This figure shows the configuration of the main body parts data for the same embodiment. [Figure 4] This figure shows the configuration of the mounting parts data for the same embodiment. [Figure 5] This figure shows the viewing screen for collaborative streaming in the same embodiment. [Figure 6] This flowchart shows the procedure for displaying a simplified avatar object according to the same embodiment. [Figure 7] This figure shows the home screen of the same embodiment. [Figure 8] This figure shows the viewing screen of the same embodiment. [Figure 9] This figure shows the viewing screen of the same embodiment. [Figure 10] This figure shows the viewing screen of the same embodiment. [Figure 11] This figure shows the viewing screen of the same embodiment. [Figure 12] This figure shows the viewing screen of the same embodiment. [Figure 13] This figure shows the distribution screen of the same embodiment. [Figure 14] This flowchart shows the procedure for switching modes in the second embodiment. [Figure 15] This figure shows the viewing screen of the second embodiment. [Figure 16] This flowchart shows the procedure for displaying a gift object in another embodiment. [Modes for carrying out the invention]

[0012] (First Embodiment) Hereinafter, a first embodiment of a program, an information processing method, and an information processing apparatus will be described.

[0013] <Information Processing System> Referring to FIG. 1, the information processing system 11 will be described. The information processing system 11 includes a plurality of user devices 12 and a server 13. The user device 12 and the server 13 transmit and receive data via a network 14. The user device 12 receives data necessary for video distribution and viewing from the server 13 based on a user's input operation.

[0014] A user who distributes a video using the user device 12 is called a distribution user, and a user who views the distributed video is called a viewing user. A user can be both a distribution user and a viewing user. That is, a user is a viewing user when viewing a video, and a distribution user when distributing a video. Also, the user device 12 used by the distribution user is referred to as the distribution user device 12A, and the user device 12 used by the viewing user is referred to as the viewing user device 12B. When not distinguishing between the distribution user and the viewing user, and between the distribution user device 12A and the viewing user device 12B, they are simply referred to as the user and the user device 12. Note that the distribution user corresponds to the first user in the claims, and the viewing user corresponds to the second user in the claims. Also, the distribution user device 12A corresponds to the first user device, and the viewing user device 12B corresponds to the second user device.

[0015] The user device 12 is a smartphone (multifunction telephone terminal), tablet terminal, personal computer, console game machine, wearable computer, or any other information processing device capable of playing videos. A wearable computer only needs to have a screen on which the user can view videos, and may be, for example, a head-mounted display worn on the user's head, or a glasses-type wearable terminal. The user device 12 may be a standalone device, or it may consist of multiple devices connected to each other so that various types of data can be sent and received from each other. An example of the latter user device 12 is a system that performs outside-in tracking.

[0016] The user device 12 comprises a control unit 20, storage 22 (storage medium), and a communication interface (I / F) 23. The control unit 20 comprises one or more arithmetic circuits such as a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Network Processing Unit). The control unit 20 also comprises memory, which is a main memory (storage medium) that the arithmetic circuits can read from and write to. The memory is composed of semiconductor memory or the like. The control unit 20 reads an operating system or other program from storage 22 or external storage into memory and executes instructions retrieved from memory. The communication I / F 23 can send and receive data with the server 13 and other user devices 12 via the network 14. The network 14 includes various networks such as a local area network and the internet. The control unit 20 corresponds to the first output control unit and the second output control unit of the claims. The storage 22 corresponds to the storage unit of the claims.

[0017] Storage 22 is an auxiliary storage device (storage medium), such as a magnetic disk, optical disk, or semiconductor memory. Multiple storage devices can be used in combination.

[0018] The control unit 20 executes various programs recorded in the storage 22 based on the user's input to the operation unit 27, thereby performing various processes related to video distribution and viewing. For the sake of explanation, the state in which video is being distributed will be referred to as the distribution mode, and the state in which videos distributed by other distribution users are viewed will be referred to as the viewing mode.

[0019] The user device 12 also includes a sensor unit 24, a speaker 25, a microphone 26, an operating unit 27, and a display 28. At least one of the sensor unit 24, speaker 25, microphone 26, operating unit 27, and display 28 may be provided in the same device as the control unit 20, or it may be provided in a manner that connects it to the device containing the control unit 20.

[0020] The sensor unit 24 is one or more sensors that detect face motion, which indicates changes in the user's facial expression, and body motion, which indicates changes in the relative position of the user's body to the sensor unit 24. Face motion includes movements such as blinking and opening and closing the mouth. The sensor unit 24 can be any known sensor. An example of the sensor unit 24 is a ToF sensor that measures the time of flight (TF) of light irradiated towards the user until it reflects off the user's face and returns, a camera that photographs the user's face, and an image processing unit that processes the data captured by the camera. The sensor unit 24 may also include an RGB camera that captures visible light and a near-infrared camera that captures near-infrared light. The RGB camera and near-infrared camera can be, for example, True Depth from "iPhone X®" or "LIDAR" (Light Detection and Ranging, or Laser Imaging Detection and Ranging) from "iPad Pro®" or other ToF sensors installed in smartphones. Specifically, this camera projects tens of thousands of invisible dots onto the user's face using a dot projector. It then detects and analyzes the reflected light from the dot pattern to form a depth map of the face and captures infrared images of the face, thereby capturing accurate facial data. The processing unit of the sensor unit 24 generates various information based on the depth map and infrared images, and compares this information with registered reference data to calculate the depth of each point on the face (the distance between each point and the near-infrared camera) and positional deviations other than depth.

[0021] Furthermore, the sensor unit 24 may have a function to track not only the user's face but also their hands (hand tracking). The sensor unit 24 may also include sensors that detect the position or orientation of human body parts other than the hands. The sensor unit 24 may further include sensors such as acceleration sensors that detect velocity and acceleration, and gyroscope sensors that detect direction and orientation. The sensor unit 24 may have a spatial mapping function that recognizes objects in the real space where the user is located based on the detection results of the ToF sensor and other known sensors, and maps the recognized objects onto a spatial map. Hereinafter, when face motion detection data and body motion detection data are not specifically distinguished, they will simply be referred to as "tracking data." Note that the image processing unit of the sensor unit 24 may be provided by the control unit 20.

[0022] The speaker 25 converts audio data into sound and outputs it. The microphone 26 inputs the voice spoken by the user and converts it into audio data. The display 28 outputs various images according to the output instructions of the control unit 20.

[0023] The operation unit 27 can be of a type appropriate to the user device 12. One example of the operation unit 27 is a touch panel integrated with the display 28. Other examples of the operation unit 27 include operation buttons, a keyboard, a mouse, or a controller operated by the user's hand, which are provided on the housing of the user device 12. The controller may incorporate various known sensors such as acceleration sensors and inertial measurement units (IMUs) such as gyroscopes. Another example of the operation unit 27 is a tracking device that identifies the user's hand movements, eye movements, head movements, and gaze direction. In this embodiment, for example, the user's instructions can be determined based on the user's hand movements, and various operations such as starting or ending video distribution, evaluating messages and videos, and displaying predetermined objects (e.g., gifts described later) can be performed. If the sensor unit 24 also has input interface functions such as a hand tracking function, the operation unit 27 can be omitted.

[0024] <server> Next, the server 13 will be described. The server 13 is used by service providers and others who provide services for distributing video. The server 13 comprises a control unit 30, a communication interface 34, and storage 35. The control unit 30 has one or more arithmetic circuits. The control unit 30 has the same configuration as the control unit 20 of the user device 12. The communication interface 34 and storage 35 also have the same configuration as the user device 12. The server 13 may consist of one device or multiple devices. If the server 13 consists of multiple devices, these devices distribute the functions of the control unit 30 and the various data stored in the storage 35. The control unit 30 corresponds to the first processing unit, second processing unit, third processing unit, and fourth processing unit of the claims.

[0025] <Video distribution method> One of the following video distribution methods can be used: client rendering, browser rendering, video distribution, or server distribution.

[0026] The client rendering method involves the distribution user device 12A and the viewing user device 12B receiving video configuration data necessary for video generation from the server 13 and rendering the video. The video configuration data includes tracking data detected by the sensor unit 24 of the distribution user device 12A. In addition, each user device 12 acquires audio data based on the speech of the distribution user and outputs the audio in synchronization with the video. In this case, the storage 22 of each user device 12 stores the application program and various data such as rendering data for rendering avatars.

[0027] The browser rendering method is a method in which the distribution user device 12A and the viewing user device 12B render video using a web browser program stored in storage 22. At this time, the distribution user device 12A sends tracking data, etc. to the server 13. The user device 12 downloads a web page from the server 13 that is written in a markup language such as HTML (Hyper Text Markup Language) and uses CSS (Cascading Style Sheets), JavaScript (registered trademark), etc. The web browser program launched on the user device 12 renders avatar objects and other objects using JavaScript that runs on the browser. In the browser rendering method, the user device 12 stores the data received from the server 13 in volatile memory and data such as the URL of the web page for displaying the video in non-volatile memory. For this reason, the rendering data for avatar objects is not stored in the user device 12.

[0028] The video distribution method involves the distribution user device 12A generating video data. The viewing user device 12B displays the video on the display 28 based on the video data received from the distribution user device 12A. Specifically, the distribution user device 12A generates video data using tracking data, etc. The distribution user device 12A also transmits the encoded video data and audio data to the server 13. The server 13 transmits the video data and audio data to the viewing user device 12B.

[0029] The server distribution method is one in which server 13 generates video data. Server 13 receives tracking data and audio data from the distribution user device 12A. Server 13 then generates video data based on this data. Server 13 transmits the generated video data to the distribution user device 12A and the viewing user device 12B. The distribution user device 12A and the viewing user device 12B output the video to the display 28 based on the received video data.

[0030] Thus, either the user device 12 or the server 13 is responsible for generating the video data. The device responsible for generating the video data can be changed depending on the distribution method. When the user device 12 is responsible for generating the video data, the viewing user device 12B may generate the video data, or the distribution user device 12A may generate the video data. In addition, the user device 12 may send the generated video data to another user device 12 as the destination as needed.

[0031] Furthermore, the distribution user device 12A and the viewing user device 12B may display the video using different methods. For example, the distribution user device 12A may display the video using the client rendering method, and the viewing user device 12B may display the video using the browser rendering method.

[0032] In this embodiment, a method by which each user device 12 distributes video using a client rendering method will be described. In the client rendering method, the user device 12 stores the video application program in storage 22. Storage 22 also stores 3D model data of avatar objects and other objects used to execute the video application program.

[0033] The control unit 20 of the distribution user device 12A transmits video configuration data, including tracking data acquired from the sensor unit 24, and audio data based on the sound collected by the microphone 26, to the server 13. The distribution user can set the parts of the avatar object corresponding to themselves to their preferred parts. The video configuration data includes identification information (part ID) indicating the parts of the avatar object. The distribution user can also adjust the position of the virtual camera in the virtual space. The video configuration data may also include the position of the virtual camera. The tracking data and audio data are timestamped. The timestamp is generated based on reference time information based on the circuit (internal clock) having a timing function in the control unit 20. Alternatively, the timestamp is generated based on time information transmitted from the server 13 or reference time information acquired from the server 13 that provides a time distribution service. The timestamp may represent the elapsed time from a reference time, or it may represent the time.

[0034] The viewing user device 12B has received the part ID of the streaming user's avatar object in advance at a predetermined timing, such as when the user starts watching the video. The viewing user device 12B applies tracking data to the streaming user's avatar object. The control unit 20 of the viewing user device 12B also performs rendering, including the avatar object and other objects. Rendering here refers to drawing processes that include acquiring the position of the virtual camera, perspective projection, and hidden surface removal based on depth information associated with each object. Rendering may consist of at least one of these processes, or it may include processes such as shading and texture mapping.

[0035] The control unit 20 receives data such as posted messages or notifications from the viewing user from the server 13. The control unit 20 outputs video data to the display 28, which is a composite of images of avatar objects and other objects, and posted messages or notifications. The control unit 20 also outputs audio data from the speaker 25, synchronized with the video data based on a timestamp.

[0036] Furthermore, the control unit 20 performs actions based on the viewer's actions, such as sending a request to display a gift object (gift display request), posting a message to the video, and evaluating the video (selecting the "Like!" button or the "favorite" button).

[0037] <Structure of data used in the video> Next, with reference to Figures 2 to 4, we will describe in detail each piece of data used for video distribution or viewing.

[0038] Figure 2 shows an example of user management data 50. User management data 50 is stored on server 13. User management data 50 includes user identification information such as user ID, coins, and points. Coins are a medium that users can purchase. Points are a medium that can be obtained through actions such as watching videos and streaming videos. In addition, user management data 50 may also include scores accumulated by participating in events. Coins, points, and scores can be used within the video streaming service. Note that user device 12 may also store user management data 50.

[0039] Figure 3 shows an example of the main body part data 51 of an avatar object. The main body part data 51 is stored in the storage 22 of the user device 12. This section describes the main body parts. Main body parts are the components that make up the parts of an avatar object. The data used to render the main body parts includes polygon data, skeletal data (bones) to represent the movement of the avatar object, texture data, etc. In addition, avatar attributes are associated with the main body parts. Avatar attributes include, for example, type (male type such as "M type", female type such as "F type"), height, skin color, body type, etc. The skeletal data, polygon data, etc. of the avatar object differ depending on the avatar attributes. By selecting avatar attributes, the user can select main body parts such as the upper body or lower body. Furthermore, some of these main body parts can be arbitrarily set by the user. For example, the user can select parts such as eyes, mouth, and hairstyle. In this embodiment, each user can register up to three avatar objects as avatars that they can use.

[0040] The main body part data 51 includes information about the avatar's slot number, model, height, skin color, body type, and other main body parts. The main body part data 51 contains various types of information for each avatar slot number. The model is the avatar attribute selected by the user. Height, skin color, and body type are identification information for the height, skin color, and body type selected by the user. Before the user makes a selection, the height, skin color, and body type remain at their default settings. The user can change the settings for height, skin color, and body type. Parts are the IDs of individual parts such as eyes, mouth, and hairstyle. Furthermore, some wearable parts allow for color selection. For example, the wearable part "T-shirt" allows for the selection of "red," "blue," and "green." Note that before the user selects main body parts, the default main body parts are applied.

[0041] Figure 4 shows an example of the attachment parts data 52. Attachment parts are parts applied to specific parts of an avatar object. Examples of attachment parts include "tops" applied to the upper body, "shoes" applied to the feet, and "skirts" applied to the lower body. Users can select their preferred attachment parts from multiple options corresponding to the avatar attributes of the avatar object. In other words, viewers can apply attachment parts to each avatar object with slot numbers "1" through "3". Note that the default attachment parts are applied before the user selects any attachment parts.

[0042] Each attachable part has attribute data associated with it that indicates a part of the avatar object. Note that attachable parts only need to be associated with a specific part of the avatar object; they can be clothing, or other items such as "cups," "mugs," "umbrellas," or "musical instruments." For example, a "cup" is associated with the hand of an avatar object. Applying the "cup" part to the hand allows the avatar object to be displayed holding a "cup." Attachable parts include standard parts stored when the application program is installed, and parts granted by Server 13 when the user exchanges coins or participates in events.

[0043] The attached part data 52 includes the part ID, item name, part, attribute, color, and the slot number of the avatar object to which it is applied. The part ID is the identification information (ID) of the selected attached part. The item name is the name of the attached part. For example, the name of the item, such as "Beginner T-shirt," is stored. The part is the part of the avatar object to which the attached part is applied. The attribute is the attribute of the avatar to which the attached part is applied, and can be set to all attributes (e.g., "All"), "M type," or "F type." The color is the color of the attached part, and indicates the color selected by the user. The applied slot number indicates the slot number of the avatar object to which the attached part is applied, among the avatar objects registered by the user.

[0044] The user device 12 may store data for unattached parts in addition to the attached parts data 52. Unattached parts are parts other than attached parts that are applied to the virtual space. For example, unattached parts are objects such as wallpaper that are displayed as a background. The data for unattached parts includes the part ID, item name, and color for each part. Unattached parts may also be stored in association with the avatar object to which they are applied.

[0045] <Collaboration Streaming Feature> Next, we will explain the collaborative streaming function (co-starring function). In collaborative streaming, users other than the streaming user can display their own corresponding avatar object in the video being streamed as co-stars. Co-starring in collaborative streaming is only possible in videos where the streaming user has enabled collaborative streaming. In other words, users permitted by the streaming user can become co-stars. In addition to the streaming user, one to a predetermined number of co-starring users can participate in collaborative streaming. The predetermined number is, for example, three people. Note that the streaming user's avatar object 110 corresponds to the first avatar object in the claims.

[0046] To participate in a collaborative stream, user device 12 sends a participation request to streaming user device 12A. When the streaming user approves the participation request, the streaming user becomes the host user and the user who submitted the participation request becomes the guest user in the collaborative stream. Then, a host avatar object corresponding to the host user and a guest avatar object corresponding to the guest user are displayed in the video.

[0047] Figure 5 shows the viewing screen 60 for collaborative streaming. This viewing screen 60 is displayed on the display 28 of the viewing user device 12B. A nearly identical screen is displayed on the streaming user device 12A.

[0048] The viewing screen 60 includes an image of the host avatar object 61. The viewing screen 60 shown in Figure 5 is an example that displays the host avatar object 61 as well as the guest avatar objects 62-64 of three guest users. The guest avatar object 62 corresponds to the third avatar object.

[0049] The first guest avatar object 62 is the avatar object corresponding to the first guest user. The guest avatar object 62 is displayed at position 71 to the right of position 70 of the host avatar object 61. Both the host avatar object 61 and the guest avatar object 62 are rendered using three-dimensional model data. Furthermore, tracking data that detects the host user's movements is applied to the host avatar object 61. In other words, the host avatar object 61 moves in accordance with the host user's body movements. Tracking data that detects the first guest user's movements is applied to the guest avatar object 62. In other words, the guest avatar object 62 moves in accordance with the first guest user's body movements. The guest avatar object 62 to which tracking data is applied is called the second avatar object.

[0050] The guest avatar object 63 of the second guest user, whose participation request was approved after the first guest user, is displayed at position 72 on the left side behind the host avatar object 61 and guest avatar object 62. The guest avatar object 64 of the third guest user, whose participation request was approved after the second guest user, is displayed at position 73 on the right side behind them. Guest avatar objects 63 and 64 are rendered using three-dimensional model data. Tracking data detecting the movements of the second guest user is applied to guest avatar object 63. Tracking data detecting the movements of the third guest user is applied to guest avatar object 64.

[0051] The guest avatar object 62 exits upon operation by the first guest user. Alternatively, the guest avatar object 62 may automatically exit from position 71 after a predetermined time has elapsed since its display began. Then, the guest avatar object 63 of the second guest user is placed at position 71. The guest avatar object 64 of the third guest user is placed at position 72. Furthermore, if there are any new guest avatar objects waiting to perform, those avatar objects are displayed at position 73. Note that if only the guest avatar objects 62 and 63 of two performers exist within the four-person capacity, and there are no waiting guest users, those guest avatar objects 62 and 63 may remain displayed.

[0052] <Simple avatar object display function> Next, the display function of the avatar object 110 corresponding to the viewing user will be described. An avatar object 110 that has a low rendering processing load and corresponds to a viewing user will be referred to as a simplified avatar object below. The image data of the simplified avatar object is pre-generated by the control unit 20 of the user device 12. The simplified avatar object is the avatar object of the slot number selected by the viewing user corresponding to the simplified avatar object. The attachment parts selected by that viewing user are applied to this avatar object. In other words, it is generated based on the main body parts and attachment parts that constitute the three-dimensional avatar object 110 used for video distribution. Furthermore, the generated image data of the simplified avatar object is stored in the storage 22 of the user device 12. Note that the image data of the simplified avatar object may also be generated by the server 13.

[0053] Simplified avatar objects are data with reduced rendering processing load. Simplified avatar objects reduce processing load by employing at least one of the following methods: rendering them as two-dimensional objects, not applying tracking data, and reducing the number of polygons. Furthermore, to reduce the processing load of the user device 12, the image data of the simplified avatar object may be a single integrated part. Also, components that perform physical behavior may be omitted from the simplified avatar object. In addition, the screen size of the simplified avatar object is smaller than the image sizes of the host avatar object 61 and guest avatar objects 62-64.

[0054] In this embodiment, the simplified avatar object is rendered as a two-dimensional object. Furthermore, tracking data is not applied to the simplified avatar object. Furthermore, the display of the simplified avatar object does not require permission from the broadcasting user; it is displayed when a viewer performs a predetermined standard action. In other words, if the broadcasting user had to perform an operation to allow or deny the display of the simplified avatar object each time a standard action is performed during video streaming, the broadcasting user's operations would become cumbersome. For this reason, the simplified avatar object is displayed without the broadcasting user's permission. Also, since tracking data is not applied to the simplified avatar object, the likelihood of viewers engaging in disruptive behavior within the video is low. For this reason, there is little need for broadcasting users to grant permission for the display of the simplified avatar object.

[0055] Furthermore, the simplified avatar object is placed in an area for displaying the GUI (Graphical User Interface). Hereafter, this area will be referred to as the UI display area. The GUI placed in the UI display area is displayed at the forefront in the three-dimensional virtual space.

[0056] Because simplified avatar objects are displayed in the UI display area, they are not associated with depth information (Z value, position in the screen depth direction) relative to the virtual camera set up in the virtual space. Depth information indicates the distance from the virtual camera that displays the screen. Objects that are farther from the virtual camera have a greater depth. Therefore, when rendering simplified avatar objects, there is no need to perform depth testing (hidden surface removal) when rendering them with other three-dimensional objects. In depth testing, it is determined whether one object is in front of or behind another object, and if it is behind the other object, the unhidden area of ​​one object is rendered, and the hidden area is not rendered. One method of depth testing is, for example, Z testing (Z-buffering). Guest avatar objects 62-64 require depth testing, but simplified avatar objects do not need to be tested because they are not associated with depth information. Therefore, the processing load for rendering simplified avatar objects can be reduced compared to the processing load for rendering guest avatar objects 62-64.

[0057] A simplified avatar object is displayed on the user device 12's display 28 when the viewer performs one of the following criteria actions. • When you start watching a video.

[0058] When you evaluate a video. When you post a message. • When a gift display request is submitted.

[0059] When the viewing user device 12B receives a command operation for a standard action from the viewing user, it transmits various data associated with the executed standard action and image data of the simplified avatar object to the server 13. At this time, instead of image data, it may transmit the part IDs of the main body part and attached parts for displaying the simplified avatar object. Alternatively, the viewing user device 12B may transmit the image data or part IDs of the simplified avatar object to the server 13 in advance before receiving the command operation for the standard action.

[0060] Server 13 transmits data corresponding to the reference action and a request to display the data corresponding to the reference action to the distribution user device 12A and multiple viewing user devices 12B. The receiving viewing user devices 12B may or may not include the viewing user device 12B that transmitted the image data of the simplified avatar object, etc. If the receiving viewing user devices 12B do not include the viewing user device 12B that transmitted the image data of the simplified avatar object, etc., that viewing user device 12B displays the screen based on the various data associated with the reference action and the image data of the simplified avatar object.

[0061] The streaming user device 12A and the viewing user device 12B receive various data associated with the executed reference action and image data of the simplified avatar object from the server 13. Then, the streaming user device 12A and the viewing user device 12B output the host avatar object 61, the simplified avatar object, and information corresponding to the reference action to the display 28.

[0062] The simplified avatar object 80 is displayed in the video for a predetermined period of time. In other words, the period during which the simplified avatar object 80 is displayed is shorter than the period during which the avatar object 110 of the streaming user is displayed. Alternatively, the simplified avatar object 80 may be removed from the video based on the actions of the streaming user or the actions of the viewer corresponding to the simplified avatar object 80.

[0063] (operation) Next, the operation of the user device 12 and the server 13 will be described. The user device 12 performs a first output control process, a second output control process, and a third output control process.

[0064] In the first output control process, the control unit 20 outputs a video to the display 28 that includes an avatar object 110 to which tracking data representing the actions of the streaming user has been applied. In the second output control process, if the user's action on the video corresponds to a predetermined standard action, the control unit 20 displays a simplified avatar object and information corresponding to the standard action on the video in addition to the avatar object 110, for a display period shorter than the display period of the avatar object 110. The simplified avatar object has a lower rendering processing load than the first avatar object and is also compatible with the user.

[0065] In the third output control process, the control unit 20 outputs a video to the display 28 that includes the broadcasting user's avatar object 110 and a guest avatar object 62 representing the actions of the co-starring user when another user joins the video as a co-starring user. Tracking data representing the actions of the co-starring user is applied to the guest avatar object 62 of the co-starring user.

[0066] Server 13 performs the first, second, third, and fourth processes. In the first process, the control unit 30 outputs a video including an avatar object 110 to which tracking data representing the actions of the distribution user has been applied, to the display 28 of the viewing user device 12B used by the viewing user, based on a request to view the video from the viewing user device 12B.

[0067] In the second process, the control unit 30 obtains information corresponding to the viewer user's reference action from the viewer user device 12B. In the third process, the control unit 30 obtains data from the viewing user device 12B for drawing a simplified avatar object that corresponds to the viewing user and has a lower rendering processing load than the distribution user's avatar object 110.

[0068] In the fourth process, if the user's action on the video corresponds to a reference action, the control unit 30 displays a simplified avatar object and information corresponding to the reference action in the video, in addition to the avatar object 110, for a display period shorter than the display period of the avatar object 110.

[0069] Next, referring to Figure 6, the procedure for the user viewing device 12B to transmit image data of a simple avatar object will be described in detail. It is assumed that the user viewing device 12B has started watching the video selected by the user viewing device.

[0070] The control unit 20 of the viewing user device 12B determines whether or not it has detected an action by the viewing user based on the user's input operation to the operation unit 27 (step S10). If it determines that there is no action (step S10: NO), the process proceeds to step S14.

[0071] On the other hand, when the control unit 20 detects an action by the viewing user (step S10: YES), it determines whether or not to display a simplified avatar object, which is the viewing user's avatar object (step S11). Specifically, the control unit 20 determines whether or not the detected action corresponds to a standard action for displaying the simplified avatar object. A list of standard actions is stored in advance in storage 22 or the like.

[0072] If the control unit 20 determines that it does not want to display the simplified avatar object (step S11: NO), it performs processing according to the action without sending the image data of the simplified avatar object (step S13). Actions that do not correspond to actions that display the simplified avatar object include following the streaming user, sharing the streaming video, capturing (screenshot) the streaming video, and ending the viewing of the video.

[0073] On the other hand, if the control unit 20 determines to display a simplified avatar object (step S11: YES), it sends image data of the simplified avatar object to the server 13 (step S12). For example, when a viewer gives a positive rating to a video, the control unit 20 sends a request to reflect the positive rating in the video and image data of the simplified avatar object to the server 13. Also, when a viewer posts a message to a video, the control unit 20 sends data containing the content of the message and image data of the simplified avatar object to the server 13. Furthermore, when a viewer sends a gift display request, the control unit 20 sends the gift display request, the ID of the gift object selected by the viewer, and image data of the simplified avatar object to the server 13.

[0074] In step S14, the control unit 20 determines whether or not the video distribution has ended. If it determines that the video distribution has not ended (step S14: NO), it returns to step S10. If it determines that the video distribution has ended (step S14: YES), it terminates the process.

[0075] (Viewing screen) Next, with reference to Figures 7 to 12, the screens that the user device 12 displays on the display 28 will be described.

[0076] Referring to Figure 7, the home screen 101 will be described. When a user launches a video application, the user device 12 receives a list of videos currently being streamed from the server 13. Based on the list, the user device 12 outputs the home screen 101 to the display 28. The home screen 101 displays a video selection unit 102 that shows the videos available for viewing. When a user selects a video from the video selection unit 102, the user device 12 sends the ID of the selected video along with a viewing request to the server 13. The server 13 updates the number of users viewing the video and sends video configuration data and audio data for rendering the video to the user device 12. As a result, the user device 12 enters viewing mode.

[0077] When the viewing user device 12B receives video configuration data, etc., from the server 13, it generates video data using the video configuration data. Then, based on the video data, it outputs the viewing screen 100 to the display 28.

[0078] The content of the video, including the avatar object 110, is almost identical in the viewing screen 100 and the distribution screen displayed on the distribution user device 12A, with some exceptions. Therefore, Figures 8 to 12 illustrate the display modes of the simplified avatar object 80 using the viewing screen 100 as an example.

[0079] Figure 8 shows an example of the viewing screen 100. The viewing screen 100 includes an image of an avatar object 110 corresponding to the streaming user. Tracking data based on the streaming user's movements is applied to the avatar object 110. Drawing data for attached parts 111 is applied to the avatar object 110. Unattached parts 125, which are wallpaper, are applied to the virtual space. The viewing user device 12B also outputs sound from the speaker 25 in synchronization with the animation of the avatar object 110.

[0080] Furthermore, the viewing screen 100 includes the number of viewers 112 and the total number of hearts 113. The number of viewers 112 is the cumulative number of viewers who have watched the video. Alternatively, the number of viewers 112 may be the number of viewers currently watching the video. The total number of hearts 113 is the number of times viewers have given a positive rating to the video since it started.

[0081] The viewing screen 100 also includes a viewing ranking button 114, a streaming user ranking button 115, a share button 116, and a follow button 117. When a viewing user operates the viewing ranking button 114, the viewing user device 12B displays the ranking of viewing users based on the number of gifts sent, among viewing users who have watched videos streamed by that streaming user. The streaming user ranking button 115 displays the ranking of streaming users based on the number of gift objects received, for all streaming users who stream videos. For example, this ranking is calculated based on all streaming users. The ranking is also sorted by the number of gift objects received, from most to least, for each type of gift object, such as a "bouquet."

[0082] When a viewer selects the share button 116, the viewer device 12B sends the video link (URL) to other social networking services, copies the link, or reads the follower list of users who mutually follow each other. When a viewer selects the follow button 117, the server 13 associates this viewer with the distribution user who distributes this video.

[0083] The viewing screen 100 also includes a message input section 118 and a message display section 119. The message input section 118 is an input form that allows viewing users to send messages to the broadcasting user. The message display section 119 displays messages sent by each viewing user device 12B, or standard messages sent by the server 13 based on actions taken by viewing users. The message display section 119 displays the most recent messages at the top (or bottom), moves downwards (or upwards) when a new message display section 119 is added, and is erased when it reaches the bottom (or top) of the displayable area of ​​the message display section 119.

[0084] The viewing screen 100 also includes a camera button 120, a like button 121, and a gift button 122. When the camera button 120 is selected, the viewing user device 12B captures the displayed viewing screen 100. The captured viewing screen 100 is then stored in the storage 22. When the like button 121 is selected, the viewing user device 12B displays an indicator of a positive rating, such as a heart icon, on the viewing screen 100. The viewing user device 12B also sends a request to the server 13 to reflect the positive rating. When the server 13 receives the request to reflect the positive rating, it updates the total number of hearts 113. Viewers can display an indicator of a positive rating on the video without making any payment.

[0085] When the gift button 122 is selected, the viewing user device 12B displays a list of gifts overlaid on the viewing screen 100. The viewing user using the viewing user device 12B selects a gift from the list of gifts. The viewing user device 12B sends the gift ID and gift display request of the selected gift to the server 13. When the viewing user device 12B receives the gift ID and gift display request from the server 13, it renders the gift object corresponding to the gift ID together with the avatar object 110. The viewing user makes a payment to display the gift object in the video. The payment that can be made for a gift object is coins and points. As mentioned above, coins are purchased by the user, so gift objects for which coins are paid are paid objects. Points are earned by the user based on user actions related to the video. Therefore, gift objects for which points are paid are free objects.

[0086] As described above, the viewing user device 12B displays a simplified avatar object 80 when the action taken by the viewing user using the viewing user device 12B corresponds to a standard action. The viewing user device 12B generates image data for the simplified avatar object 80 at times such as when a video application program is launched or when viewing a video begins. Specifically, it obtains the part ID or image data of the avatar object 110 selected by the viewing user from among the avatar objects 110 with slot numbers "1" to "3". The image data of the avatar object 110 obtained at this time has the attached parts 111 that the viewing user has previously applied applied to it. In other words, since the viewing user's own simplified avatar object 80 is displayed in the video, it becomes possible for the viewing user to make adjustments to the attached parts applied to the simplified avatar object 80, such as matching them to the streaming user or to match the season in the real world.

[0087] The control unit 20 then optimizes the image data. As described above, for example, the viewing user device 12B converts the image data of the avatar object 110 into two dimensions. It also makes the simplified avatar object 80 smaller than the size of the avatar object 110. The viewing user device 12B generates image data for the simplified avatar object 80 and stores it in the storage 22. When a viewing user using the viewing user device 12B performs a standard action, the viewing user device 12B reads the image data of the simplified avatar object 80 from the storage 22 and sends it to the server 13.

[0088] Alternatively, the server 13 may perform the image data generation process for the simplified avatar object 80. In this case, the server 13 receives the ID of the main body part and the ID of the attached part from the viewing user device 12B and generates the image data for the simplified avatar object 80 based on these IDs. Or, the image data generation process for the simplified avatar object 80 may be performed so that each viewing user device 12B generates the image data when rendering the viewing screen 100. In this case, the viewing user device 12B receives the ID of the main body part and the ID of the attached part from the viewing user device 12B of the viewing user who performed the action, via the server 13, and generates the image data for the simplified avatar object 80 based on these IDs.

[0089] In the viewing screen 100 shown in Figure 8, the simplified avatar object 80 of the viewing user who started watching the video is displayed. When another viewing user starts watching, the viewing user device 12B receives image data of the simplified avatar object 80 corresponding to the other viewing user from the server 13 and displays the simplified avatar object 80. In addition, when a viewing user using the viewing user device 12B starts watching a video, the viewing user device 12B sends the image data of the simplified avatar object 80 to the server 13. Furthermore, the image of the simplified avatar object 80 read from the storage 22 is displayed on the display 28, or the image of the simplified avatar object 80 is displayed based on the image data of the simplified avatar object 80 received from the server 13.

[0090] Because the viewing screen 100 has limited display space, it is difficult to lay out a large number of simple avatar objects 80. Therefore, the viewing user device 12B displays the simple avatar objects 80 one by one. At this time, the image of the simple avatar object 80 is displayed in such a way that it can be determined that it is associated with a message display unit 119 such as "You have entered the room." In the example in Figure 8, it is displayed outside the message display unit 119, at the same position as the message display unit 119 in the vertical direction of the viewing screen 100.

[0091] For example, the viewing user device 12B displays the simplified avatar objects 80 based on the order in which it received image data of the simplified avatar objects 80 from the server 13. The viewing user device 12B outputs the image of the simplified avatar object 80 at the same time as the message display unit 119 indicating that video viewing has started. The viewing user device 12B also displays the image of the simplified avatar object 80 adjacent to the message display unit 119 for a certain period of time. The viewing user device 12B may also erase the image of the simplified avatar object 80 before the message display unit 119 is erased. In this way, the message display unit 119 can be made viewable while securing display space for other simplified avatar objects 80. Alternatively, the image of the simplified avatar object 80 may be erased at the same time as the message display unit 119.

[0092] In the viewing screen 100 shown in Figure 9, a simplified avatar object 80 of a viewing user who gave a positive rating to the video is displayed. When the viewing user device 12B receives a request to display the rating index 129 and image data of the simplified avatar object 80 from the server 13, it displays the rating index 129 and the image of the simplified avatar object 80 based on these. Furthermore, when a viewing user using the viewing user device 12B performs an operation to give a positive rating to the video, the viewing user device 12B sends the image data of the simplified avatar object 80 along with a request to display the rating index 129 to the server 13.

[0093] The viewing user device 12B displays the image of the simplified avatar object 80 at the same time as the message display unit 119 indicating that a positive rating has been given. In addition, the viewing user device 12B displays the image of the simplified avatar object 80 adjacent to the message display unit 119 indicating that a positive rating has been given for a certain period of time.

[0094] In the viewing screen 100 shown in Figure 10, the simplified avatar object 80 of the viewing user who posted the message is displayed. When the viewing user device 12B receives the message posted by another viewing user and the image data of the simplified avatar object 80 from the server 13, it displays the message display unit 119 and the image of the simplified avatar object 80 based on these. In addition, when a viewing user using the viewing user device 12B posts a message, the viewing user device 12B sends the message along with the image data of the simplified avatar object 80 to the server 13.

[0095] The viewing user device 12B outputs the image of the simple avatar object 80 at the same time as the message display unit 119 which displays the message posted by the viewing user. In addition, the viewing user device 12B displays the image of the simple avatar object 80 adjacent to the message display unit 119 which displays the content of the posted message for a certain period of time.

[0096] In the viewing screen 100 shown in Figure 11, the simplified avatar object 80 of the viewing user who sent the gift object is displayed. When a viewing user using the viewing user device 12B performs an operation to request the display of the gift object 123, the viewing user device 12B sends the image data of the simplified avatar object 80 corresponding to the viewing user along with the gift ID to the server 13. Furthermore, when the viewing user device 12B receives the gift ID and the image data of the simplified avatar object 80, it displays the gift object 123 and the simplified avatar object 80 based on these. The viewing user device 12B has previously received the image data of the gift object 123 from the server 13.

[0097] For example, it is preferable that the viewing user device 12B simultaneously displays the gift object 123 of the received gift ID, the simple avatar object 80, and the message display unit 119. This makes it possible to identify the viewing user who sent the gift object 123.

[0098] The viewing user device 12B may erase the image of the simple avatar object 80 before the message display unit 119 is erased. This allows the message display unit 119 to remain viewable while securing display space for other simple avatar objects 80. Alternatively, the image of the simple avatar object 80 may be erased simultaneously with the message display unit 119.

[0099] The viewing screen 100 shown in Figure 12 displays the viewer's profile as shown by the streaming user. When the streaming user device 12A receives a confirmation operation from the streaming user to verify the viewer's profile, it outputs the viewer's profile information to the display 28. The viewer's profile information is stored in the storage 22 by the viewer user device 12B. The viewer user device 12B also sends the viewer's profile information to the server 13 at predetermined timings, such as when the viewer starts watching a video or when it receives a confirmation operation from the streaming user.

[0100] When a streaming user views a viewing user's profile, server 13 requests image data of the simplified avatar object 80 from the viewing user device 12B used by that viewing user. Upon receiving the image data of the simplified avatar object 80, server 13 transmits the image data of the simplified avatar object 80 to both the streaming user device 12A and the viewing user device 12B.

[0101] The viewing user device 12B displays an image of the simplified avatar object 80 based on the image data received from the server 13. At this time, the simplified avatar object 80 may be displayed in a manner that simulates walking from the edge of the viewing screen 100 to a predetermined position.

[0102] Figure 13 shows an example of a distribution screen 103 displayed on the display 28 by the distribution user device 12A. The distribution screen 103 includes an image of a simplified avatar object 80, similar to the viewing screen 100. The distribution user device 12A displays the profile information 130 of the viewing user corresponding to the simplified avatar object 80, in association with the simplified avatar object 80. Note that this profile information 130 is not displayed on the viewing user device 12B that is watching the video. This prevents the viewing user's profile from being disclosed to an unspecified number of other users against the viewing user's own intentions. In addition, the viewing user can be aware that the distribution user has checked the viewing user's profile.

[0103] Streaming screen 103 includes the number of viewers (112), the total number of hearts (113), and the number of collaborators (131). The number of collaborators (131) indicates the number of collaborators participating in the collaborative stream.

[0104] The streaming screen 103 also includes a share button 116, a co-star confirmation button 137, a streaming user ranking button 115, a settings button 138, and an audio button 139. During a collaborative stream, selecting the co-star confirmation button 137 allows the user to view the co-star's profile information 130. Even when viewing the co-star's profile information 130, the viewing user device 12B displays the viewing screen 100, which does not include the profile information 130, as shown in Figure 12. Selecting the settings button 138 allows the user to configure the streaming settings. When the audio button 139 is selected, the streaming user device 12A either stops (mutes) the audio output or releases the audio muted state.

[0105] The streaming screen 103 also includes a comment button 140, a capture button 141, a game button 142, a video viewing button 143, and a gacha execution button 144. When the comment button 140 is operated, the streaming user device 12A displays an input form for entering comments. When the capture button 141 is operated, the streaming user device 12A captures the video. When the game button 142 is operated, the streaming user device 12A displays a list of games that can be played during video streaming. The game commentary screen is displayed in the video. When the video viewing button 143 is operated, the streaming user device 12A displays a screen for selecting videos provided by other video streaming services. When the gacha execution button 144 is operated, the streaming user device 12A displays a list of available lotteries (gacha).

[0106] The effects of the first embodiment will be described. (1-1) When a viewer performs a reference action, the user device 12 displays a simplified avatar object 80, which is the second avatar object, on the video including the avatar object 110, which is the first avatar object. This increases the opportunities for viewers to participate in the video distribution. In addition, since the simplified avatar object 80 has a smaller rendering processing load than the distribution user's avatar object 110, the processing load of the second output control processing that displays the simplified avatar object 80 can be reduced. Furthermore, since the message display unit 119 corresponding to the reference action is displayed together with the simplified avatar object 80, the actions performed by the viewer can be made more prominent.

[0107] (1-2) In addition to the avatar object 110 corresponding to the streaming user, guest avatar objects 62 that reflect the actions of co-streaming users and guest avatar objects 63 and 64 that do not reflect the actions of co-streaming users are displayed in the video. As a result, co-streaming users can participate in the video distribution by displaying their own avatar object 110 in the video. Furthermore, since the size of the simplified avatar object 80 on the screen is smaller than the avatar object 110 and guest avatar objects 62-64 corresponding to the streaming user, it is possible to avoid interfering with the effects of those avatar objects 110.

[0108] (1-3) Guest avatar objects 62-64 are displayed only if permitted by the streaming user. The simplified avatar object 80 can be displayed without permission from the streaming user, but it does not reflect the actions of the viewer. This reduces the complexity of the permission process for streaming users during video broadcasts. Additionally, by only allowing collaborations between users who are unlikely to engage in disruptive behavior, disruptive behavior by other users can be suppressed.

[0109] (1-4) The simplified avatar object 80 is displayed in front of the streaming user's avatar object 110 and other objects in a direction parallel to the depth direction of the screen. Therefore, since image processing such as depth comparison with other objects is unnecessary for the simplified avatar object 80, the load on the second output control processing that displays the simplified avatar object 80 can be reduced.

[0110] (1-5) The user device 12 displays the simplified avatar object 80 in association with the message display unit 119 corresponding to the standard action performed by the viewing user. Therefore, the broadcasting user and the viewing user can associate the action performed by the viewing user with the simplified avatar object 80.

[0111] (1-6) The user device 12 can display a simplified avatar object 80 when a viewing user performs a standard action, as well as when a broadcasting user performs a standard action.

[0112] (1-7) When a broadcasting user performs an operation to check the profile information of a viewing user or guest user, the broadcasting user device 12A displays the simplified avatar object 80 and the profile information 130. On the other hand, the viewing user device 12B displays the simplified avatar object 80 but does not display the profile information 130. This prevents the viewing user's profile information from being unintentionally made public to an unspecified number of other viewing users.

[0113] (Second Embodiment) Next, a second embodiment of the information processing system will be described. The second embodiment differs from the first embodiment in that the display mode for showing the simplified avatar object 80 is changed according to the number of viewing users. Hereinafter, parts that are the same as in the first embodiment will be denoted by the same reference numerals and their detailed descriptions will be omitted.

[0114] The control unit 30 of server 13 stores the number of viewing users in memory or elsewhere. When the control unit 30 receives a video viewing request from the viewing user device 12B, it updates the number of viewing users. The number of viewing users may be the cumulative number of viewing users, including users who stopped watching the video midway, or it may be the number of viewing users who are currently watching the video.

[0115] Server 13 switches between a first mode, which displays images of the simplified avatar objects 80 individually based on the actions of the viewing users, and a second mode, which displays multiple simplified avatar objects 80 simultaneously. The condition for switching from the first mode to the second mode is that the number of viewing users in the first mode exceeds the maximum number of users in the first mode.

[0116] Referring to Figure 14, the procedure for server 13 to switch modes will be explained. The control unit 30 of server 13 determines whether the number of viewing users has exceeded the maximum number (step S20). If the control unit 30 determines that the number of viewing users is less than or equal to the maximum number (step S20: NO), it proceeds to step S24 while maintaining the first mode.

[0117] When the control unit 30 determines that the number of viewing users has exceeded the maximum number (step S20: YES), it determines whether or not to switch modes (step S21). When the number of viewing users exceeds the maximum number, the control unit 30 sends a notification that the mode can be switched and causes the distribution user device 12A to output the notification. Then, when the server 13 receives a mode switching request from the distribution user device 12A that has received instructions from the distribution user, it determines to switch the mode to the second mode. When the server 13 receives a request to maintain the mode from the distribution user device 12A that has received instructions from the distribution user, it determines to maintain the mode in the first mode.

[0118] If the control unit 30 determines that it does not need to switch the mode to the second mode (step S21: NO), it maintains the first mode (step S23) and proceeds to step S24. When the control unit 30 determines that it is time to switch the mode to the second mode (step S21: YES), it switches the video display mode to the second mode (step S22). Specifically, the control unit 30 sends a mode switching request to the distribution user device 12A and the viewing user device 12B. When the viewing user device 12B receives the mode switching request or at a predetermined timing before that, it sends image data of the simplified avatar object 80 corresponding to each viewing user to the server 13. At this time, the viewing user device 12B may send the part ID of the simplified avatar object instead of the image data of the simplified avatar object 80.

[0119] In the second mode, the server 13 transmits the video configuration data sent from the distribution user device 12A to the viewing user device 12B. The server 13 also transmits the image data of the simplified avatar object 80 received from the viewing user device 12B to both the distribution user device 12A and the viewing user device 12B. Each user device 12 then displays the simplified avatar object 80 based on the image data received from the server 13.

[0120] In this case, the user device 12 may display the simplified avatar objects 80 for all viewing users. Alternatively, it may display the simplified avatar objects 80 for a predetermined number of viewing users. For example, if there are 50 or more viewing users, the simplified avatar objects 80 corresponding to 20 viewing users may be displayed. Furthermore, the simplified avatar objects 80 for all viewing users may be displayed in multiple stages. In this case, when the display of the simplified avatar objects 80 is changed, the newly displayed simplified avatar objects 80 may be displayed in a way that they march in from off-screen to on-screen in the order in which viewing began. In other words, the newly displayed simplified avatar objects 80 may be moved in one direction. Similarly, when the display of the simplified avatar objects 80 is changed, the exiting simplified avatar objects 80 may be displayed in a way that they march off-screen.

[0121] Furthermore, the user device 12 may reduce the rendering load of the simplified avatar object 80 in the second mode compared to the rendering load of the simplified avatar object 80 in the first mode. Alternatively, the server 13 may reduce the rendering load when transmitting image data of the simplified avatar object 80 to each user device 12. For example, the simplified avatar object 80 in the second mode may be displayed not as individual objects, but as image data that combines multiple simplified avatar objects 80.

[0122] Server 13 determines whether or not the video distribution has ended (step S24). If Server 13 determines that the video distribution has not ended (step S24: NO), the process returns to step S20. If Server 13 determines that the video distribution has ended (step S24: YES), this process ends.

[0123] Furthermore, if the number of viewers that triggers the switch from Mode 1 to Mode 2 is the number of viewers currently watching, the number of viewers may fall below the maximum number after switching to Mode 2. If the number of viewers falls below the maximum number after switching to Mode 2, the system may be configured to maintain Mode 2. Alternatively, if the number of viewers falls below the maximum number after switching to Mode 2, the system may be configured to revert to Mode 1.

[0124] Figure 15 shows the viewing screen 100 in the second mode. The viewing screen 100 includes the streaming user's avatar object 110 and simplified avatar objects 80 of multiple viewing users. The simplified avatar objects 80 may be lined up in front of the avatar object 110 located on the stage, as illustrated in Figure 15. Alternatively, the simplified avatar objects 80 may be lined up surrounding the avatar object 110.

[0125] When a viewer performs an operation on the viewer user device 12B to send a gift display request, the viewer user device 12B sends the gift ID, user ID, and image data of the simplified avatar to the server 13. The server 13 sends the gift ID, user ID, and image data of the simplified avatar object 80 to the user device 12. At this time, the server 13 displays the viewer user's simplified avatar object 80 on the user device 12 in a way that makes it stand out more than other simplified avatar objects 80. For example, the simplified avatar object 80 is displayed next to the broadcaster user's avatar object 110. If the avatar object 110 is located on the stage, the simplified avatar object 80 is also located on the stage.

[0126] According to the second embodiment, in addition to the effects described in (1-1) to (1-7) of the first embodiment, the following effects can be obtained. (2-1) When the number of viewers exceeds the maximum number, it is possible to display simplified avatar objects 80 for multiple viewers. This makes it possible to create the effect that viewers are participating in the live broadcast of the streamer.

[0127] (Third embodiment) Next, a third embodiment of the information processing system will be described. In the third embodiment, the display mode of the gift object 123 is changed based on whether or not the viewer who requested the display is watching a video, which is different from the first embodiment.Hereafter, parts that are the same as in the first embodiment will be denoted by the same reference numerals and their detailed descriptions will be omitted.

[0128] Referring to Figure 16, the procedure for the server 13 to send a request to display the gift object 123 to the user device 12 will be described below. In the following, the request to display the gift object 123 will be referred to as a gift display request.

[0129] Server 13 determines whether or not it has received a gift display request (step S31). At this time, Server 13 has received image data of the simple avatar object 80 at the same time as or before the gift display request. If it has not received a gift display request (step S31: NO), proceed to step S37. If Server 13 determines that it has received a gift display request (step S31: YES), it determines whether or not it is time to display the gift object 123 (step S32).

[0130] Here, the server 13 determines the timing for displaying the gift object 123 based on the attributes of the gift object 123. If the display timing is not particularly limited, such as with stuffed animals, the server 13 displays the gift object 123 on the viewing user device 12B upon receiving a gift display request. If it is a wearable object, the server 13 notifies the distribution user device 12A that a wearable object gift object 123 has been sent. In response, when the distribution user performs an operation to allow the display of the wearable object, the server 13 displays the gift object 123 on the viewing user device 12B. However, the distribution user allowing the display of a wearable object is just one example. The server 13 may also display the gift object 123 on the viewing user device 12B upon receiving a gift display request for a wearable object.

[0131] Furthermore, the display position of gift objects 123 may be predetermined. For example, gift objects 123 such as "wallpaper," "hanging scroll," and "tower cake" are displayed in predetermined display positions for a certain period of time, but new gift objects 123 cannot be displayed until the gift objects 123 already displayed in the display position are cleared. In such cases, the system determines that it is time to display the gift object 123 when the display position associated with the gift object 123 becomes available.

[0132] If server 13 determines that it is not time to display gift object 123 (step S32: NO), it waits for the timing to display gift object 123. When server 13 determines that it is time to display the gift object 123 (step S32: YES), it determines whether the viewer who sent the gift object 123 is currently watching the video (step S33).

[0133] When Server 13 determines that the viewer who sent the gift object 123 is currently watching the video (Step S33: YES), it displays the simplified avatar object 80 and the gift object 123 on the user device 12 (Step S34). Specifically, Server 13 transmits the gift ID and image data of the simplified avatar object 80 to the user device 12. When User Device 12 receives the image data of the gift ID and the simplified avatar object 80, it renders the gift object 123 corresponding to the gift ID and displays the image of the simplified avatar object 80. At this time, it is preferable to output the simplified avatar object 80 and the gift object 123 at the same time. Alternatively, it is preferable that the display period of the simplified avatar object 80 and the display period of the gift object 123 overlap.

[0134] On the other hand, if server 13 determines that the viewer who sent the gift object 123 is not currently watching the video (step S33: NO), it displays the gift object 123 without displaying the simple avatar object 80 (step S35). Specifically, server 13 sends the gift ID to user device 12. When user device 12 receives the gift ID, it draws the gift object 123 corresponding to the gift ID.

[0135] In other words, there may be a slight delay between when the viewing user device 12B sends a gift display request and when the gift object 123 is displayed. Also, depending on the attributes of the gift object 123, it may take some time for the gift object 123 to be displayed. Furthermore, between the time the viewing user device 12B sends the gift display request and the time the gift object 123 is displayed, the viewing user who sent the gift object 123 may end their viewing.

[0136] In this embodiment, as in the above embodiment, a simplified avatar object 80 corresponding to the viewer who sent the gift object 123 is displayed. However, if the viewer finishes watching the video before the gift object 123 is displayed, the viewer or the broadcaster may feel uncomfortable if the simplified avatar object 80 is displayed even though the viewer has finished watching the video. For this reason, the simplified avatar object 80 is not displayed if the viewer finishes watching the video before the gift object 123 is displayed.

[0137] In step S37, if server 13 does not terminate the delivery (step S37: NO), it returns to step S31 and repeatedly checks whether or not a gift display request has been received. If server 13 terminates the delivery (step S37: YES), the process terminates.

[0138] According to the third embodiment, in addition to the effects described in (1-1) to (1-7) of the first embodiment, the following effects can be obtained. (3-1) The user device 12 displays the gift object 123 and the simple avatar object 80 if the viewer who sent the gift display request is continuing to watch. If the viewer who sent the gift display request is not continuing to watch, the simple avatar object 80 is not displayed, and the gift object 123 is displayed. This prevents the broadcaster and other viewers from feeling uncomfortable because the viewer's simple avatar object 80 is displayed even though the viewer has left the stream.

[0139] Each of the above embodiments can be implemented with the following modifications. The above embodiments and the following modifications can be combined with each other to the extent that they do not contradict each other technically. <User> In each of the above embodiments, the broadcasting user means the user who transmits at least one of the video information and the audio information. For example, the broadcasting user is a user who performs solo video broadcasting or collaborative broadcasting that multiple people can participate in. Alternatively, the broadcasting user may be a user who hosts or organizes a video chat or voice chat that multiple people can participate in and at least one of view, or an event (such as a party) in a virtual space that multiple people can participate in and at least one of view. In other words, the broadcasting user can also be called a host user, organizer user, or event organizer user.

[0140] On the other hand, a viewing user refers to a user who receives at least one of the following: information related to video and information related to audio. However, a viewing user may also be a user who can react to the above information. For example, a viewing user is a user who watches video streams or collaborative streams. Alternatively, a viewing user may be a user who participates in and watches video chats, voice chats, or events. Therefore, a viewing user can also be referred to as a guest user, participating user, listener, spectator, or supporter.

[0141] <Collaboration Stream> In each of the above embodiments, one simplified avatar object 80 is displayed, but multiple simplified avatar objects 80 may be displayed. Even in this case, it is preferable to display the correspondence between the message display unit 119 of the reference action and the simplified avatar object 80 in a way that can be determined.

[0142] In each of the above embodiments, participation of co-starring users in the collaborative stream is enabled when the streaming user grants permission. Alternatively, or in addition to this, participation of co-starring users in the collaborative stream may be enabled even without permission from the streaming user. In this case, only the avatar objects of users whose status information meets predetermined conditions may be displayed as avatar objects 110 to which tracking data is applied. In addition, the avatar objects of users whose activity in the video does not meet predetermined conditions will be displayed as simplified avatar objects 80. Status information includes information indicating the user's activity in the video, or information about users who are directly or indirectly related to the video.

[0143] In collaborative streaming, the guest avatar objects 63 and 64 displayed at positions 72 and 73 on the rear left and rear right sides may be objects that require less rendering processing load than the host avatar object 61 and guest avatar object 62. For example, tracking data for the second and third guest users is not applied to the guest avatar objects 63 and 64.

[0144] Furthermore, the data used to render guest avatar objects 63 and 64 requires less processing load than the data used to render host avatar objects 61 and others. For example, the data used to render guest avatar objects 63 and 64 is data used to render two-dimensional objects. In other words, guest avatar objects 63 and 64 are rendered as two-dimensional objects.

[0145] The image data used to render the guest avatar objects 63 and 64 is generated by the viewing user device 12B that renders the video. Alternatively, it may be transmitted from the user device 12 used by the second and third guest users.

[0146] The rendering data used to draw guest avatar objects 63 and 64 may be data with a reduced polygon count. Reducing the polygon count can lower the image processing load. Alternatively, the rendering data used to draw guest avatar objects 63 and 64 may be data that integrates multiple parts, such as the main body part and attached parts that make up the avatar object. Reducing the number of parts can lower the image processing load compared to rendering each part individually.

[0147] Alternatively, in guest avatar objects 63 and 64, the components that perform physical behavior associated with the host avatar object 61 may be omitted. For example, even if guest avatar objects 63 and 64 overlap with other objects, collision detection does not need to be performed.

[0148] In addition to the distribution user device 12A, the user device 12 used by each guest user may also be configured to send audio data based on the guest user's speech to the server 13. In other words, the user device 12 that displays the video outputs the host user's voice and the voices of each guest user in synchronization with the video.

[0149] Alternatively, among the user devices 12 used by guest users, only the user device 12 used by the first guest user may be configured to send audio data to the server 13. The audio of the host user and the first guest user may be output, but the audio of the second and third guest users may not be output.

[0150] <Simple Avatar Object> In the above embodiment, the user device 12 rendered the simplified avatar object 80 as a two-dimensional object without applying the viewing user's tracking data. Alternatively, the user device 12 may render the simplified avatar object 80 as a two-dimensional object to which the viewing user's tracking data is applied. Alternatively, the simplified avatar object 80 may be rendered as a three-dimensional object without applying the viewing user's tracking data. Alternatively, the simplified avatar object 80 may be rendered as a three-dimensional object with a reduced polygon count. In the above embodiment, the viewing user's avatar object 110 was made two-dimensional, but the user device 12 may pre-store image data of the simplified avatar object 80 corresponding to the three-dimensional avatar object 110. This image data may be image data of the avatar object 110 that has been deformed (stylized). For example, it may be data relating to a simplified avatar object 80 with a smaller body proportion, where the head of the avatar object 110 has been enlarged.

[0151] In the first embodiment, actions by the viewer user such as following the streaming user, sharing the streaming video, capturing (screenshot) the streaming video, and ending the video viewing were not considered to be the standard actions. However, the simplified avatar object 80 may be displayed when any of these actions are performed.

[0152] The simplified avatar object 80 may change in response to the viewer's actions within the video. When the simplified avatar object 80 is displayed, the viewer may not know what triggered its display. This is especially true for smartphones and other devices with small screens, where the amount of information that can be displayed is limited. For this reason, a simplified avatar object 80 may be prepared for each viewer action that triggers its display. The simplified avatar object 80 may be a still image, or it may have predetermined motion data applied to it. For example, if the viewer selects the like button 121, a simplified avatar object 80 making a "good sign" may be displayed. If the viewer selects the gift button 122, a simplified avatar object 80 holding a gift box may be displayed. If the viewer posts a message, a simplified avatar object 80 holding an envelope may be displayed. Furthermore, while the distribution user device 12A is displaying profile information 130, the simplified avatar object 80 may be holding a plate displaying the viewer's account name or raising its hand. The user device 12 creates image data of multiple simplified avatar objects 80 corresponding to a reference action and stores it in the storage 22. In this configuration, when a simplified avatar object 80 is displayed, the distribution user and the viewing user can understand the trigger for the display of the simplified avatar object 80.

[0153] <Standard Action> In each of the above embodiments, the system for displaying the simplified avatar object 80 (second avatar object) is embodied in an information processing system in which a streaming user distributes a video using a streaming user device 12A, and a viewing user watches the video using a viewing user device 12B. Alternatively, the system for displaying the simplified avatar object 80 may be embodied in a system that outputs an image of the metaverse (virtual space) in which the avatar object corresponding to the user moves to the user device 12. In this metaverse, a second user may visit a small space in which the first user converses or performs. The small space is a room, a live venue, etc. In this embodiment, the simplified avatar object 80 may be displayed when the other user visits the small space. In this case, if the number of avatar objects of users gathered in the small space increases, the processing load on the user device 12 increases. Therefore, the processing load can be reduced by displaying the simplified avatar object 80.

[0154] In the first embodiment, the action of the streaming user posting a comment to the video may be used as the base action. When the comment button 140 specifies a viewer to whom the comment is to be sent, a simple avatar object 80 corresponding to that viewer may be displayed on the streaming screen and the viewing screen. Also, for example, when the streaming user selects a viewer ranking button (not shown) displayed on the streaming screen, the simple avatar objects 80 of the viewer users included in the viewer ranking may be displayed in order.

[0155] <Display mode for simple avatar objects> In the second embodiment, if the number of viewing users exceeds the maximum number in the first mode, the user device 12 renders the viewing user's simplified avatar object 80 in the second mode. Alternatively, the rendering mode may be selected according to the type of viewing user device 12B. For example, if the viewing user device 12B is a smartphone, the simplified avatar object 80 is rendered in the first mode. If the viewing user device 12B is a personal computer or a head-mounted display, the simplified avatar object 80 is rendered in the second mode. At this time, the viewing user device 12B may acquire its own spec information and determine whether the spec information allows for the rendering of the simplified avatar object 80 in the second mode. If the viewing user device 12B determines that it is possible to render the simplified avatar object 80 in the second mode, it may switch to the second mode.

[0156] In the second embodiment, the drawing mode may be switched according to the actions of the streaming user. Specifically, the streaming user device 12A determines whether the streaming user is singing or having a conversation. If the streaming user device 12A determines that the streaming user is singing, it sends a display request for the second mode to the viewing user device 12B via the server 13. Upon receiving the display request for the second mode, the viewing user device 12B draws the simplified avatar object 80 in the second mode using the image data of the viewing user's simplified avatar object 80 that it has received in advance. If the streaming user device 12A determines that the streaming user is having a conversation, it sends a display request for the first mode to the viewing user device 12B via the server 13. Upon receiving the display request for the first mode, the viewing user device 12B draws the simplified avatar object 80 in the first mode using the image data of the viewing user's simplified avatar object 80 that it has received in advance.

[0157] In each of the above embodiments, the participation of co-starring users in the collaborative stream is enabled when the streaming user grants permission. Alternatively, or in addition to this, participation of co-starring users in the collaborative stream may be enabled even without permission from the streaming user. In this case, only the avatar objects of users whose video activity meets the predetermined conditions may be displayed as avatar objects 110 to which tracking data is applied. In addition, the avatar objects of users whose video activity does not meet the predetermined conditions are displayed as simplified avatar objects 80.

[0158] <Prescribed conditions regarding viewing users> In each of the above embodiments, a simplified avatar object 80 is drawn based on the actions of the viewing user. In addition, the simplified avatar object 80 may be drawn when the viewing user's status information satisfies predetermined conditions related to the viewing user. Status information includes information indicating the user's activity with respect to the video, and information about the user that is directly or indirectly related to the video. Information indirectly related to the video is, for example, information that the server 13 can obtain from an external server. The server 13 obtains information indirectly related to the video using the API (Application Programming Interface) of the external server.

[0159] For example, status information is information indicating the friendly relationship between the broadcasting user and the viewing user. When a viewing user who follows the broadcasting user performs a specified action, the user device 12 may render a simplified avatar object 80 of that viewing user. In other words, it is sufficient that the broadcasting user's identification information is included in the follow list stored in the viewing user device 12B, etc. Status information may also be information about the group to which the viewing user belongs. The predetermined conditions may be that the user belongs to a fan club that supports the broadcasting user or to a group to which the broadcasting user belongs. Status information may also be information indicating the viewing user's activity with respect to the video. The predetermined condition may be that the user is ranked above a predetermined rank in the broadcasting user's support ranking. Alternatively, the condition for displaying the simplified avatar object 80 for a viewing user may be that they are watching the video for the first time. Alternatively, the condition for displaying the simplified avatar object 80 for a viewing user may be that the viewing user's cumulative viewing time is above a predetermined time, the number of views is above a predetermined number, or the cumulative viewing time of videos broadcast by a specific broadcaster is above a predetermined time, and the number of views is above a predetermined number. The cumulative viewing time and number of views may be the cumulative viewing time and number of views of videos broadcast by a specific broadcaster. Furthermore, the cumulative viewing time and number of views may be the sum of the cumulative viewing time and number of views for each video distributed by multiple broadcasters. Alternatively, the conditions for a viewer to display the simple avatar object 80 may be that the number of times they have sent a gift display request is greater than or equal to a predetermined number. Alternatively, the conditions for a viewer to display the simple avatar object 80 may be that the number of times they have posted a message is greater than or equal to a predetermined number. Alternatively, the conditions for a viewer to display the simple avatar object 80 may be that they have sent a gift object of a predetermined amount or more. Alternatively, the conditions for a viewer to display the simple avatar object 80 may be that the amount charged has reached a certain threshold. In addition, after a simple avatar object 80 has been displayed once, it may be decided not to display that simple avatar object 80 again. In this case, by restricting the viewer users who can display the simple avatar object 80, all simple avatar objects 80 that meet the display conditions can be displayed even on smartphones with limited display area.

[0160] In the above embodiment, the user device 12 may render simple avatar objects 80 corresponding to multiple viewers when multiple viewers send gifts to the broadcasting user. For example, viewer user device 12B is displaying a group chat screen where multiple viewers are chatting. The group chat is one of the following: text chat, voice chat, and video chat, or a combination thereof. One viewer user device 12B that makes a gift display request sends a gift ID and a group ID to the server 13. Instead of the group ID, the user IDs of the users belonging to the group may be sent. When the server 13 receives a gift display request, it requests image data of the simple avatar object 80 from the viewer user device 12B of the viewer user associated with the group ID. The viewer user device 12B of the viewer user belonging to the group sends the image data of the simple avatar object 80 to the server 13. The server 13 sends the received image data of the simple avatar object 80 to each user device 12. The user device 12 simultaneously renders the gift object corresponding to the gift ID, along with the simple avatar objects 80 of the viewer users belonging to the group. Alternatively, the user device 12 renders a gift object corresponding to the gift ID, along with a simplified avatar object 80 of a representative among the viewing users belonging to the group. The representative may be the viewing user who made the gift display request, or a viewing user predetermined within the group. If a representative is predetermined, the user ID of that user is recorded in the user device 12 or the server 13.

[0161] If the distribution end button is selected on the distribution user device 12A, image data may be generated showing the distribution user's avatar object 110 and the viewing users' simplified avatar objects 80 together. The user device 12 displays the distribution user's avatar object, as well as the simplified avatar objects 80 of all or some of the viewing users, in the video. The user device 12 also captures the screen and displays it on the display 28 as a group photo. Furthermore, the user device 12 stores the generated image data in the storage 22.

[0162] In each of the above embodiments, the user device 12 generates image data for the simplified avatar object 80 and sends it to the server 13. Alternatively, or in addition to this, the server 13 may generate image data. Or, the user device 12 may generate image data for the simplified avatar object 80 when generating video data. In this case, the part ID of the main body part and the part ID of the attached parts of each viewer's avatar object 110 are obtained in advance. The device that generates the image data for the simplified avatar object 80 can be changed depending on the video distribution method, etc.

[0163] (Information Processing System) The sensor unit 24 acquires detection data that detects changes in the user's facial expression, head movements, and the user's relative position to the sensor unit 24, but it may acquire at least one of these. In addition, the sensor unit 24 may acquire other detection data, such as detection data that detects movements other than head movements, in addition to or instead of at least one of the user's facial expression, head movements, and user's relative position.

[0164] The virtual space displayed in the video may be an augmented reality (AR) space. For example, animations of avatar objects and gift objects based on data transmitted from the distribution user device 12A may be superimposed on an image of the real world captured by the camera of the viewing user device 12B. Alternatively, the distribution user device 12A may generate a video by superimposing animations of avatar objects and gift objects onto an image of the real world captured by its own camera, encode the video, and send it to the server 13.

[0165] In each of the above embodiments, the user device 12 is an information processing device such as a smartphone, mobile phone, tablet terminal, personal computer, console game machine, or wearable computer such as a head-mounted display. Alternatively, the information processing system may be a system installed in a video distribution studio. The information processing system includes an information processing device, a server, a sensor unit 24 attached to the body of the distribution user, a tracking system for detecting the position of the sensor unit 24, an operation unit 27, a speaker 25, and a display 28. The tracking system may include a multi-axis laser emitter that emits pulsed laser light for synchronization. The sensor unit 24 includes a sensor that detects laser light and detects its own position and orientation while synchronizing with the synchronization pulse. For example, the sensor unit 24 can be the Vive Tracker® and Vive Base Station® provided by HTC Corporation®.

[0166] The control unit 20 of the user device 12 is configured to execute a first output control process, a second output control process, and a third output control process by executing an application program. At least one of these may be executed by the server 13 or other devices constituting the information processing system 11. In addition, the control unit 30 of the server 13 is configured to execute a first process, a second process, a third process, and a fourth process. At least one of these may be executed by the user device 12 or other devices constituting the information processing system 11. Next, the technical concepts that can be understood from the above embodiments and alternative examples are described below. [1] One or more computers, A first output control process outputs a video to the display that includes a first avatar object to which tracking data representing the actions of the first user has been applied, A program that, when an action taken by a second user on the video corresponds to a predetermined standard action, executes a second output control process which, in addition to the first avatar object, displays a second avatar object that has a lower rendering processing load than the first avatar object and corresponds to the second user, as well as information corresponding to the standard action, on the video for a display period shorter than the display period of the first avatar object. [2] The program described in [1], wherein the second output control process outputs the voice spoken by the first user and does not output the voice spoken by the second user. [3] The program according to [1], wherein the computer further causes the computer to perform a third output control process that outputs the video to the display, which includes the first avatar object and a third avatar object to which tracking data representing the movements of the co-starring user is applied when another user participates in the video as a co-starring user, and in the second output control process, the second avatar object is displayed which is smaller in size on the screen than the first avatar object and the third avatar object. [4] The program according to [1], wherein the first user performs an authorization operation on the first user device to allow another user to participate as a co-performer, and further causes the program to output the video to the display, which includes the first avatar object and a third avatar object to which tracking data representing the actions of the co-performer user has been applied, and in the second output control process, the program displays the second avatar object in the video, provided that the condition that the second user has performed the reference action is met. [5] The program according to [1], wherein the second output control process displays the second avatar object in association with a message display unit containing content corresponding to the reference action. [6] The program according to [1], wherein the second output control process further displays the second avatar object of the second user that is the target of the standard action in the video when the action taken by the first user on the video corresponds to the standard action. [7] The program according to [6], wherein the first user's reference action is that the first user device used by the first user sends a request to display the second user's profile information, the first user device displays the second user's profile information and the second avatar object, and the second user device used by the second user displays the second avatar object. [8] The program described in [1], wherein the second output control process displays the second avatar object only when the second user whose status information satisfies predetermined conditions has performed the criteria action. [9] The program according to [8], wherein the status information of the second user is information indicating a friendly relationship with the first user, and the predetermined condition is that the identification information of the first user is included in the second user's follow list.

[10] The program described in [8], wherein the status information of the second user is information relating to the group to which the second user belongs, and the predetermined condition is that the second user belongs to the group to which the first user belongs.

[11] The program according to [8], wherein the status information of the second user is information indicating activity with respect to the video, and the predetermined condition is that the number of views, which is the number of times the first user has watched the video distributed by the first user, is equal to or greater than a predetermined number, or the cumulative viewing time, which is the cumulative time the first user has watched the video, is equal to or greater than a predetermined time.

[12] The program described in [8], wherein the status information of the second user is information indicating activity with respect to the video, and the predetermined condition is that the number of times the first user has posted a message or sent a request to display a gift object with respect to the video distributed by the first user is greater than or equal to a predetermined number.

[13] The program described in [8], wherein the status information of the second user is information indicating activity with respect to the video, and the predetermined condition is that the charges for the video distributed by the first user have reached a threshold value.

[14] The computer, The program according to [1], further executing a process to determine whether the number of second users viewing the video exceeds a predetermined number, and in the second output control process, if the number of second users is less than or equal to the predetermined number, displaying multiple second avatar objects one by one, and if the number of second users exceeds the predetermined number, displaying multiple second avatar objects within the same period.

[15] The program according to [1], which causes the computer to further execute the following processes when the distribution of the video is terminated by the first user: display on the screen a state in which the second avatar object and the first avatar object are assembled; and store in the storage unit image data captured from the screen including the assembled state.

[16] The program according to [1], wherein in the second output control process, when displaying a gift object, if the second user device used by the second user who sent the request to display the gift object is in viewing mode, the second avatar object and the gift object are displayed, and if the second user device is not in viewing mode, the gift object is displayed without displaying the second avatar object.

[17] The reference action is the program described in any one of [1] to

[16] , wherein the second user device used by the second user has started watching the video.

[18] The program described in any one of [1] to

[16] , wherein the reference action is that the second user device used by the second user transmits information relating to the evaluation of the video.

[19] The program described in any one of [1] to

[16] , wherein the reference action is that the second user device used by the second user has posted a message to the video.

[20] The program according to any one of [1] to

[16] , wherein the reference action is that the second user device used by the second user sends a request to display a gift object for the video.

[21] One or more computers, An information processing method comprising: a first output control process that outputs a video to a display that includes a first avatar object to which tracking data representing the actions of a first user has been applied; and a second output control process that, when an action taken by a second user on the video corresponds to a predetermined standard action, displays in the video, in addition to the first avatar object, a second avatar object which has a lower rendering processing load than the first avatar object and corresponds to the second user, and information corresponding to the standard action, for a display period shorter than the display period of the first avatar object.

[22] Information processing device comprising: a first processing unit that causes a video including a first avatar object to which tracking data representing the actions of a first user has been applied to be output to the display of a second user device used by a second user based on a request to view the video from the second user device; a second processing unit that obtains from the second user device information corresponding to a predetermined standard action which is an action of the second user on the video; a third processing unit that obtains from the second user device data for drawing a second avatar object which has a smaller rendering processing load than the first avatar object and corresponds to the second user; and a fourth processing unit that, when the action of the second user on the video corresponds to the standard action, causes the second avatar object and information corresponding to the standard action to be displayed in the video in addition to the first avatar object for a display period shorter than the display period of the first avatar object.

[23] An information processing method comprising: a first process in which one or more computers cause a video including a first avatar object to which tracking data representing the actions of a first user is applied to be output to the display of a second user device used by a second user based on a request to view the video from the second user device; a second process in which the second user's actions on the video correspond to predetermined standard actions and are obtained from the second user device; a third process in which the second user device obtains data for drawing a second avatar object that has a lower rendering processing load than the first avatar object and corresponds to the second user; and a fourth process in which, if the second user's action on the video corresponds to the standard action, the second avatar object and the information corresponding to the standard action are displayed in the video in addition to the first avatar object for a display period shorter than the display period of the first avatar object. [Explanation of symbols]

[0167] 11… Information Processing Systems 12...User device 13… Server

Claims

1. On one or more computers, A first output control process that outputs a video containing a first avatar object corresponding to the first user to the display, In addition to the first avatar object, a second avatar object, which has a lower rendering processing load than the first avatar object and corresponds to a second user viewing the video, is displayed in the video as part of a second output control process, In the second output control process, A program that displays multiple second avatar objects within the same period of time when the number of second users exceeds a predetermined number.

2. In the second output control process described above, The program according to claim 1, which, when the number of second users is less than or equal to a predetermined number, displays the second avatar objects one by one in the video.

3. In the second output control process, The program according to claim 1, wherein the rendering load of the second avatar object displayed when the number of second users exceeds a predetermined number is reduced to less than the rendering load of the second avatar object displayed when the number of second users is less than or equal to the predetermined number.

4. The computer includes a process for displaying on the screen the state in which the second avatar object and the first avatar object are gathered together when the distribution of the video is terminated by the first user. The program according to claim 1, further comprising the process of storing image data captured from a screen including the aforementioned assembled state in a storage unit.

5. The program according to claim 1, wherein in the second output control process, when displaying a gift object, if the second user device used by the second user who sent the request to display the gift object is in viewing mode, the second avatar object and the gift object are displayed, and if the second user device is not in viewing mode, the gift object is displayed without displaying the second avatar object.

6. In the second output control process, The program according to any one of claims 1 to 5, which displays the second avatar object when the action taken by the second user on the video corresponds to a predetermined standard action.

7. The program according to claim 6, wherein in the second output control process, the second avatar object is displayed in association with a message display unit that includes content corresponding to the reference action.

8. In the second output control process described above, The program according to claim 1, which, when an action taken by the first user on the video corresponds to a standard action, displays the second avatar object of the second user that is the target of the standard action on the video.

9. The aforementioned standard action of the first user is that the first user device used by the first user has sent a request to display the second user's profile information. The program according to claim 8, wherein the first user device displays the profile information of the second user and the second avatar object, and the second user device used by the second user displays the second avatar object.

10. The program according to claim 6, wherein the second output control process displays the second avatar object only when the second user whose status information satisfies predetermined conditions has performed the standard action.

11. The program according to claim 10, wherein the status information of the second user is information indicating a friendly relationship with the first user, and the predetermined condition is that the identification information of the first user is included in the second user's follow list.

12. The program according to claim 10, wherein the status information of the second user is information relating to the group to which the second user belongs, and the predetermined condition is that the second user belongs to the group to which the first user belongs.

13. The program according to claim 10, wherein the status information of the second user is information indicating activity with respect to the video, and the predetermined condition is that the number of views, which is the number of times the first user has viewed the video distributed by the first user, is equal to or greater than a predetermined number, or the cumulative viewing time, which is the cumulative time the first user has viewed the video, is equal to or greater than a predetermined time.

14. The program according to claim 10, wherein the status information of the second user is information indicating activity with respect to the video, and the predetermined condition is that the number of times the first user has posted a message or sent a request to display a gift object with respect to the video distributed by the first user is greater than or equal to a predetermined number.

15. The program according to claim 10, wherein the status information of the second user is information indicating activity with respect to the video, and the predetermined condition is that the charges for the video distributed by the first user have reached a threshold value.

16. The program according to claim 6, wherein the reference action is that the second user device used by the second user starts watching the video.

17. The program according to claim 6, wherein the reference action is that the second user device used by the second user transmits information regarding the evaluation of the video.

18. The program according to claim 6, wherein the reference action is that a second user device used by the second user posts a message regarding the video.

19. The program according to claim 6, wherein the reference action is that the second user device used by the second user sends a request to display a gift object for the video.

20. In the second output control process, The program according to any one of claims 1 to 5, which, while displaying multiple second avatar objects, changes the position of the second avatar object corresponding to the second user who made a request to display a gift object.

21. The program according to any one of claims 1 to 5, wherein the second avatar object is an object to which at least one of the following is applied: that it is a two-dimensional object, that tracking data based on the actions of the second user is not applied, and that the number of polygons is reduced compared to the first avatar object.

22. One or more computers, A first output control process that outputs a video containing a first avatar object corresponding to the first user to the display, In addition to the first avatar object, a second avatar object, which has a lower rendering processing load than the first avatar object and corresponds to a second user viewing the video, is displayed in the video as part of a second output control process. In the second output control process, An information processing method for displaying multiple second avatar objects within the same period of time when the number of second users exceeds a predetermined number.

23. An information processing device comprising one or more computers, A process that causes a video containing a first avatar object corresponding to the first user to be output to the display of the second user device used by the second user, based on a video viewing request from the second user device. A process for obtaining data from the second user device for drawing a second avatar object, which has a lower rendering processing load than the first avatar object and corresponds to the second user who is watching the video, An information processing device that performs the process of displaying the second avatar object in the video, in addition to the first avatar object.

24. One or more computers, A process that causes a video containing a first avatar object corresponding to the first user to be output to the display of the second user device used by the second user, based on a video viewing request from the second user device. A process for obtaining data from the second user device for drawing a second avatar object, which has a lower rendering processing load than the first avatar object and corresponds to the second user who is watching the video, An information processing method that performs a fourth process of displaying the second avatar object in the video, in addition to the first avatar object.

Images