Program, information processing method, and information processing device

Abstract

To reduce a rendering processing load while allowing a second user viewing a video to have more opportunities to participate in video distribution.SOLUTION: A user device 12 executes: a first output control process that outputs to a display a video including a first avatar object to which tracking data showing movement of a distributing user is applied; and a second output control process that, when a second user is viewing the video and the second user's action regarding the video corresponds to a predetermined reference action, outputs to the display 28 a video including a second avatar object to which movement of the second user is not applied and the first avatar object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Conventionally, there has been known an information processing system that distributes moving images including animations of avatar objects based on the movements of a user (see, for example, Patent Document 1). In this system, moving images including two characters are generated in real time. Also, the voice of a voice actor is synthesized in real time with the moving images and distributed over a network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-184689 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, in an information processing system that distributes videos, consideration is being given to increasing opportunities for viewing users to participate in videos being distributed, with the aim of increasing the number of viewing users, or increasing the cumulative viewing time or number of views.

[0005] However, even when multiple users participate in the distribution of video as in the above system, the number of participants is limited, and this does not lead to an increase in opportunities for viewing users to participate. [Means for solving the problem]

[0006] In one aspect, a program executed by an information processing device is provided, the program causing one or more computers to execute a first output control process of outputting a video including a first avatar object to which tracking data representing a motion of a first user is applied, on a display, and a second output control process of displaying, in addition to the first avatar object, a second avatar object corresponding to the second user and requiring a smaller processing load for drawing than the first avatar object, and information corresponding to the reference action, on the video, within a display period shorter than a display period of the first avatar object, when an action of a second user on the video corresponds to a predetermined reference action.

[0007] In another aspect, an information processing method is provided, in which one or more computers execute a first output control process of outputting a video including a first avatar object to which tracking data representing a motion of a first user is applied, to a display, and a second output control process of displaying, in addition to the first avatar object, a second avatar object corresponding to the second user and requiring a smaller processing load for drawing than the first avatar object, and information corresponding to the reference action, in the video, within a display period shorter than a display period of the first avatar object, when an action of a second user on the video corresponds to a predetermined reference action.

[0008] In another aspect, an information processing device is provided, the information processing device comprising: a first processing unit that causes a video including a first avatar object to which tracking data representing a motion of a first user is applied to be output on a display of a second user device used by a second user based on a video viewing request from the second user device, a second processing unit that acquires from the second user device information corresponding to a predetermined reference action which is an action of the second user on the video, a third processing unit that acquires from the second user device data for drawing a second avatar object corresponding to the second user and which requires a smaller processing load for drawing than the first avatar object, and a fourth processing unit that causes the second avatar object and information corresponding to the reference action to be displayed on the video in addition to the first avatar object within a display period shorter than a display period of the first avatar object when the action of the second user on the video corresponds to the reference action.

[0009] In another aspect, there is provided an information processing method, the information processing method includes a first process in which one or more computers cause a second user device used by a second user to output a video including a first avatar object to which tracking data representing a motion of a first user is applied, on a display of the second user device based on a video viewing request from the second user device, a second process in which information corresponding to a predetermined reference action which is an action of the second user on the video, is obtained from the second user device, a third process in which data for drawing a second avatar object corresponding to the second user and which has a smaller processing load for drawing than the first avatar object is obtained from the second user device, and a fourth process in which, when the action of the second user on the video corresponds to the reference action, the second avatar object and information corresponding to the reference action are displayed on the video in addition to the first avatar object within a display period shorter than a display period of the first avatar object. Effect of the Invention

[0010] According to the present invention, it is possible to increase opportunities for viewers who watch videos to participate in video distribution while reducing the processing load of drawing. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram showing an information processing system according to a first embodiment. [Diagram 2] FIG. 11 is a diagram showing user management data in the embodiment. [Diagram 3] FIG. 2 is a diagram showing a configuration of main body part data according to the embodiment. [Figure 4] 13 is a diagram showing a configuration of mounting parts data in the embodiment; FIG. [Diagram 5] FIG. 13 is a diagram showing a viewing screen for collaboration distribution in the embodiment. [Figure 6] 13 is a flowchart showing a procedure for displaying a simple avatar object in the embodiment. [Figure 7] FIG. 2 is a diagram showing a home screen of the embodiment. [Figure 8] FIG. 2 is a diagram showing a viewing screen of the embodiment. [Figure 9] FIG. 2 is a diagram showing a viewing screen of the embodiment. [Figure 10] FIG. 2 is a diagram showing a viewing screen of the embodiment. [Figure 11] FIG. 2 is a diagram showing a viewing screen of the embodiment. [Figure 12] FIG. 2 is a diagram showing a viewing screen of the embodiment. [Figure 13] FIG. 13 is a diagram showing a distribution screen of the embodiment. [Figure 14] 10 is a flowchart showing a procedure for switching modes in the second embodiment. [Figure 15] FIG. 11 is a diagram showing a viewing screen in the second embodiment. [Figure 16] 13 is a flowchart showing a procedure for displaying a gift object according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (First embodiment) Hereinafter, a first embodiment of a program, an information processing method, and an information processing device will be described.

[0013] <Information processing system> An information processing system 11 will be described with reference to Fig. 1. The information processing system 11 includes a plurality of user devices 12 and a server 13. The user devices 12 and the server 13 transmit and receive data via a network 14. The user devices 12 receive data necessary for distribution and viewing of videos from the server 13 based on an input operation by a user.

[0014] A user who uses a user device 12 to distribute videos is called a distributor user, and a user who watches the distributed videos is called a viewing user. A user can be both a distributor user and a viewing user. That is, a user is a viewing user when he or she watches videos, and a distributor user when he or she distributes videos. Furthermore, a user device 12 used by a distributor user is called a distributor user device 12A, and a user device 12 used by a viewing user is called a viewing user device 12B. When there is no distinction between a distributor user and a viewing user, or between a distributor user device 12A and a viewing user device 12B, they are simply referred to as a user and a user device 12. The distributor user corresponds to the first user in the claims, and the viewing user corresponds to the second user in the claims. Furthermore, the distributor user device 12A corresponds to the first user device, and the viewing user device 12B corresponds to the second user device.

[0015] The user device 12 is a smartphone (multi-function phone terminal), a tablet terminal, a personal computer, a console game machine, a wearable computer, or an information processing device capable of playing videos other than these devices. The wearable computer may have a screen on which the user can watch videos, and may be, for example, a head-mounted display worn on the user's head, or a glasses-type wearable terminal. The user device 12 may be a device that operates independently, or may be composed of multiple devices connected to each other so that various data can be transmitted and received between them. An example of the latter user device 12 is, for example, a system that performs tracking using an outside-in method.

[0016] The user device 12 includes a control unit 20, a storage 22 (storage medium), and a communication interface (I / F) 23. The control unit 20 includes one or more arithmetic circuits, such as a central processing unit (CPU), a graphic processing unit (GPU), and a neural network processing unit (NPU). The control unit 20 also includes a memory, which is a main storage device (storage medium) that the arithmetic circuits can read and write. The memory is composed of a semiconductor memory or the like. The control unit 20 loads an operating system and other programs from the storage 22 or an external storage into the memory, and executes commands retrieved from the memory. The communication I / F 23 can transmit and receive data to and from the server 13 and other user devices 12 via the network 14. The network 14 includes various networks, such as a local area network and the Internet. The control unit 20 corresponds to a first output control unit and a second output control unit in the claims. The storage 22 corresponds to a memory unit in the claims.

[0017] The storage 22 is an auxiliary storage device (storage medium), and is, for example, a storage medium such as a magnetic disk, an optical disk, a semiconductor memory, etc. The storage 22 can also be used in combination with a plurality of storages.

[0018] The control unit 20 executes various processes related to video distribution and viewing by executing various programs recorded in the storage 22 based on the user's input operation on the operation unit 27. For ease of explanation, the state in which a video is distributed will be referred to as a distribution mode, and the state in which a video distributed by another distribution user is viewed will be referred to as a viewing mode.

[0019] The user device 12 also includes a sensor unit 24, a speaker 25, a microphone 26, an operation unit 27, and a display 28. At least one of the sensor unit 24, the speaker 25, the microphone 26, the operation unit 27, and the display 28 may be provided in the same device as the control unit 20, or may be provided in a manner connected to a device including the control unit 20.

[0020] The sensor unit 24 is one or more sensors that detect face motion indicating a change in the user's facial expression and body motion indicating a change in the relative position of the user's body with respect to the sensor unit 24. Face motion includes movements such as blinking and opening and closing of the mouth. The sensor unit 24 may be a publicly known sensor. An example of the sensor unit 24 includes a ToF sensor that measures the time of flight (Time of Flight) until light irradiated toward the user is reflected by the user's face and returns, a camera that photographs the user's face, and an image processing unit that processes the data photographed by the camera. The sensor unit 24 may also include an RGB camera that captures visible light and a near-infrared camera that captures near-infrared light. For the RGB camera or near-infrared camera, for example, True Depth of the "iPhoneX (registered trademark)", or "LIDAR" (Light Detection and Ranging, or Laser Imaging Detection and Ranging) of the "iPad Pro (registered trademark)", or other ToF sensors mounted on smartphones can be used. Specifically, this camera projects tens of thousands of invisible dots (points) onto the user's face or the like using a dot projector. Then, it detects and analyzes the reflected light of the dot pattern to form a depth map of the face, and captures an infrared image of the face or the like, thereby capturing accurate facial data. The arithmetic processing unit of the sensor unit 24 generates various information based on the depth map and the infrared image, and compares the information with registered reference data to calculate the depth of each point on the face (the distance between each point and the near-infrared camera) and positional deviations other than the depth.

[0021] The sensor unit 24 may have a function of tracking not only the user's face but also the hand (hand tracking). The sensor unit 24 may also include a sensor that detects the position or orientation of a human body part other than the hand. The sensor unit 24 may further include a sensor such as an acceleration sensor that detects speed or acceleration, or a gyro sensor that detects direction or orientation. The sensor unit 24 may have a spatial mapping function that recognizes an object in the real space where the user exists based on the detection result of the above-mentioned ToF sensor or other known sensors, and maps the recognized object on a spatial map. Hereinafter, when the face motion detection data and the body motion detection data are not particularly distinguished from each other, they are simply referred to as "tracking data". The image processing unit of the sensor unit 24 may be provided in the control unit 20.

[0022] The speaker 25 converts voice data into voice and outputs it. The microphone 26 inputs voice spoken by the user and converts it into voice data. The display 28 outputs various images in response to output instructions from the control unit 20.

[0023] The operation unit 27 may be one that corresponds to the type of the user device 12. An example of the operation unit 27 is a touch panel integrated with the display 28. Other examples of the operation unit 27 include operation buttons provided on the housing of the user device 12, a keyboard, a mouse, and a controller operated by the user's hand. The controller may incorporate various known sensors such as an acceleration sensor and an inertial measurement unit (IMU) such as a gyroscope. Another example of the operation unit 27 may be a tracking device that identifies the user's hand movement, eye movement, head movement, line of sight, and the like. In this embodiment, for example, the user's instruction is determined based on the user's hand movement, and various operations such as starting or ending the distribution of a video, rating a message or a video, and displaying a predetermined object (for example, a gift to be described later) can be performed. Note that, when the sensor unit 24 also has an input interface function such as a hand truck function, the operation unit 27 can be omitted.

[0024] <Server> Next, the server 13 will be described. The server 13 is used by a service provider or the like that provides a service for distributing videos. The server 13 includes a control unit 30, a communication I / F 34, and a storage 35. The control unit 30 includes one or more arithmetic circuits. The control unit 30 has the same configuration as the control unit 20 of the user device 12. The communication I / F 34 and the storage 35 have the same configuration as the user device 12. The server 13 may be configured as one device, or may be configured as multiple devices. When the server 13 is configured as multiple devices, these devices have the function of the control unit 30 and various data stored in the storage 35 in a distributed manner. The control unit 30 corresponds to the first processing unit, the second processing unit, the third processing unit, and the fourth processing unit in the claims.

[0025] <Video distribution method> The video delivery method may be one of a client rendering method, a browser rendering method, a video delivery method, and a server delivery method.

[0026] In the client rendering method, the broadcasting user device 12A and the viewing user device 12B receive video composing data required for generating a video from the server 13 and render the video. The video composing data includes tracking data detected by the sensor unit 24 of the broadcasting user device 12A. Each user device 12 also obtains audio data based on the broadcasting user's speech and outputs the audio in synchronization with the video. In this case, the storage 22 of each user device 12 stores application programs and various data such as rendering data for rendering avatars.

[0027] The browser rendering method is a method in which the distributor user device 12A and the viewer user device 12B render moving images using a web browser program stored in the storage 22. At this time, the distributor user device 12A transmits tracking data and the like to the server 13. The user device 12 downloads a web page written in a markup language such as HTML (Hyper Text Markup Language) and using CSS (Cascading Style Sheets), JavaScript (registered trademark), and the like from the server 13. The web browser program started on the user device 12 renders avatar objects and other objects using JavaScript running on the browser. In the browser rendering method, the user device 12 stores data received from the server 13 in a volatile memory, and stores data such as the URL of a web page for displaying moving images in a non-volatile memory. For this reason, the rendering data of the avatar object is not stored in the user device 12.

[0028] In the video distribution method, the broadcasting user device 12A generates video data. The viewing user device 12B displays video on the display 28 based on the video data received from the broadcasting user device 12A. Specifically, the broadcasting user device 12A generates video data using tracking data and the like. The broadcasting user device 12A also transmits the encoded video data and audio data to the server 13. The server 13 transmits the video data and audio data to the viewing user device 12B.

[0029] The server distribution method is a method in which the server 13 generates video data. The server 13 receives tracking data and audio data from the broadcasting user device 12A. The server 13 then generates video data based on these data. The server 13 transmits the generated video data to the broadcasting user device 12A and the viewing user device 12B. The broadcasting user device 12A and the viewing user device 12B output video to the display 28 based on the received video data.

[0030] In this way, either the user device 12 or the server 13 is responsible for generating the video data. The device responsible for generating the video data can be changed depending on the distribution method. When the user device 12 is responsible for generating the video data, the viewing user device 12B may generate the video data, or the distributor user device 12A may generate the video data. Furthermore, as necessary, the user device 12 transmits the generated video data to another user device 12 as a transmission destination.

[0031] The broadcaster user device 12A and the viewer user device 12B may display video in different formats. For example, the broadcaster user device 12A may display video in a client rendering format and the viewer user device 12B may display video in a browser rendering format.

[0032] In this embodiment, a method in which each user device 12 delivers video using a client rendering method will be described. In the client rendering method, the user device 12 stores a video application program in the storage 22. The storage 22 also stores three-dimensional model data of avatar objects and other objects used in executing the video application program.

[0033] The control unit 20 of the broadcasting user device 12A transmits to the server 13 moving image composing data including tracking data acquired from the sensor unit 24 and audio data based on audio collected by the microphone 26. The broadcasting user can set parts of the avatar object corresponding to the broadcasting user to preferred parts. The moving image composing data includes identification information (parts ID) indicating parts of the avatar object. The broadcasting user can also adjust the position of the virtual camera in the virtual space. The moving image composing data may include the position of the virtual camera. The tracking data and audio data are given a time stamp. The time stamp is generated based on reference time information based on a circuit (internal clock) having a clocking function of the control unit 20. Alternatively, the time stamp is generated based on time information transmitted from the server 13 or reference time information acquired from the server 13 providing a time distribution service. The time stamp may represent the elapsed time from a reference time, or may represent the time.

[0034] The viewing user device 12B receives in advance the part ID of the avatar object of the broadcasting user at a predetermined timing, such as the timing when viewing of a video is started. The viewing user device 12B applies the tracking data to the avatar object of the broadcasting user. The control unit 20 of the viewing user device 12B performs rendering including the avatar object and objects other than the avatar object. The rendering here refers to a drawing process including acquisition of the position of a virtual camera, perspective projection, hidden surface removal based on depth information associated with each object, and the like. The rendering may be at least one of these processes, or may include processes such as shading and texture mapping.

[0035] The control unit 20 receives data such as messages or notifications posted by viewing users from the server 13. The control unit 20 outputs video data that combines images of avatar objects and other objects with the posted messages or notifications to the display 28. The control unit 20 also outputs audio data from the speaker 25 in synchronization with the video data based on a time stamp.

[0036] Furthermore, the control unit 20 transmits a display request for a gift object (gift display request), posts a message for the video, and evaluates the video (selecting the "Like" button or the "favorite" button, etc.) based on the operation of the viewing user.

[0037] <Data structure used in video> Next, each piece of data used for distribution or viewing of a moving image will be described in detail with reference to FIG. 2 to FIG.

[0038] FIG. 2 shows an example of user management data 50. The user management data 50 is stored in the server 13. The user management data 50 includes data on a user ID, which is user identification information, coins, and points. Coins are a medium that a user can purchase. Points are a medium that can be obtained by actions such as watching videos and distributing videos. In addition, the user management data 50 may include a score that is accumulated by participating in an event. The coins, points, and scores can be used within the video distribution service. The user device 12 may store the user management data 50.

[0039] 3 shows an example of body part data 51 of an avatar object. The body part data 51 is stored in the storage 22 of the user device 12. The body parts are described below. The body parts are parts that constitute the body parts of the avatar object. The data for drawing the body parts includes polygon data, skeletal data (bones) for expressing the movements of the avatar object, texture data, and the like. In addition, the body parts are associated with avatar attributes. The avatar attributes are, for example, a type such as a male type such as "M type" or a female type such as "F type", height, skin color, body type, and the like. The skeletal data, polygon data, and the like of the avatar object differ depending on the avatar attributes. The user selects the avatar attributes to select body parts such as the upper body and the lower body. In addition, some of the body parts can be arbitrarily set by the user. For example, the user can select parts such as the eyes, mouth, and hairstyle. In this embodiment, each user can register three avatar objects as avatars that the user can use.

[0040] The body part data 51 includes information on the slot number, model, height, skin color, body type, and other body parts of the avatar. The body part data 51 has various information for each slot number of the avatar. The model is a model that is an avatar attribute selected by the user. The height, skin color, and body type are identification information of the height, skin color, and body type selected by the user. Before the user makes a selection, the height, skin color, and body type remain as initial settings. The user can change the settings of the height, skin color, and body type. The parts are IDs of individual parts such as eyes, mouth, and hairstyle. In addition, among the multiple attachment parts, there are attachment parts whose colors can be selected. For example, the colors "red", "blue", and "green" can be selected for the attachment part that is a "T-shirt". In addition, the initial settings of the body parts are applied before the user selects a body part.

[0041] FIG. 4 shows an example of the attached parts data 52. The attached parts will be explained. The attached parts are parts that are applied to a predetermined part of the avatar object. The attached parts are, for example, a "top" that is applied to the upper body, "shoes" that are applied to the feet, and a "skirt" that is applied to the lower body. The user can select a preferred attached part from a plurality of attached parts according to the avatar attributes of the avatar object. In other words, the viewing user can apply an attached part to each of the avatar objects with slot numbers "1" to "3". Note that before the user selects an attached part, the initial set attached part is applied.

[0042] The wearable parts are associated with attribute data indicating the body part of the avatar object. The wearable parts may be any parts associated with a predetermined body part of the avatar object, and may be a "cup", a "mug", an "umbrella", an "instrument" and the like in addition to clothing of the avatar object. For example, a "cup" is associated with the hand part of the avatar object. By applying a "cup" part to the hand, an avatar object holding a "cup" can be displayed. The wearable parts include standard parts stored when the application program is installed, and parts that the server 13 gives by the user by exchanging for coins, participating in an event, etc.

[0043] The attached part data 52 includes a part ID, an item name, a body part, an attribute, a color, and a slot number of the avatar object to which the attached part is applied. The part ID is identification information (ID) of the selected attached part. The item name is the name of the attached part. For example, the name of an item such as "Beginner T-shirt" is stored. The body part is the body part of the avatar object to which the attached part is applied. The attribute is the attribute of the avatar to which the attached part is applied, and is set to all attributes (for example, "All"), "M type", or "F type". The color is the color of the attached part, and indicates the color selected by the user. The applied slot number indicates the slot number of the avatar object to which the attached part is applied, among the avatar objects registered by the user.

[0044] The user device 12 may store non-attached parts data in addition to the attached parts data 52. Non-attached parts are parts other than attached parts that are applied to the virtual space. For example, non-attached parts are objects such as wallpaper that are displayed as a background. The non-attached parts data includes a part ID, an item name, and a color for each part. The non-attached parts may be stored in association with the avatar object to which they are applied.

[0045] <Collaboration distribution function> Next, the collaboration distribution function (co-starring function) will be described. In collaboration distribution, users other than the distribution user can display their corresponding avatar objects as co-stars in the video being distributed. Co-starring in collaboration distribution can only be performed in videos that have been set by the distribution user to allow collaboration distribution. In other words, users permitted by the distribution user can become co-stars. In addition to the distribution user, one or more co-star users up to a predetermined number can participate in collaboration distribution. The predetermined number is, for example, three people. The avatar object 110 of the distribution user corresponds to the first avatar object in the claims.

[0046] To participate in a collaboration distribution, the user device 12 transmits a participation request to the distributor user device 12A. When the distributor user approves the participation request, the distributor user becomes the host user in the collaboration distribution, and the user who made the participation request becomes the guest user. Then, a host avatar object corresponding to the host user and a guest avatar object corresponding to the guest user are displayed in the video.

[0047] 5 shows a viewing screen 60 of a collaboration broadcast. This viewing screen 60 is displayed on the display 28 of the broadcast user device 12B. Note that a substantially similar screen is also displayed on the broadcast user device 12A.

[0048] The viewing screen 60 includes an image of a host avatar object 61. The viewing screen 60 shown in Fig. 5 shows an example in which, in addition to the host avatar object 61, guest avatar objects 62 to 64 of three guest users are displayed. The guest avatar object 62 corresponds to the third avatar object.

[0049] The first guest avatar object 62 is an avatar object corresponding to the first guest user. The guest avatar object 62 is displayed at a position 71 to the right of a position 70 of the host avatar object 61. The host avatar object 61 and the guest avatar object 62 are rendered using three-dimensional model data. In addition, tracking data that detects the host user's movements is applied to the host avatar object 61. That is, the host avatar object 61 moves in accordance with the body movements of the host user. The guest avatar object 62 has tracking data that detects the first guest user's movements applied to it. That is, the guest avatar object 62 moves in accordance with the body movements of the first guest user. The guest avatar object 62 to which the tracking data is applied is called a second avatar object.

[0050] A guest avatar object 63 of a second guest user whose participation request is approved after the first guest user is displayed at position 72 on the left side behind the host avatar object 61 and the guest avatar object 62. A guest avatar object 64 of a third guest user whose participation request is approved after the second guest user is displayed at position 73 on the right side behind. The guest avatar objects 63 and 64 are rendered using three-dimensional model data. Tracking data that detects the movement of the second guest user is applied to the guest avatar object 63. Tracking data that detects the movement of the third guest user is applied to the guest avatar object 64.

[0051] The guest avatar object 62 exits by an operation of the first guest user. Alternatively, the guest avatar object 62 may automatically exit from the position 71 when a predetermined time has elapsed since the display of the guest avatar object 62 started. Then, the guest avatar object 63 of the second guest user is placed at the position 71. Moreover, the guest avatar object 64 of the third guest user is placed at the position 72. Furthermore, if there is a new guest avatar object waiting to appear together, the avatar object is displayed at the position 73. Note that, in the allowable frame for four people, if there are only the guest avatar objects 62, 63 of the two co-stars and no guest user is waiting, the guest avatar objects 62, 63 may continue to be displayed.

[0052] <Simple avatar object display function> Next, a display function of the avatar object 110 corresponding to the viewing user will be described. An avatar object 110 that requires a small processing load for drawing and corresponds to a viewing user will be referred to as a simple avatar object hereinafter. Image data of the simple avatar object is generated in advance by the control unit 20 of the user device 12. The simple avatar object is an avatar object of a slot number selected by the viewing user corresponding to the simple avatar object. The attached parts selected by the viewing user are applied to this avatar object. In other words, it is generated based on the main body parts and attached parts constituting the three-dimensional avatar object 110 used for video distribution. Moreover, the image data of the generated simple avatar object is stored in the storage 22 of the user device 12. Note that the image data of the simple avatar object may be generated by the server 13.

[0053] The simple avatar object is data with a reduced processing load for drawing. The processing load of the simple avatar object is reduced by using at least one of the following techniques: drawing the simple avatar object as a two-dimensional object, not applying tracking data, and reducing the number of polygons. To further reduce the processing load of the user device 12, the image data of the simple avatar object may be an integrated data of a plurality of parts. Also, components that perform physical behavior may be omitted from the simple avatar object. Also, the size of the simple avatar object on the screen is smaller than the size of the images of the host avatar object 61 and the guest avatar objects 62 to 64.

[0054] In this embodiment, the simplified avatar objects are rendered as two-dimensional objects and do not have tracking data applied to them. Furthermore, the display of the simplified avatar object does not require permission from the broadcasting user, and is displayed when the viewing user performs a predetermined reference action. In other words, if the broadcasting user performs an operation to allow or disallow the display of the simplified avatar object each time a reference action is performed while broadcasting a video, the broadcasting user's operations become cumbersome. For this reason, the simplified avatar object is displayed without the broadcasting user's permission. In addition, since tracking data is not applied to the simplified avatar object, it is unlikely that the viewing user will commit a nuisance in the video. For this reason, there is little need for the broadcasting user to give permission for the display of the simplified avatar object.

[0055] The simple avatar object is placed in an area for displaying a GUI (Graphical User Interface). Hereinafter, this area is referred to as a UI display area. The GUI placed in the UI display area is displayed in the foreground in the three-dimensional virtual space.

[0056] Since the simple avatar object is displayed in the UI display area, depth information (Z value, position in the screen depth direction) is not associated with the virtual camera set in the virtual space. The depth information indicates the distance from the virtual camera that displays the screen. An object located farther from the virtual camera has a larger depth. Therefore, when drawing the simple avatar object, it is not necessary to perform a depth test (hidden surface removal) in rendering with other three-dimensional objects. In the depth test, it is determined whether one object is in front of or behind the other object, and if it is behind the other object, the unhidden area of ​​the one object is drawn and the hidden area is not drawn. One method of the depth test is, for example, a Z test (Z-buffering). Although a depth test needs to be performed for the guest avatar objects 62 to 64, the simple avatar object does not need to be subjected to a depth test because the simple avatar object is not associated with depth information. Therefore, the processing load for drawing the simple avatar object can be reduced compared to the processing load for drawing the guest avatar objects 62 to 64.

[0057] The simple avatar object is displayed on the display 28 of the user device 12 when the viewing user performs one of the following standard actions. When you start watching a video.

[0058] When you rate a video. When you post a message. When you submit a gift viewing request.

[0059] When the viewing user device 12B accepts an instruction operation of the reference action by the viewing user, it transmits various data associated with the executed reference action and image data of the simplified avatar object to the server 13. At this time, instead of image data, part IDs of the body parts and attached parts for displaying the simplified avatar object may be transmitted. Furthermore, the viewing user device 12B may transmit image data or part IDs of the simplified avatar object to the server 13 in advance before accepting the instruction operation of the reference action.

[0060] The server 13 transmits data corresponding to the reference action and a display request for the data corresponding to the reference action to the broadcasting user device 12A and multiple viewing user devices 12B. The destination viewing user device 12B may or may not include the viewing user device 12B that transmitted the image data of the simplified avatar object, etc. If the destination viewing user device 12B does not include the viewing user device 12B that transmitted the image data of the simplified avatar object, etc., that viewing user device 12B displays a screen based on various data associated with the reference action and the image data of the simplified avatar object.

[0061] The broadcasting user device 12A and the viewing user device 12B receive various data associated with the executed reference action and image data of the simplified avatar object from the server 13. Then, the broadcasting user device 12A and the viewing user device 12B output to the display 28, in addition to the host avatar object 61, information corresponding to the simplified avatar object and the reference action.

[0062] The simple avatar object 80 is displayed in the video for a predetermined period of time. In other words, the period during which the simple avatar object 80 is displayed is shorter than the period during which the broadcasting user's avatar object 110 is displayed. Alternatively, the simple avatar object 80 may be erased from the video based on an operation of the broadcasting user or an operation of the viewing user corresponding to the simple avatar object 80.

[0063] (operation) Next, the operations of the user device 12 and the server 13 will be described. The user device 12 performs a first output control process, a second output control process, and a third output control process.

[0064] In the first output control process, control unit 20 outputs to display 28 a video including avatar object 110 to which tracking data representing the broadcast user's movements has been applied. In the second output control process, when an action of the viewing user on a video corresponds to a predetermined reference action, control unit 20 displays a simplified avatar object and information corresponding to the reference action on the video within a display period shorter than the display period of avatar object 110, in addition to avatar object 110. The simplified avatar object has a smaller processing load for drawing than the first avatar object and corresponds to the viewing user.

[0065] In the third output control process, when another user participates in the video as a co-star, the control unit 20 outputs a video including the avatar object 110 of the distribution user and a guest avatar object 62 representing the action of the co-star user to the display 28. The guest avatar object 62 of the co-star user has tracking data applied thereto that represents the action of the co-star user.

[0066] The server 13 performs a first process, a second process, a third process, and a fourth process. In the first process, the control unit 30 outputs a video including an avatar object 110 to which tracking data representing the movements of the distribution user has been applied, to the display 28 of the viewing user device 12B based on a video viewing request from the viewing user device 12B used by the viewing user.

[0067] In the second process, the control unit 30 acquires information corresponding to the reference action of the viewing user from the viewing user device 12B. In the third process, the control unit 30 acquires data for drawing a simplified avatar object corresponding to the viewing user, the simplified avatar object having a smaller processing load for drawing than the avatar object 110 of the broadcast user, from the viewing user device 12B.

[0068] In the fourth process, when the action of the viewing user on the video corresponds to the standard action, the control unit 30 displays in the video, in addition to the avatar object 110, a simplified avatar object and information corresponding to the standard action within a display period shorter than the display period of the avatar object 110.

[0069] Next, a procedure in which the viewing user device 12B transmits image data of a simple avatar object will be described in detail with reference to Fig. 6. It is assumed that the viewing user device 12B has started viewing a video selected by the viewing user.

[0070] The control unit 20 of the viewing user device 12B determines whether or not an action by the viewing user has been detected based on an input operation by the user to the operation unit 27 (step S10). If it is determined that no action has been detected (step S10: NO), the process proceeds to step S14.

[0071] On the other hand, when the control unit 20 detects an action of the viewing user (step S10: YES), it judges whether or not to display a simplified avatar object, which is an avatar object of the viewing user (step S11). Specifically, the control unit 20 judges whether or not the detected action corresponds to a standard action for displaying a simplified avatar object. A list indicating the standard actions is stored in advance in the storage 22 or the like.

[0072] When the control unit 20 determines not to display the simple avatar object (step S11: NO), it performs processing according to the action without transmitting image data of the simple avatar object (step S13). Actions that do not correspond to actions that display a simple avatar object include following a broadcast user, sharing a broadcast video, capturing a broadcast video (screenshot), and ending viewing of a video.

[0073] On the other hand, when the control unit 20 determines to display a simple avatar object (step S11: YES), it transmits image data of the simple avatar object to the server 13 (step S12). For example, when a viewing user gives a good rating to a video, the control unit 20 transmits to the server 13 a request to reflect the good rating in the video and image data of the simple avatar object. In addition, when a viewing user posts a message to the video, the control unit 20 transmits data including the content of the message and image data of the simple avatar object to the server 13. In addition, when a viewing user transmits a gift display request, the control unit 20 transmits to the server 13 the gift display request, the ID of the gift object selected by the viewing user, and image data of the simple avatar object.

[0074] In step S14, the control unit 20 judges whether the distribution of the moving image has ended. If it is judged that the distribution of the moving image has not ended (step S14: NO), the process returns to step S10. If it is judged that the distribution of the moving image has ended (step S14: YES), the process ends.

[0075] (Viewing screen) Next, screens that the user device 12 displays on the display 28 will be described with reference to FIGS.

[0076] The home screen 101 will be described with reference to FIG. 7. When the user starts the video application, the user device 12 receives a list of videos being distributed from the server 13. The user device 12 outputs the home screen 101 to the display 28 based on the list. The home screen 101 displays a video selection section 102 indicating videos that can be viewed. When the user selects the video selection section 102, the user device 12 transmits the ID of the selected video together with a viewing request to the server 13. The server 13 updates the number of viewing users and transmits video configuration data and audio data for rendering the video to the user device 12. This causes the user device 12 to enter a viewing mode.

[0077] Upon receiving the moving image composing data and the like from the server 13, the viewing user device 12B generates moving image data using the moving image composing data, and then outputs the viewing screen 100 to the display 28 based on the moving image data.

[0078] With some exceptions, the viewing screen 100 and the distribution screen displayed on the broadcast user device 12A have almost the same content of moving images including the avatar object 110. For this reason, the display mode of the simple avatar object 80 will be described using the viewing screen 100 as an example in Figures 8 to 12.

[0079] 8 shows an example of a viewing screen 100. The viewing screen 100 includes an image of an avatar object 110 corresponding to the broadcasting user. Tracking data based on the broadcasting user's movements is applied to the avatar object 110. Drawing data of worn parts 111 is applied to the avatar object 110. Non-worn parts 125, which are wallpaper, are applied to the virtual space. In addition, the viewing user device 12B outputs sound from the speaker 25 in synchronization with the animation of the avatar object 110.

[0080] Furthermore, the viewing screen 100 includes a viewer count 112 and a total heart count 113. The viewer count 112 is the cumulative number of viewers who have viewed the video. Alternatively, the viewer count 112 may be the number of viewers currently viewing the video. Furthermore, the total heart count 113 is the number of times viewers have given the video a positive rating since the video started.

[0081] The viewing screen 100 also includes a viewing ranking button 114, a broadcasting user ranking button 115, a share button 116, and a follow button 117. When the viewing user operates the viewing ranking button 114, the viewing user device 12B displays the ranking of viewing users who have viewed videos broadcast by that broadcasting user, based on the number of gifts sent. The broadcasting user ranking button 115 displays the ranking of broadcasting users, based on the number of gift objects received, among all broadcasting users who broadcast videos. For example, this ranking is calculated for all broadcasting users. The ranking is also a ranking of the number of gift objects received for each type of gift object, such as "bouquet," in descending order.

[0082] When the viewing user selects the share button 116, the viewing user device 12B transmits the link (URL) of the video on another social network service, copies the link, or reads the follow list of mutually following users, etc. When the viewing user selects the follow button 117, the server 13 associates this viewing user with the distributor user who distributes this video.

[0083] The viewing screen 100 also includes a message input section 118 and a message display section 119. The message input section 118 is an input form that enables a viewing user to send a message to a broadcasting user. The message display section 119 is a message sent by each viewing user device 12B, or a standard message sent by the server 13 based on the action of the viewing user. The message display sections 119 are displayed so that the most recent message is at the top (or bottom), and when a new message display section 119 is added, it moves downward (or upward), and is erased when it reaches the bottom (or top) of the displayable area of ​​the message display section 119.

[0084] The viewing screen 100 also includes a camera button 120, a good rating button 121, and a gift button 122. When the camera button 120 is selected, the viewing user device 12B captures the displayed viewing screen 100. The captured viewing screen 100 is stored in the storage 22. When the good rating button 121 is selected, the viewing user device 12B displays an indicator indicating a good rating, such as a heart mark, on the viewing screen 100. The viewing user device 12B also transmits a request to reflect the good rating to the server 13. When the server 13 receives the request to reflect the good rating, it updates the total number of hearts 113. The viewing user can display an indicator indicating a good rating on the video without paying any fee.

[0085] When the gift button 122 is selected, the viewing user device 12B displays a list of gifts superimposed on the viewing screen 100. The viewing user using the viewing user device 12B selects a gift from the list of gifts. The viewing user device 12B transmits the gift ID of the selected gift and a gift display request to the server 13. Furthermore, when the viewing user device 12B receives the gift ID and the gift display request from the server 13, it renders a gift object corresponding to the gift ID together with the avatar object 110. The viewing user pays a consideration to display the gift object in the video. The consideration that can be paid for the gift object is coins and points. As described above, coins are purchased by the user, and therefore the gift object to which coins are paid is a paid object. Points are acquired by the user based on the user's actions regarding the video. Therefore, the gift object to which points are paid is a non-refundable object.

[0086] As described above, the viewing user device 12B displays the simplified avatar object 80 when the action of the viewing user using the viewing user device 12B corresponds to the reference action. The viewing user device 12B performs a process of generating image data of the simplified avatar object 80 at the timing of starting a video application program, starting viewing of a video, or the like. Specifically, the viewing user device 12B acquires the part ID or image data of the avatar object 110 selected by the viewing user from among the avatar objects 110 having slot numbers "1" to "3". The attached parts 111 previously applied by the viewing user are applied to the image data of the avatar object 110 acquired at this time. In other words, since the viewing user's own simplified avatar object 80 is displayed in the video, the viewing user can devise ways to match the attached parts applied to the simplified avatar object 80 to the broadcasting user or to match the season in the real world.

[0087] The control unit 20 then reduces the weight of the image data. As described above, for example, the viewing user device 12B two-dimensionalizes the image data of the avatar object 110. Also, the simplified avatar object 80 is made smaller than the size of the avatar object 110. The viewing user device 12B generates image data of the simplified avatar object 80 and stores it in the storage 22. Then, when a viewing user using the viewing user device 12B performs a reference action, the image data of the simplified avatar object 80 is read from the storage 22 and transmitted to the server 13.

[0088] The generation process of the image data of the simplified avatar object 80 may be performed by the server 13. In this case, the server 13 receives the IDs of the body parts and the attached parts from the viewing user device 12B, and generates the image data of the simplified avatar object 80 based on these IDs. Alternatively, the generation process of the image data of the simplified avatar object 80 may be performed when each viewing user device 12B renders the viewing screen 100. In this case, the viewing user device 12B receives the IDs of the body parts and the attached parts from the viewing user device 12B of the viewing user who performed the action via the server 13, and generates the image data of the simplified avatar object 80 based on these IDs.

[0089] 8, a simplified avatar object 80 of a viewing user who has started viewing a video is displayed. When another viewing user has started viewing, the viewing user device 12B receives image data of the simplified avatar object 80 corresponding to the other viewing user from the server 13 and displays the simplified avatar object 80. When a viewing user using the viewing user device 12B starts viewing a video, the viewing user device 12B transmits image data of the simplified avatar object 80 to the server 13. The viewing user device 12B also displays an image of the simplified avatar object 80 read from the storage 22 on the display 28, or displays an image of the simplified avatar object 80 based on image data of the simplified avatar object 80 received from the server 13.

[0090] Since the display space of the viewing screen 100 is limited, it is difficult to lay out a large number of simple avatar objects 80. Therefore, the viewing user device 12B displays the simple avatar objects 80 one by one. At this time, the image of the simple avatar object 80 is displayed so that it can be determined that it is associated with a message display unit 119 such as "Entered the room." In the example of FIG. 8, the simple avatar object 80 is displayed outside the message display unit 119 and at the same position as the message display unit 119 in the vertical direction of the viewing screen 100.

[0091] For example, the viewing user device 12B displays the simple avatar object 80 based on the order in which the image data of the simple avatar object 80 is received from the server 13. The viewing user device 12B outputs the image of the simple avatar object 80 at the same timing as the message display unit 119 indicating that viewing of a video has started. The viewing user device 12B also displays the image of the simple avatar object 80 adjacent to the message display unit 119 for a certain period of time. The viewing user device 12B may erase the image of the simple avatar object 80 before the message display unit 119 is erased. In this way, it is possible to ensure a display space for other simple avatar objects 80 while making the message display unit 119 viewable. Alternatively, the image of the simple avatar object 80 may be erased simultaneously with the message display unit 119.

[0092] 9, a simplified avatar object 80 of a viewing user who has given a good rating to a video is displayed. When the viewing user device 12B receives a display request for an indicator 129 indicating a good rating and image data of the simplified avatar object 80 from the server 13, the viewing user device 12B displays images of the indicator 129 and the simplified avatar object 80 based on the received request. In addition, when a viewing user using the viewing user device 12B performs an operation to give a good rating to a video, the viewing user device 12B transmits the image data of the simplified avatar object 80 together with the display request for the indicator 129 to the server 13.

[0093] The viewing user device 12B displays the image of the simple avatar object 80 at the same timing as the message display unit 119 indicating that a favorable rating has been given. In addition, the viewing user device 12B displays the image of the simple avatar object 80 adjacent to the message display unit 119 indicating that a favorable rating has been given for a certain period of time.

[0094] 10, a simplified avatar object 80 of the viewing user who posted a message is displayed. When the viewing user device 12B receives a message posted by another viewing user and image data of the simplified avatar object 80 from the server 13, the viewing user device 12B displays the message display unit 119 and an image of the simplified avatar object 80 based on the message and image data. When a viewing user using the viewing user device 12B performs an operation to post a message, the viewing user device 12B transmits image data of the simplified avatar object 80 together with the message to the server 13.

[0095] The viewing user device 12B outputs an image of the simple avatar object 80 at the same timing as the message display unit 119 showing the message posted by the viewing user. In addition, the viewing user device 12B displays the image of the simple avatar object 80 for a certain period of time adjacent to the message display unit 119 showing the content of the posted message.

[0096] 11, a simplified avatar object 80 of the viewing user who has given the gift object is displayed. When a viewing user using the viewing user device 12B performs an operation to transmit a display request for the gift object 123, the viewing user device 12B transmits image data of the simplified avatar object 80 corresponding to the viewing user to the server 13 together with the gift ID. Furthermore, when the viewing user device 12B receives the gift ID and the image data of the simplified avatar object 80, the viewing user device 12B displays the gift object 123 and the simplified avatar object 80 based on these. The viewing user device 12B has received the image data of the gift object 123 from the server 13 in advance.

[0097] For example, it is preferable that the viewing user device 12B simultaneously displays the gift object 123 of the received gift ID, the simple avatar object 80, and the message display section 119. This makes it possible to distinguishably display the viewing user who has given the gift object 123.

[0098] The viewing user device 12B may erase the image of the simple avatar object 80 before erasing the message display unit 119. In this way, it is possible to ensure a display space for other simple avatar objects 80 while making the message display unit 119 viewable. Alternatively, the image of the simple avatar object 80 and the message display unit 119 are erased at the same time.

[0099] The viewing screen 100 shown in Fig. 12 illustrates a scene in which a profile of a viewing user is displayed by a broadcasting user. When the broadcasting user device 12A receives a confirmation operation by the broadcasting user to confirm the profile of the viewing user, it outputs the profile information of the viewing user to the display 28. The profile information of the viewing user is stored in the storage 22 by the viewing user device 12B. In addition, the viewing user device 12B transmits the profile information of the viewing user to the server 13 at a predetermined timing, such as when the viewing user starts watching a video or when a confirmation operation by the broadcasting user is received.

[0100] When a broadcasting user views a viewing user's profile, the server 13 requests image data of a simplified avatar object 80 from the viewing user device 12B used by that viewing user. Then, upon receiving the image data of the simplified avatar object 80, the server 13 transmits the image data of the simplified avatar object 80 to the broadcasting user device 12A and the viewing user device 12B.

[0101] The viewing user device 12B displays an image of the simple avatar object 80 based on the image data received from the server 13. At this time, the simple avatar object 80 may be displayed in such a way that it appears to be walking from the edge of the viewing screen 100 to a predetermined position.

[0102] FIG. 13 shows an example of a distribution screen 103 displayed by the broadcast user device 12A on the display 28. The distribution screen 103 includes an image of a simple avatar object 80, similar to the viewing screen 100. The broadcast user device 12A displays profile information 130 of the viewing user corresponding to the simple avatar object 80 in association with the simple avatar object 80. Note that this profile information 130 is not displayed on the viewing user device 12B that is viewing the video. This prevents the viewing user's profile from being made public to an unspecified number of other users against the viewing user's own intention. Furthermore, the viewing user can know that the broadcast user is checking the viewing user's own profile.

[0103] The distribution screen 103 includes the number of viewers 112, the total number of hearts 113, and the number of collaborators 131. The number of collaborators 131 indicates the number of collaborators participating in the collaboration distribution.

[0104] The distribution screen 103 also includes a share button 116, a confirm co-star button 137, a broadcast user ranking button 115, a settings button 138, and an audio button 139. By selecting the confirm co-star button 137 during collaboration distribution, the profile information 130 of the co-stars can be viewed. When checking the profile information 130 of the co-stars, the viewing user device 12B also displays the viewing screen 100 not including the profile information 130 as shown in FIG. 12. By selecting the settings button 138, distribution settings can be made. When the audio button 139 is selected, the broadcast user device 12A stops (mutes) the audio output or cancels the audio stop state.

[0105] The broadcast screen 103 also includes a comment button 140, a photo button 141, a game button 142, a video viewing button 143, and a gacha execution button 144. When the comment button 140 is operated, the broadcast user device 12A displays an input form for entering a comment. When the photo button 141 is operated, the broadcast user device 12A captures the video. When the game button 142 is operated, the broadcast user device 12A displays a list of games that can be played during the distribution of the video. A game commentary screen is displayed on the video. When the video viewing button 143 is operated, the broadcast user device 12A displays a screen for selecting a video provided by another video distribution service. When the gacha execution button 144 is operated, the broadcast user device 12A displays a list of available lotteries (gacha).

[0106] The effects of the first embodiment will be described. (1-1) When the viewing user performs a reference action, the user device 12 displays a simplified avatar object 80, which is a second avatar object, in a video including an avatar object 110, which is a first avatar object. This increases the opportunities for the viewing user to participate in the distribution of the video. In addition, since the processing load for drawing the simplified avatar object 80 is smaller than that for the avatar object 110 of the broadcasting user, the processing load of the second output control process for displaying the simplified avatar object 80 can be reduced. In addition, since a message display unit 119 corresponding to the reference action is displayed together with the simplified avatar object 80, the action performed by the viewing user can be made to stand out.

[0107] (1-2) In addition to the avatar object 110 corresponding to the broadcasting user, a guest avatar object 62 reflecting the actions of a co-starring user and guest avatar objects 63, 64 not reflecting the actions of a co-starring user are displayed in the video. This allows the co-starring users to participate in the distribution of the video through the display of their own avatar objects 110 in the video. Also, since the size of the simple avatar object 80 on the screen is smaller than the avatar object 110 corresponding to the broadcasting user and the guest avatar objects 62 to 64, it is possible to avoid impeding the presentation of those avatar objects 110.

[0108] (1-3) The guest avatar objects 62-64 are displayed when permission is given by the broadcasting user. Furthermore, the simple avatar object 80 can be displayed without permission from the broadcasting user, but does not reflect the actions of the viewing user. This makes it possible to prevent the broadcasting user from having to make permission operations while broadcasting a video from becoming cumbersome. Furthermore, by only allowing the appearance of users who are unlikely to commit nuisance behavior, it is possible to prevent nuisance behavior by other users.

[0109] (1-4) The simple avatar object 80 is displayed in front of the broadcast user's avatar object 110 and other objects in a direction parallel to the depth direction of the screen. Therefore, image processing such as comparing the depth of the simple avatar object 80 with other objects is not required for the simple avatar object 80, and the load of the second output control process for displaying the simple avatar object 80 can be reduced.

[0110] (1-5) The user device 12 displays the simple avatar object 80 in association with the message display unit 119 corresponding to the reference action performed by the viewing user. This allows the distributor user and the viewing user to associate the action performed by the viewing user with the simple avatar object 80.

[0111] (1-6) The user device 12 can display the simple avatar object 80 when the broadcasting user executes a reference action, in addition to when the viewing user performs a reference action.

[0112] (1-7) When the broadcasting user performs an operation to check the profile information of a viewing user or a guest user, the broadcasting user device 12A displays the simplified avatar object 80 and the profile information 130. On the other hand, the viewing user device 12B displays the simplified avatar object 80, but does not display the profile information 130. This makes it possible to prevent the profile information of the viewing user from being unintentionally made public to an unspecified number of other viewing users.

[0113] Second embodiment Next, a second embodiment of the information processing system will be described. The second embodiment differs from the first embodiment in that the display mode of the simple avatar object 80 is changed according to the number of viewing users. Hereinafter, the same parts as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0114] The control unit 30 of the server 13 stores the number of viewing users in a memory or the like. When the control unit 30 receives a request to view a video from the viewing user device 12B, the control unit 30 updates the number of viewing users. The number of viewing users may be the cumulative number of viewing users including viewing users who stopped viewing the video midway, or may be the number of viewing users who are currently viewing the video.

[0115] The server 13 switches between a first mode in which images of the simple avatar objects 80 are individually displayed based on the action of the viewing user, and a second mode in which a plurality of the simple avatar objects 80 are simultaneously displayed. The condition for switching from the first mode to the second mode is that the number of viewing users in the first mode exceeds the maximum number of users in the first mode.

[0116] A procedure for the server 13 to switch modes will be described with reference to Fig. 14. The control unit 30 of the server 13 judges whether the number of viewing users exceeds the maximum number (step S20). If the control unit 30 judges that the number of viewing users is equal to or less than the maximum number (step S20: NO), the control unit 30 maintains the first mode and proceeds to step S24.

[0117] When the control unit 30 determines that the number of viewing users exceeds the maximum number (step S20: YES), it determines whether to switch the mode (step S21). When the number of viewing users exceeds the maximum number, the control unit 30 transmits a notification that the mode can be switched, and causes the notification to be output to the broadcast user device 12A. Then, when the server 13 receives a mode switching request from the broadcast user device 12A that has accepted a command operation from the broadcast user, it determines to switch the mode to the second mode. When the server 13 receives a request to maintain the mode from the broadcast user device 12A that has accepted a command operation from the broadcast user, it determines to maintain the mode in the first mode.

[0118] When the control unit 30 determines not to switch the mode to the second mode (step S21: NO), the first mode is maintained (step S23), and the process proceeds to step S24. When the control unit 30 determines to switch the mode to the second mode (step S21: YES), it switches the display mode of the video to the second mode (step S22). Specifically, the control unit 30 transmits a mode switching request to the broadcasting user device 12A and the viewing user device 12B. Upon receiving the mode switching request or at a predetermined timing before that, the viewing user device 12B transmits image data of the simplified avatar object 80 corresponding to each viewing user to the server 13. At this time, the viewing user device 12B may transmit the part ID of the simplified avatar object instead of the image data of the simplified avatar object 80.

[0119] In the second mode, the server 13 transmits the moving image configuration data transmitted from the broadcasting user device 12A to the viewing user device 12B. The server 13 also transmits image data of the simplified avatar object 80 received from the viewing user device 12B to the broadcasting user device 12A and the viewing user device 12B. Each user device 12 also displays the simplified avatar object 80 based on the image data received from the server 13.

[0120] At this time, the user device 12 may display the simplified avatar objects 80 of all the viewing users. Also, among all the viewing users, a predetermined number of simplified avatar objects 80 may be displayed. For example, when there are 50 or more viewing users, the simplified avatar objects 80 of the viewing users corresponding to 20 viewing users may be displayed. Then, the simplified avatar objects 80 of all the viewing users may be displayed in a plurality of times. In this case, when the display of the simplified avatar objects 80 is replaced, the newly displayed multiple simplified avatar objects 80 may be displayed so as to march from outside the screen to enter the screen in the order in which the viewing started. In other words, the newly displayed multiple simplified avatar objects 80 may be moved in one direction. Similarly, when the display of the simplified avatar objects 80 is replaced, the exiting multiple simplified avatar objects 80 may be displayed so as to march outside the screen.

[0121] Furthermore, the user device 12 may set a load of a rendering process of the simple avatar object 80 in the second mode to be smaller than a load of a rendering process of the simple avatar object 80 in the first mode. Alternatively, the server 13 may transmit image data of the simple avatar object 80 to each user device 12 with a smaller load of the rendering process. For example, the simple avatar object 80 in the second mode may be displayed as image data of a plurality of simple avatar objects 80 compiled together, rather than as an individual object.

[0122] The server 13 judges whether the distribution of the moving image has ended (step S24). If the server 13 judges that the distribution of the moving image has not ended (step S24: NO), the process returns to step S20. If the server 13 judges that the distribution of the moving image has ended (step S24: YES), the process ends.

[0123] In addition, when the number of viewing users that is the condition for switching from the first mode to the second mode is the number of viewing users currently viewing, the number of viewing users may become equal to or less than the maximum number after transition to the second mode. If the number of viewing users becomes equal to or less than the maximum number after transition to the second mode, the second mode may be maintained. Alternatively, if the number of viewing users becomes equal to or less than the maximum number after transition to the second mode, the mode may be returned to the first mode.

[0124] Fig. 15 shows a viewing screen 100 in the second mode. The viewing screen 100 includes an avatar object 110 of the broadcasting user and simplified avatar objects 80 of a plurality of viewing users. The simplified avatar objects 80 may be lined up in front of the avatar object 110 located on the stage, as exemplified in Fig. 15. Alternatively, the simplified avatar objects 80 may be lined up so as to surround the avatar object 110.

[0125] When the viewing user performs an operation on the viewing user device 12B to transmit a gift display request, the viewing user device 12B transmits the gift ID, the user ID, and image data of the simple avatar to the server 13. The server 13 transmits the gift ID, the user ID, and image data of the simple avatar object 80 to the user device 12. At this time, the server 13 causes the user device 12 to display the simplified avatar object 80 of this viewing user so that it stands out more than other simplified avatar objects 80. For example, the server 13 causes the simplified avatar object 80 to be displayed next to the avatar object 110 of the broadcasting user. When the avatar object 110 is located on the stage, the simplified avatar object 80 is also located on the stage.

[0126] According to the second embodiment, in addition to the effects (1-1) to (1-7) of the first embodiment, the following effects can be obtained. (2-1) When the number of viewing users exceeds the maximum number, it is possible to display simple avatar objects 80 of multiple viewing users. This makes it possible to create an effect in which the viewing users feel as if they are participating in the live concert venue of the distribution user.

[0127] Third embodiment Next, a third embodiment of the information processing system will be described. The third embodiment differs from the first embodiment in that when the gift object 123 is displayed, the display mode of the gift object 123 is changed based on whether the viewing user who made the display request is viewing a video. Hereinafter, the same reference numerals are used for the same parts as those in the first embodiment, and detailed description thereof will be omitted.

[0128] 16, a procedure will be described in which the server 13 transmits a display request for the gift object 123 to the user device 12. In the following, the display request for the gift object 123 is referred to as a gift display request.

[0129] The server 13 judges whether or not a gift display request has been received (step S31). At this time, the server 13 has received image data of the simple avatar object 80 at the same time as or before the gift display request. If the gift display request has not been received (step S31: NO), the server 13 proceeds to step S37. If the server 13 judges that a gift display request has been received (step S31: YES), the server 13 judges whether or not it is time to display the gift object 123 (step S32).

[0130] Here, the server 13 determines the timing to display the gift object 123 according to the attributes of the gift object 123. In the case of a stuffed toy or the like, where the display timing is not particularly limited, the server 13 causes the viewing user device 12B to display the gift object 123 upon receipt of a gift display request. In the case of a wearable object, the server 13 notifies the viewing user device 12A that the gift object 123, which is a wearable object, has been given. In response to this, when the broadcasting user performs an operation to permit the display of the wearable object, the server 13 causes the viewing user device 12B to display the gift object 123. However, the case where the broadcasting user permits the display of the wearable object is just one example. The server 13 may cause the viewing user device 12B to display the gift object 123 upon receipt of a gift display request for a wearable object.

[0131] Also, there are cases where the display position of the gift object 123 is fixed. For example, gift objects 123 such as "wallpaper", "hanging scroll", and "tower cake" are displayed for a certain period of time at a predetermined display position, but a new gift object 123 cannot be displayed unless a gift object 123 already displayed at a display position is erased. In such a case, when a display position associated with the gift object 123 becomes vacant, it is determined that it is the timing to display the gift object 123.

[0132] When the server 13 determines that it is not time to display the gift object 123 (step S32: NO), the server 13 waits for the timing to display the gift object 123. When the server 13 determines that it is time to display the gift object 123 (step S32: YES), it determines whether the viewing user who has given the gift object 123 is currently viewing a video (step S33).

[0133] When the server 13 determines that the viewing user who has given the gift object 123 is currently viewing a video (step S33: YES), the server 13 displays the simplified avatar object 80 and the gift object 123 on the user device 12 (step S34). Specifically, the server 13 transmits image data of the gift ID and the simplified avatar object 80 to the user device 12. When the user device 12 receives the image data of the gift ID and the simplified avatar object 80, it draws the gift object 123 corresponding to the gift ID and displays the image of the simplified avatar object 80. At this time, it is preferable that the simplified avatar object 80 and the gift object 123 are output at the same timing. Alternatively, it is preferable that the display period of the simplified avatar object 80 and the display period of the gift object 123 overlap.

[0134] On the other hand, when the server 13 determines that the viewing user who has given the gift object 123 is not currently viewing a video (step S33: NO), the server 13 displays the gift object 123 without displaying the simple avatar object 80 (step S35). Specifically, the server 13 transmits a gift ID to the user device 12. Upon receiving the gift ID, the user device 12 renders the gift object 123 corresponding to the gift ID.

[0135] That is, there may be some lag between the timing when the viewing user device 12B transmits the gift display request and the timing when the gift object 123 is displayed. Also, it may take some time for the gift object 123 to be displayed depending on the attribute of the gift object 123. And, there is a possibility that the viewing user who gave the gift object 123 will finish viewing during the period from when the viewing user device 12B transmits the gift display request to when the gift object 123 is displayed.

[0136] In this embodiment, as in the above embodiment, a simplified avatar object 80 corresponding to a viewing user who has given a gift object 123 is displayed. However, if the viewing user finishes watching the video before the gift object 123 is displayed, the viewing user or the distributor user may feel uncomfortable if the simplified avatar object 80 is displayed even though the viewing user has finished watching the video. For this reason, if the viewing user finishes watching the video before the gift object 123 is displayed, the simplified avatar object 80 is not displayed.

[0137] In step S37, if the server 13 does not end the distribution (step S37: NO), the process returns to step S31 and repeatedly determines whether or not the gift display request has been received. If the server 13 has ended the distribution (step S37: YES), the process ends.

[0138] According to the third embodiment, in addition to the effects (1-1) to (1-7) of the first embodiment, the following effects can be obtained. (3-1) When the viewing user who transmitted the gift display request continues viewing, the user device 12 displays the gift object 123 and the simplified avatar object 80. Also, when the viewing user who transmitted the gift display request does not continue viewing, the user device 12 does not display the simplified avatar object 80, but displays the gift object 123. This makes it possible to prevent the broadcasting user and other viewing users from feeling uncomfortable when the viewing user's simplified avatar object 80 is displayed even though the viewing user has left the content.

[0139] The above-described embodiments may be modified as follows: The above-described embodiments and the following modifications may be combined with each other to the extent that no technical contradiction occurs. <user> In each of the above embodiments, a distribution user refers to a user who transmits at least one of information related to video and information related to audio. For example, a distribution user is a user who performs independent video distribution or collaborative distribution in which multiple people can participate. Alternatively, a distribution user may be a user who hosts or organizes a video chat or voice chat in which multiple people can participate and / or watch, or an event (such as a party) in a virtual space in which multiple people can participate and / or watch. In other words, a distribution user can also be called a host user, organizer user, or organizer user.

[0140] On the other hand, a viewing user means a user who receives at least one of information related to video and information related to audio. However, a viewing user can be a user who is capable of not only receiving the above information but also reacting to it. For example, a viewing user is a user who watches video distribution or collaboration distribution. Or, a viewing user may be a user who participates in and / or watches video chat, voice chat, or an event. Therefore, a viewing user can also be referred to as a guest user, a participating user, a listener, a viewing user, a supporting user, etc.

[0141] <Collaboration distribution> In the above embodiments, the simple avatar objects 80 are displayed one by one, but it is also possible to display a plurality of simple avatar objects 80. Even in this case, it is preferable to display the message display area 119 of the standard action and the simple avatar objects 80 so that the correspondence between them can be distinguished.

[0142] In each of the above embodiments, when the distribution user allows participation, the co-starring user can participate in the collaboration distribution. Alternatively or in addition to this, the co-starring user may be allowed to participate in the collaboration distribution even if the distribution user does not allow it. In this case, only the avatar objects of users whose status information of the viewing user satisfies a predetermined condition may be displayed as avatar objects 110 to which tracking data is applied. Also, the avatar objects of users whose activities in the video do not satisfy a predetermined condition are displayed as simplified avatar objects 80. The status information is information indicating the user's activities in relation to the video, or information of users directly or indirectly related to the video.

[0143] In collaborative distribution, the guest avatar objects 63, 64 displayed at the rear left position and the rear right position 72, 73 may be objects that impose a smaller processing load on drawing than the host avatar object 61 and the guest avatar object 62. For example, the tracking data of the second guest user and the third guest user is not applied to the guest avatar objects 63, 64.

[0144] Furthermore, the data for drawing the guest avatar objects 63 and 64 imposes a smaller processing load on the drawing than the data for drawing the host avatar objects 61, etc. For example, the data for drawing the guest avatar objects 63 and 64 is data for drawing two-dimensional objects. In other words, the guest avatar objects 63 and 64 are drawn as two-dimensional objects.

[0145] The image data for rendering the guest avatar objects 63 and 64 may be generated by the viewing user device 12B that renders the video, or may be transmitted from the user device 12 used by the second guest user and the third guest user.

[0146] The drawing data for drawing the guest avatar objects 63, 64 may be data with a reduced number of polygons. By reducing the number of polygons, the load on image processing can be reduced. Alternatively, the drawing data for drawing the guest avatar objects 63, 64 may be data that integrates multiple parts, such as main body parts and attached parts, that make up the avatar object. By reducing the number of parts, the load on image processing can be reduced compared to drawing each part individually.

[0147] Alternatively, the guest avatar objects 63, 64 may omit a component that performs a physical behavior associated with the host avatar object 61. For example, even if the guest avatar objects 63, 64 overlap with another object, collision detection may not be performed.

[0148] In addition to the broadcasting user device 12A, the user device 12 used by each guest user may transmit voice data based on the speech of the guest user to the server 13. In other words, the user device 12 displaying the video outputs the voice of the host user and the voice of each guest user in synchronization with the video.

[0149] Alternatively, among the user devices 12 used by the guest users, only the user device 12 used by the first guest user may transmit voice data to the server 13. It is also possible to output the voices of the host user and the first guest user, but not output the voices of the second guest user and the third guest user.

[0150] <Simple avatar object> In the above embodiment, the user device 12 renders the simple avatar object 80 as a two-dimensional object without applying the tracking data of the viewing user. Alternatively, the user device 12 may render the simple avatar object 80 as a two-dimensional object to which the tracking data of the viewing user is applied. The simple avatar object 80 may be rendered as a three-dimensional object without applying the tracking data of the viewing user. The simple avatar object 80 may be rendered as a three-dimensional object with a reduced number of polygons. In the above embodiment, the avatar object 110 of the viewing user is two-dimensionalized, but the user device 12 may store image data of the simple avatar object 80 corresponding to the avatar object 110 of the three-dimensional object in advance. This image data may be image data obtained by deforming the avatar object 110. For example, the data may be data regarding a low-life-size simple avatar object 80 with an enlarged head of the avatar object 110.

[0151] In the first embodiment, actions of a viewing user, such as following a broadcasting user, sharing a broadcast video, capturing (taking a screenshot) a broadcast video, and ending viewing of a video, are not considered to be standard actions. However, when any of these actions is performed, the simplified avatar object 80 may be displayed.

[0152] The simple avatar object 80 may be changed according to the action of the viewing user in the video. When the simple avatar object 80 is displayed, the user watching the video may not know the trigger for the simple avatar object 80 to be displayed. In particular, in a smartphone or the like with a small screen size, the amount of information that can be displayed is limited. For this reason, a simple avatar object 80 may be prepared for each action of the viewing user that triggers the display of the simple avatar object 80. In addition, the simple avatar object 80 may be a still image, or predetermined motion data may be applied. For example, when the viewing user selects the good evaluation button 121, the simple avatar object 80 making a "good sign" may be displayed. In addition, when the viewing user selects the gift button 122, the simple avatar object 80 holding a gift box may be displayed. In addition, when the viewing user posts a message, the simple avatar object 80 holding an envelope may be displayed. In addition, while the broadcasting user device 12A is displaying the profile information 130, the simple avatar object 80 may hold a plate displaying the viewing user's account name or may raise its hand. The user device 12 creates image data of a plurality of simple avatar objects 80 corresponding to the reference action and stores the image data in the storage 22. According to this aspect, when the simple avatar object 80 is displayed, the broadcasting user and the viewing user can understand the trigger for displaying the simple avatar object 80.

[0153] <Standard Action> In the above embodiments, the system for displaying the simplified avatar object 80 (second avatar object) is embodied in an information processing system in which a broadcasting user broadcasts a video using a broadcasting user device 12A, and a viewing user watches the video using a viewing user device 12B. Alternatively, the system for displaying the simplified avatar object 80 may be embodied in a system for outputting an image of a metaverse (virtual space) in which an avatar object corresponding to a user moves to a user device 12. In this metaverse, a second user may visit a small space in which a first user has a conversation or a performance. The small space may be a room, a live venue, or the like. In this aspect, the simplified avatar object 80 may be displayed when the other user visits the small space. In this case, if the number of avatar objects of users gathered in the small space increases, the processing load of the user device 12 increases. Therefore, the processing load can be reduced by displaying the simplified avatar object 80.

[0154] In the first embodiment, the reference action may be an action in which the distributor user posts a comment on a video. When a viewing user who is the destination of a comment is specified with comment button 140, a simplified avatar object 80 corresponding to that viewing user may be displayed on the distribution screen and the viewing screen. Also, for example, when a viewing ranking button (not shown) displayed on the distribution screen is selected by the distributor user, simplified avatar objects 80 of viewing users included in the viewing ranking may be displayed in order.

[0155] <Simple avatar object display mode> In the second embodiment, when the number of viewing users exceeds the maximum number in the first mode, the user device 12 draws the simplified avatar objects 80 of the viewing users in the second mode. Alternatively, the drawing mode may be selected according to the type of the viewing user device 12B. For example, when the viewing user device 12B is a smartphone, the simplified avatar objects 80 are drawn in the first mode. When the viewing user device 12B is a personal computer or a head-mounted display, the simplified avatar objects 80 are drawn in the second mode. At this time, the viewing user device 12B may acquire its own spec information and determine whether or not the spec information is capable of drawing the simplified avatar objects 80 in the second mode. Then, when the viewing user device 12B determines that the simplified avatar objects 80 are capable of drawing in the second mode, the viewing user device 12B may transition to the second mode.

[0156] In the second embodiment, the drawing mode may be switched depending on the action of the broadcast user. Specifically, the broadcast user device 12A judges whether the broadcast user is singing or talking. When the broadcast user device 12A judges that the broadcast user is singing, it transmits a display request in the second mode to the viewing user device 12B via the server 13. When the display request in the second mode is received, the viewing user device 12B draws the simplified avatar object 80 in the second mode using image data of the simplified avatar object 80 of the viewing user received in advance. When the broadcast user device 12A judges that the broadcast user is talking, it transmits a display request in the first mode to the viewing user device 12B via the server 13. When the display request in the first mode is received, the viewing user device 12B draws the simplified avatar object 80 in the first mode using image data of the simplified avatar object 80 of the viewing user received in advance.

[0157] In each of the above embodiments, co-star users are allowed to participate in a collaboration broadcast when the broadcasting user allows participation. Alternatively or in addition to this, co-star users may be allowed to participate in a collaboration broadcast even if the broadcasting user does not allow it. In this case, only the avatar objects of users whose activities in the video satisfy a predetermined condition may be displayed as avatar objects 110 to which tracking data is applied. Also, avatar objects of users whose activities in the video do not satisfy the predetermined condition are displayed as simplified avatar objects 80.

[0158] <Prescribed conditions for viewing users> In each of the above embodiments, the simple avatar object 80 is drawn based on the action of the viewing user. In addition, the simple avatar object 80 may be drawn when the status information of the viewing user satisfies a predetermined condition related to the viewing user. The status information is information indicating the user's activity with respect to the video, or information of a user directly or indirectly related to the video. Information indirectly related to the video is, for example, information that the server 13 can acquire from an external server. The server 13 acquires the information indirectly related to the video using an API (Application Programming Interface) of the external server.

[0159] For example, the status information is information indicating a friendly relationship between a broadcast user and a viewing user. When a viewing user who follows a broadcast user performs a reference action, the user device 12 may draw a simplified avatar object 80 of the viewing user. That is, it is sufficient that the identification information of the broadcast user is included in the follow list stored in the viewing user device 12B or the like. The status information may also be information regarding a group to which the viewing user belongs. The predetermined condition may be that the viewing user belongs to a fan club that supports the broadcast user or to a group to which the broadcast user belongs. The status information may also be information indicating the viewing user's activity with respect to a video. The predetermined condition may be that the viewing user is ranked at or above a predetermined rank in the support ranking of the broadcast user. Alternatively, the condition for a viewing user who displays the simplified avatar object 80 is that the viewing user has viewed a video for the first time. Alternatively, the condition for a viewing user who displays the simplified avatar object 80 is that the viewing user's cumulative viewing time is a predetermined time or more, the number of views is a predetermined number or more, and the cumulative viewing time of videos distributed by a specific broadcaster is a predetermined time or more, and the number of views is a predetermined number or more. The cumulative viewing time and the number of views may be the cumulative viewing time and the number of views of videos distributed by a specific broadcaster. The cumulative viewing time and the number of views may be cumulative viewing time and the number of views obtained by adding up the cumulative viewing time and the number of views for each video distributed by a plurality of distributors. Alternatively, the condition for a viewing user who displays the simplified avatar object 80 is that the number of times a gift display request has been transmitted is a predetermined number or more. Alternatively, the condition for a viewing user who displays the simplified avatar object 80 is that the number of times a message has been posted is a predetermined number or more. Alternatively, the condition for a viewing user who displays the simplified avatar object 80 is that the viewing user has given a gift object of a predetermined amount or more. Alternatively, the condition for a viewing user who displays the simplified avatar object 80 is that the billing amount has reached a reference value. Also, after the simplified avatar object 80 is displayed once, the simplified avatar object 80 may not be displayed. In this case, by restricting the viewing users who can display the simplified avatar object 80, all the simplified avatar objects 80 that satisfy the display conditions can be displayed even on a smartphone with a limited display area.

[0160] In the above embodiment, when multiple viewing users give gifts to a broadcasting user, the user device 12 may draw simplified avatar objects 80 corresponding to those viewing users. For example, the viewing user device 12B displays a screen of a group chat in which multiple viewing users chat. The group chat is any one of text chat, voice chat in which conversation is possible by voice, and video chat, or a combination thereof. One viewing user device 12B that makes a gift display request transmits a gift ID and a group ID to the server 13. Instead of the group ID, a user ID of a user belonging to the group may be transmitted. When the server 13 receives the gift display request, it requests image data of the simplified avatar object 80 from the viewing user device 12B of the viewing user associated with the group ID. The viewing user device 12B of the viewing user belonging to the group transmits image data of the simplified avatar object 80 to the server 13. The server 13 transmits the received image data of the simplified avatar object 80 to each user device 12. The user device 12 simultaneously draws the simplified avatar object 80 of the viewing user belonging to the group together with the gift object corresponding to the gift ID. Alternatively, the user device 12 renders a simplified avatar object 80 of a representative of the viewing users belonging to the group together with the gift object corresponding to the gift ID. The representative may be the viewing user who made the gift display request, or may be a viewing user predetermined within the group. When the representative is predetermined, the user device 12 or the server 13 records the user ID of the user.

[0161] When the broadcast end button is selected on the broadcasting user device 12A, image data may be generated in which the broadcasting user's avatar object 110 and the viewing users' simple avatar objects 80 are gathered together. The user device 12 displays the simple avatar objects 80 of all or some of the viewing users in addition to the broadcasting user's avatar object in the video. The user device 12 also captures the screen and displays it on the display 28 as a group photo. The user device 12 further stores the generated image data in the storage 22.

[0162] In each of the above embodiments, the user device 12 generates image data of the simple avatar object 80 and transmits it to the server 13. Instead of or in addition to this, the server 13 may generate the image data. Alternatively, the user device 12 may generate the image data of the simple avatar object 80 when generating video data. In this case, the part IDs of the main body parts and the part IDs of the attached parts of the avatar object 110 of each viewing user are obtained in advance. The device that generates the image data of the simple avatar object 80 can be changed depending on the video distribution method, etc.

[0163] (Information Processing System) Although the sensor unit 24 has been described as acquiring detection data detecting changes in the user's facial expression, head movement, and the user's relative position with respect to the sensor unit 24, at least one of these may be acquired. Furthermore, the sensor unit 24 may acquire other detection data, such as detection data detecting movement other than head movement, in addition to or instead of at least one of changes in the user's facial expression, head movement, and the user's relative position.

[0164] The virtual space displayed in the video may be an augmented reality (AR) space. For example, animations of avatar objects, gift objects, etc. based on data transmitted from the broadcasting user device 12A may be superimposed on an image of the real world captured by the camera of the viewing user device 12B. Alternatively, the broadcasting user device 12A may generate a video by superimposing animations of avatar objects, gift objects, etc. on an image of the real world captured by its own camera, encode the video, and transmit it to the server 13.

[0165] In each of the above embodiments, the user device 12 is an information processing device such as a smartphone, a mobile phone, a tablet terminal, a personal computer, a console game machine, a wearable computer such as a head-mounted display, etc. Alternatively, the information processing system may be a system provided in a studio for video distribution. The information processing system includes an information processing device, a server, a sensor unit 24 worn on the body of a distribution user, a tracking system for detecting the position of the sensor unit 24, an operation unit 27, a speaker 25, and a display 28. The tracking system may include a multi-axis laser emitter that emits a pulsed laser light for synchronization. The sensor unit 24 includes a sensor that detects laser light, and detects its own position and orientation while synchronizing with a synchronization pulse. For example, the sensor unit 24 may be a Vive Tracker (registered trademark) and a Vive Base Station (registered trademark) provided by HTC Corporation (registered trademark).

[0166] The control unit 20 of the user device 12 executes a first output control process, a second output control process, and a third output control process by executing an application program. At least one of these processes may be executed by the server 13 or another device constituting the information processing system 11. The control unit 30 of the server 13 executes a first process, a second process, a third process, and a fourth process. At least one of these processes may be executed by the user device 12 or another device constituting the information processing system 11. Next, the technical ideas that can be understood from the above embodiment and other examples will be described below. [1] On one or more computers: a first output control process for outputting a video including a first avatar object to which tracking data representing a motion of a first user is applied, on a display; a second output control process for, when an action of a second user on the video corresponds to a predetermined standard action, displaying in addition to the first avatar object, a second avatar object corresponding to the second user and having a smaller processing load for drawing than the first avatar object, and information corresponding to the standard action in the video within a display period shorter than the display period of the first avatar object. [2] The program according to [1], wherein in the second output control process, the voice spoken by the first user is output and the voice spoken by the second user is not output. [3] The program described in [1] further causes the computer to execute a third output control process to output to the display the video including the first avatar object and a third avatar object to which tracking data representing the movements of the co-star user is applied when another user participates in the video as a co-star, and the second output control process displays the second avatar object, which is smaller on the screen than the first avatar object and the third avatar object. [4] The program described in [1] further executes a third output control process to output to the display the video including the first avatar object and a third avatar object to which tracking data representing the actions of the co-star user has been applied, on condition that the first user has performed an authorization operation on the first user device to allow another user to participate as a co-star, and in the second output control process, displays the second avatar object in the video, on condition that the second user has performed the reference action. [5] The program described in [1], in which the second output control process displays the second avatar object in association with a message display section including content corresponding to the reference action. [6] The second output control process further comprises the step of displaying the second avatar object of the second user that is the subject of the reference action in the video when the action of the first user on the video corresponds to the reference action. [7] The program described in [6], wherein the reference action of the first user is that a first user device used by the first user sends a request to display profile information of the second user, and the profile information of the second user and the second avatar object are displayed on the first user device, and the second avatar object is displayed on a second user device used by the second user. [8] The program described in [1], in which the second output control process displays the second avatar object only when the second user, whose status information satisfies a predetermined condition, performs the reference action. [9] The program described in [8], wherein the status information of the second user is information indicating a friendly relationship with the first user, and the specified condition is that the identification information of the first user is included in the follow list of the second user.

[10] The program described in [8], wherein the status information of the second user is information regarding a group to which the second user belongs, and the specified condition is that the second user belongs to a group that includes the first user.

[11] The program described in [8], wherein the status information of the second user is information indicating activity regarding the video, and the specified condition is that the number of views, which is the number of times the video distributed by the first user has been viewed, is greater than or equal to a specified number, or the cumulative viewing time, which is the accumulated time for which the video distributed by the first user has been viewed, is greater than or equal to a specified time.

[12] The program described in [8], wherein the status information of the second user is information indicating activity regarding the video, and the specified condition is that the number of times the first user has posted a message regarding the video or sent a request to display a gift object is greater than or equal to a specified number.

[13] The program described in [8], wherein the status information of the second user is information indicating activity regarding the video, and the specified condition is that the charge for the video distributed by the first user has reached a standard value.

[14] The computer, The program described in [1] further executes a process for determining whether the number of second users watching the video exceeds a predetermined number, and in the second output control process, when the number of the second users is equal to or less than the predetermined number, displays multiple second avatar objects one by one, and when the number of the second users exceeds the predetermined number, displays multiple second avatar objects within the same period.

[15] The program described in [1], further causing the computer to execute the following processes: when the first user ends the distribution of the video, displaying on a screen a state in which the second avatar object and the first avatar object are gathered together; and storing in a memory unit image data capturing the screen including the gathered state.

[16] The program described in [1], in the second output control process, when displaying a gift object, if the second user device used by the second user who sent the request to display the gift object is continuing in viewing mode, the second avatar object and the gift object are displayed, and if the second user device is not in the viewing mode, the gift object is displayed without displaying the second avatar object.

[17] The program described in any one of [1] to

[16] , wherein the reference action is that a second user device used by the second user has started watching the video.

[18] The program described in any one of [1] to

[16] , wherein the reference action is that a second user device used by the second user has sent information regarding a rating for the video.

[19] The program described in any one of [1] to

[16] , wherein the reference action is that a second user device used by the second user posts a message for the video.

[20] The program described in any one of [1] to

[16] , wherein the reference action is that a second user device used by the second user sends a request to display a gift object to the video.

[21] One or more computers: An information processing method comprising: a first output control process for outputting a video including a first avatar object to a display, the first avatar object having tracking data applied thereto that represents the action of a first user; and a second output control process for displaying, when an action of a second user on the video corresponds to a predetermined standard action, a second avatar object that corresponds to the second user and has a smaller rendering processing load than the first avatar object, and information corresponding to the standard action on the video, in addition to the first avatar object, within a display period shorter than the display period of the first avatar object.

[22] An information processing device comprising: a first processing unit that outputs a video including a first avatar object to which tracking data representing a first user's action has been applied, on a display of a second user device used by a second user based on a video viewing request from the second user device; a second processing unit that acquires from the second user device information corresponding to a predetermined standard action, which is an action of the second user on the video; a third processing unit that acquires from the second user device data for drawing a second avatar object corresponding to the second user and which has a smaller processing load for drawing than the first avatar object; and a fourth processing unit that, when the action of the second user on the video corresponds to the standard action, displays the second avatar object and information corresponding to the standard action in addition to the first avatar object in the video within a display period shorter than the display period of the first avatar object.

[23] An information processing method in which one or more computers execute the following steps: a first process of outputting a video including a first avatar object to which tracking data representing a first user's action has been applied, on a display of a second user device used by a second user based on a video viewing request from the second user device; a second process of acquiring from the second user device information corresponding to a predetermined reference action, which is an action of the second user on the video, from the second user device; a third process of acquiring from the second user device data for drawing a second avatar object corresponding to the second user and which has a smaller processing load for drawing than the first avatar object; and a fourth process of, when the action of the second user on the video corresponds to the reference action, displaying in the video, in addition to the first avatar object, the second avatar object and information corresponding to the reference action within a display period shorter than the display period of the first avatar object. [Explanation of symbols]

[0167] 11...Information processing system 12...User device 13…Server

Claims

1. On one or more computers, a first output control process for outputting a video including a first avatar object to which tracking data representing a motion of a first user is applied, on a display; A program that executes a second output control process to display, in addition to the first avatar object, a second avatar object corresponding to the second user, which does not have tracking data representing the second user's actions on the video, in the video within a display period shorter than the display period of the first avatar object, when the second user's action on the video corresponds to a predetermined standard action.

2. The method further comprises causing the computer to execute a third output control process for outputting, when another user participates in the video as a co-star, the video including the first avatar object and a third avatar object to which tracking data representing the actions of the co-star user who is the co-star has been applied, to the display; The program according to claim 1 , wherein the second output control process displays the second avatar object having a smaller size on a screen than the first avatar object and the third avatar object.

3. The program described in claim 2, wherein in the second output control process, the voice spoken by the first user is output, and the voice spoken by some of the multiple co-starring users is not output.

4. causing the computer to further execute a third output control process for outputting, on the display, the video including the first avatar object and a third avatar object to which tracking data representing the actions of the co-star user who is the co-star has been applied, on condition that the first user has performed an authorization operation on a first user device to allow another user to participate as a co-star; The program according to claim 1 , wherein the second output control process displays the second avatar object in the video when a condition that the second user has performed the reference action is satisfied.

5. The program described in claim 1, wherein in the second output control process, the second avatar object is displayed in association with a message display section including content corresponding to the reference action.

6. The program described in claim 1, wherein the second output control process further comprises, when an action by the first user on the video corresponds to the reference action, displaying the second avatar object of the second user that is the subject of the reference action in the video.

7. The reference action of the first user is that a first user device used by the first user transmits a request to display profile information of the second user; The program according to claim 6 , wherein the first user device displays profile information of the second user and the second avatar object, and the second user device used by the second user displays the second avatar object.

8. The program described in claim 1, wherein the second output control process displays the second avatar object only when the second user whose status information satisfies a predetermined condition performs the reference action.

9. The program described in Claim 8, wherein the status information of the second user is information indicating a friendly relationship with the first user, and the specified condition is that the second user's follow list includes identification information of the first user.

10. The program described in Claim 8, wherein the status information of the second user is information regarding a group to which the second user belongs, and the specified condition is that the second user belongs to a group that includes the first user.

11. The program described in claim 8, wherein the status information of the second user is information indicating activity regarding the video, and the specified condition is that the number of views, which is the number of times the video distributed by the first user has been viewed, is greater than or equal to a specified number, or the cumulative viewing time, which is the accumulated time for which the video distributed by the first user has been viewed, is greater than or equal to a specified time.

12. The program described in Claim 8, wherein the status information of the second user is information indicating activity regarding the video, and the specified condition is that the number of times that the first user has posted a message regarding the video distributed by the first user or sent a request to display a gift object is greater than or equal to a specified number.

13. The program described in Claim 8, wherein the status information of the second user is information indicating activity regarding the video, and the specified condition is that charges for the video distributed by the first user have reached a reference value.

14. In the computer, further executing a process of determining whether the number of the second users viewing the video exceeds a predetermined number; The program described in claim 1, wherein in the second output control process, when the number of the second users is a predetermined number or less, multiple second avatar objects are displayed one by one, and when the number of the second users exceeds the predetermined number, multiple second avatar objects are displayed within the same period.

15. A process for displaying, on a screen, a state in which the second avatar object and the first avatar object are gathered together when the distribution of the video is ended by the first user, on the computer; The program according to claim 1 , further comprising: storing, in a storage unit, image data obtained by capturing a screen including the collected state.

16. The program described in claim 1, wherein in the second output control process, when displaying a gift object, if the second user device used by the second user who sent a request to display the gift object is continuing in viewing mode, the second avatar object and the gift object are displayed, and if the second user device is not in the viewing mode, the gift object is displayed without displaying the second avatar object.

17. A program described in any one of claims 1 to 16, wherein the reference action is that a second user device used by the second user has started watching the video.

18. A program described in any one of claims 1 to 16, wherein the reference action is that a second user device used by the second user transmits information regarding a rating for the video.

19. A program described in any one of claims 1 to 16, wherein the reference action is that a second user device used by the second user posts a message for the video.

20. A program described in any one of claims 1 to 16, wherein the reference action is that a second user device used by the second user sends a request to display a gift object to the video.

21. One or more computers, a first output control process for outputting a video including a first avatar object to which tracking data representing a motion of a first user is applied, on a display; An information processing method which executes a second output control process in which, when an action of a second user on the video corresponds to a predetermined standard action, in addition to the first avatar object, a second avatar object corresponding to the second user, which is an avatar object to which tracking data representing the action of the second user is not applied, is displayed on the video within a display period shorter than the display period of the first avatar object.

22. A first processing unit that outputs a video including a first avatar object to which tracking data representing a movement of a first user is applied on a display of a second user device used by a second user based on a video viewing request from the second user device; a second processing unit that acquires information corresponding to a predetermined reference action, which is an action of the second user on the video, from the second user device; a third processing unit that acquires data from the second user device for rendering a second avatar object corresponding to the second user, the second avatar object not being applied with tracking data representing a motion of the second user; an information processing device comprising: a fourth processing unit that, when an action of the second user on the video corresponds to the reference action, displays the second avatar object in addition to the first avatar object in the video within a display period shorter than a display period of the first avatar object.

23. One or more computers, a first process of outputting a video including a first avatar object to which tracking data representing a movement of a first user is applied, on a display of a second user device based on a video viewing request from a second user device used by a second user; a second process of acquiring information corresponding to a predetermined reference action, which is an action of the second user on the video, from the second user device; a third process of obtaining data from the second user device for rendering a second avatar object corresponding to the second user, the second avatar object not being applied with tracking data representing a motion of the second user; An information processing method comprising: a fourth process of displaying the second avatar object in the video within a display period shorter than the display period of the first avatar object, in addition to the first avatar object, when the second user's action on the video corresponds to the reference action.

Images