Program, information processing method and information processing system

Abstract

It is possible to prevent the virtual space image from becoming monotonous without increasing the burden on the user. [Solution] The information processing system includes an output control unit (17) that outputs the screen of a virtual space video distributed by a user to a display device, a generation unit (15) that associates a recorded image generated using at least a portion of the screen with attribute information of the recorded image and stores the associated images in a recorded image storage unit (19), and a selection unit (16) that uses the attribute information of the recorded image to select a recorded image from the recorded images stored in the recorded image storage unit (19) based on selection conditions, and the output control unit (17) outputs the selected recorded image to an image display object located in the virtual space during the period in which the video is being distributed.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] There is a video distribution system in which a distribution user distributes a video. In this system, a function for generating a still image by capturing the distribution screen based on an operation by a viewing user or the like is already known (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-005768 Summary of the Invention [Problem to be solved by the invention]

[0004] Users can share the captured still images with other users via social networking services (SNS). If information about video distribution, including still images, is viewed by many SNS users, the promotional effect of the video and the distribution user who distributes the video will increase.

[0005] However, even if the number of viewers temporarily increases, if the content of the video distribution is monotonous, viewers will not necessarily continue to watch. Many measures have been considered to prevent the content of videos from becoming monotonous, and one of these is to effectively utilize still images captured by the distribution users.

[0006] On the other hand, while streaming a video, the broadcaster must pay attention to messages from viewers, gift objects, interactions with viewers, etc. Particularly for beginners who are unfamiliar with streaming, there is a possibility that streaming a video will be difficult if there are too many things to pay attention to.

[0007] Therefore, it is desirable to have a method for effectively utilizing captured still images, etc. that is convenient by reducing the effort required of the distribution user. This problem can arise not only in systems that distribute video, but also in systems in which virtual space images are shared among multiple user devices. [Means for solving the problem]

[0008] The present disclosure provides an information processing system including an output control unit that displays virtual space images synchronized among multiple user devices on a display device, a generation unit that associates recorded images generated using at least a portion of the virtual space images with attribute information of the recorded images and stores the associated images in a storage unit, and a selection unit that uses the attribute information of the recorded images to select the recorded images from among the recorded images stored in the storage unit based on selection conditions, and the output control unit displays the selected recorded images on an image display object located in the virtual space during a period in which the virtual space images are displayed.

[0009] The present disclosure provides an information processing method, in which one or more computers execute a process of displaying a virtual space image synchronized among multiple user devices on a display device, a process of storing a recorded image generated using at least a portion of the video in association with attribute information of the recorded image in a storage unit, and a process of selecting the recorded image from the recorded images stored in the storage unit based on selection conditions using the attribute information of the recorded image, and in the process of displaying the virtual space image on the display device, the selected recorded image is displayed on an image display object located in the virtual space during a period in which the virtual space image is displayed.

[0010] The present disclosure provides a program that causes one or more computers to function as an output control unit that displays virtual space images synchronized among multiple user devices on a display device, a generation unit that associates recorded images generated using at least a portion of the virtual space images with attribute information of the recorded images and stores them in a storage unit, and a selection unit that uses the attribute information of the recorded images to select the recorded images from the recorded images stored in the storage unit based on selection conditions, and the output control unit displays the selected recorded images on an image display object located in the virtual space during a period in which the virtual space images are displayed. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to prevent virtual space images from becoming monotonous without increasing the burden on the user. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of the information processing apparatus according to the embodiment. [Figure 3] FIG. 2 is a block diagram illustrating functions of a user device according to the embodiment. [Figure 4] FIG. 2 is a block diagram illustrating the functions of a management server according to the embodiment. [Figure 5] FIG. 2 is a diagram schematically illustrating user management information according to the embodiment. [Figure 6] FIG. 2 is a diagram schematically showing image attribute information according to the embodiment. [Figure 7] FIG. 2 is a diagram schematically illustrating the configuration of a distribution room according to the embodiment. [Figure 8] 10 is a flowchart showing the procedure of a display process according to the embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a screen displayed on a user device according to the embodiment. [Figure 10]10A and 10B are diagrams showing image display objects displayed on the user device of the embodiment; [Figure 11] 10A and 10B are diagrams showing image display objects displayed on the user device of the embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0013] <Outline of the information processing system> The information processing system is composed of multiple information processing devices. The information processing system displays a video in a virtual space on a display of a user device. The information processing system associates a recorded image generated using at least a portion of the video with attribute information of the recorded image and stores the associated images in a storage unit. The information processing system also uses the attribute information of the recorded image to select a recorded image from the recorded images stored in the storage unit based on selection conditions, and outputs the selected recorded image to an image display object located in the virtual space during the period in which the video is being distributed.

[0014] An embodiment of an information processing system, an information processing method, and a program will be described below. <Information Processing System> An information processing system 10 will be described with reference to Fig. 1. The information processing system 10 includes a management server 11 and a user device 12. The management server 11 and the user device 12 are connected via a network NW.

[0015] In one aspect, the management server 11 is managed by a platform provider (PF) that provides a platform for distributing and viewing videos in a virtual space. The video distribution is not an archived video distributed at a time desired by the user 13 who views the video, but a live distribution that allows the distributing user 13 and the viewing user 13 to interact with each other. In this embodiment, the video includes a video of an avatar object corresponding to the distributing user 13.

[0016] The management server 11 comprises one or more devices. In other words, the management server 11 may be a group of servers. The management server 11 is connected to a user device 12 via a network NW such as the Internet. The user device 12 is a device on which a program for distributing and viewing videos is installed, or a device used by a user 13 who uses a video providing service for distributing and viewing videos using a web browser or the like. The management server 11 is responsible for relaying data related to video distribution.

[0017] FIG. 1 shows a broadcasting user 13A and a viewing user 13B. The broadcasting user 13A is a user 13 who broadcasts videos using a user device 12. The viewing user 13B is a user 13 who watches the broadcasted videos on the user device 12. A single user 13 can be a broadcasting user 13A at one time and a viewing user 13B at another time. That is, the user 13 is a broadcasting user 13A when broadcasting videos, and a viewing user 13B when watching videos. The user device 12 that broadcasts videos is referred to as a broadcasting user device 12A, and the user device 12 that displays videos for viewing is referred to as a viewing user device 12B. The user device 12 can switch between a mode for broadcasting videos and a mode for displaying videos for viewing. In this embodiment, when the broadcasting user 13A and the viewing user 13B are described without distinction, they are simply referred to as users 13. When the broadcasting user device 12A and the viewing user device 12B are described without distinction, they are simply referred to as user devices 12.

[0018] <Hardware> The hardware of the user device 12 will be described with reference to FIG. 2. The user device 12 may be a smartphone (multifunction phone), a tablet device, a personal computer, a console game machine, a wearable computer, or any other information processing device capable of playing videos. The wearable computer may be, for example, a head-mounted display worn on the head of the user 13, or a glasses-type wearable device. The user device 12 may be a standalone device or a set of multiple devices connected so that various data can be transmitted and received between them. An example of the latter user device 12 is a system that performs outside-in tracking.

[0019] The user device 12 includes a processing circuit 100, a storage 104, a sensor unit 112, a display 109, an input device 110, and peripheral devices 111. The user device 12 illustrated in Fig. 2 includes the sensor unit 112 that detects the movement of the broadcast user 13A. Note that the sensor unit 112 may be a separate device from the user device 12 that is communicatively connected to the user device 12.

[0020] The processing circuit 100 includes a CPU 101, which is a processor that executes one or more of the control processes disclosed in this specification, and a memory 102. Processing data and instructions may be stored in the memory 102. The processing data and instructions may also be stored in a storage 104, which is a storage medium disk such as a hard drive (HDD) or a portable storage medium, or in a storage medium provided separately from the processing circuit 100. The storage 104 is connected to the CPU 101 via a storage controller 103 and a bus 108. In this embodiment, an application is stored in the storage 104. The application includes a program for distributing and viewing videos and various data required to execute the program.

[0021] Each function disclosed herein may be implemented using circuitry, which may include, but is not limited to, a CPU 101, a general-purpose processor, a special-purpose processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), other conventional circuitry, and / or a combination thereof, configured or programmed to perform the disclosed functions. A processor is a processing circuit or circuit because it includes transistors and other circuits therein. A processor may be a programmed processor that executes a program stored in a memory. In this disclosure, a unit or means, such as a processing circuit, is hardware that performs or is programmed to perform the described functions. The hardware may be any hardware disclosed herein or otherwise known that is programmed or configured to perform the described functions.

[0022] Furthermore, the processor is not limited by the form of computer-readable medium on which the process instructions are stored. For example, the instructions may be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or other non-transitory computer-readable medium of an information processing device with which processing circuit 100 communicates, such as a server or computer. The process may also be stored in network-based storage, cloud-based storage, or other mobile-accessible storage and executable by processing circuit 100.

[0023] Furthermore, the descriptions or blocks in the flowcharts disclosed herein can be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps in a process. Furthermore, the descriptions or blocks in the flowcharts disclosed herein can perform two or more functions or steps in an order different from that shown or described in the specification, such as substantially simultaneously or in reverse order.

[0024] The hardware elements for implementing the processing circuit 100 may be implemented by various circuit elements. Furthermore, each function disclosed herein may be implemented by a circuit including one or more processing circuits.

[0025] The processing circuit 100 also includes a network controller 106 for connecting to a network NW. The network NW may be a public network such as the Internet, a private network such as a local area network (LAN) or a wide area network (WAN), or any combination thereof, and may also include a public switched telephone network (PSTN) or an integrated services digital network (ISDN®) or a sub-network. The network NW may also be a wired network such as an Ethernet® network or a universal serial bus (USB) cable, or a wireless network such as a cellular network including 3G, 4G, and 5G wireless cellular systems. The wireless network may also be Wi-Fi®, wireless LAN, Bluetooth®, or other known forms of wireless communication. Furthermore, the network controller 106 may be compliant with other direct communication standards such as Bluetooth®, near field communication (NFC), infrared, etc.

[0026] The processing circuit 100 further includes a display controller 105 and an input / output interface 107. The display controller 105, the network controller 106, and the input / output interface 107 are connected to a bus 108. The display controller 105 is connected to a display 109. The input / output interface 107 is connected to an input device 110 and peripheral devices 111.

[0027] The sensor unit 112 will now be described. The sensor unit 112 is one or more sensors that detect data indicating the movement of the user 13. The movement of the user 13 includes face motion, which indicates a change in the user's facial expression, and body motion, which indicates a change in the position and orientation of the user's body relative to the sensor unit 112. Face motion includes movements such as blinking and opening and closing of the mouth. A known sensor can be used as the sensor unit 112. The sensor unit 112 has a sensor that measures the distance to a measurement object by irradiating the measurement object with light. Examples of such sensors include true depth sensors or "LIDAR" (Light Detection and Ranging, or Laser Imaging Detection and Ranging) sensors. This allows the sensor unit 112 to capture accurate face data. The processing unit of the sensor unit 112 generates various information based on the depth map and the infrared image, compares the information with registered reference data, and calculates the depth of each point on the face (the distance between each point and the near-infrared camera) and positional deviations other than in the depth direction. In another example, the sensor unit 112 includes a ToF sensor that measures the time of flight (time of flight) between when light is irradiated onto a measurement object such as the face of the user 13 and when it is reflected back, or the phase difference between the irradiated light and the reflected light, using a distance measurement method other than the above-described methods. The ToF sensor calculates the distance to the measurement object based on the time of flight or phase difference. Alternatively or in addition to this, the sensor unit 112 may include a camera that captures an image of the face of the user 13 and an image processing unit that processes the image data captured by the camera.

[0028] The sensor unit 112 outputs facial motion and body motion based on the movements of the user 13 to the processing circuit 100 as tracking data. The tracking data is data that detects the orientation and position of parts of the user 13's body. The tracking data is included in motion data that moves an avatar object. In this embodiment, a character object to which the motion data detected by the sensor unit 112 is applied is called an avatar object. Motion data refers to all data for moving an avatar object corresponding to the user 13.

[0029] For example, the motion data includes not only tracking data but also emote data. The emote data is data for making the avatar object perform a pre-registered predetermined movement, such as "clap." This pre-registered movement of the avatar object will be referred to as an "emote" hereinafter.

[0030] The sensor unit 112 may also have a function of tracking not only the face of the user 13 but also the hands (hand tracking). The sensor unit 112 may also include a sensor that detects the position or orientation of a human body part other than the hand. The sensor unit 112 may further include a sensor that detects speed or acceleration, a sensor that detects direction or orientation (such as a gyro sensor), etc.

[0031] The input device 110 may be a touch panel, a keyboard, a mouse, a controller operated by the user 13 with their hands (e.g., a game controller such as a game pad or joystick), or operation buttons provided on the housing of the user device 12. The controller may incorporate various known sensors, such as an acceleration sensor, a gyro, or an inertial measurement unit (IMU). In another example, the input device 110 may be a tracking device that identifies the hand movements, eye movements, head movements, gaze direction, etc. of the user 13. In this embodiment, for example, based on the hand movements of the user 13, it is possible to determine the instructions of the user 13 and perform various operations, such as starting or ending video distribution, rating messages or videos, and displaying predetermined objects.

[0032] The peripheral device 111 includes a microphone that collects sounds in the real world, such as the voice of the user 13, and a speaker that outputs sound effects and the voices of other users 13, etc. The management server 11 has the same processing circuit 100 and storage 104 as the user device 12. In the management server 11, the input device 110, the peripheral device 111, the sensor unit 112, the display 109, and the like can be omitted.

[0033] <User device> The functions of the user device 12 will be described with reference to Fig. 3. The processing circuit 100 executes a program recorded in the storage 104, causing the user device 12 to function as a generation unit 15, a selection unit 16, and an output control unit 17. The user device 12 also includes an asset storage unit 18 and a recorded image storage unit 19.

[0034] The output control unit 17 displays virtual space images synchronized among multiple user devices 12 on the display 109. In this embodiment, the output control unit 17 displays a video of the virtual space distributed by the distribution user 13A on the display 109. In addition, when the user device 12 is outputting a video for viewing, the output control unit 17 outputs the voice, etc., spoken by the distribution user 13A from the speaker.

[0035] Furthermore, the output control unit 17 outputs the selected recorded image to an image display object located in the virtual space during the period in which the video is being distributed. The generation unit 15 associates the recorded image, which is an image generated based on at least a portion of the screen of the moving image in the virtual space, with attribute information of the recorded image and stores them in the recorded image storage unit 19. The generation unit 15 generates the recorded image based on the operation of the broadcast user device 12A by the broadcast user 13A.

[0036] The selection unit 16 uses the attribute information of the recorded image to select a recorded image from among the recorded images stored in the recorded image storage unit 19 based on selection conditions. The selection unit 16 acquires the voice of the broadcast user 13A and performs voice recognition. For example, the selection unit 16 converts the voice information into text data.

[0037] The asset storage unit 18 stores asset data for outputting each screen or sound of the application. The asset data is updated as needed with update data sent by the management server 11.

[0038] The recorded image storage unit 19 stores a recorded image (captured image) generated by the user device 12 taking a screenshot (capture). The recorded image is a still image file or a video file. The recorded image storage unit 19 stores attribute information at the time the screenshot was taken in association with the recorded image.

[0039] <Administration Server> The management server 11 will be described with reference to Fig. 4. The management server 11 functions as a distribution management unit 20 and a user information management unit 21 by the processing circuit 100 thereof executing a program recorded in the storage 104. The management server 11 also includes a user management information storage unit 22 and an asset storage unit 23. These storage units 22 and 23 are configured by the storage 104.

[0040] The distribution management unit 20 starts distribution of the video in response to a distribution request from the distribution user device 12 A. Furthermore, the distribution management unit 20 transmits data for displaying the video to the viewer user device 12 B in response to a viewing request from the viewer user device 12 B.

[0041] The distribution management unit 20 receives a message input by the viewing user 13B from the viewing user device 12B. The distribution management unit 20 transmits the message from the viewing user 13B to the broadcast user device 12A and the viewing user device 12B. The distribution management unit 20 also receives identification information and a display request for the object provided by the viewing user 13B. The distribution management unit 20 transmits the identification information and display request for the object provided by the viewing user 13B to the broadcast user device 12A and the viewing user device 12B. An example of an object provided by the viewing user 13B is a gift object. A reward may be given to the broadcast user 13A by providing the gift object.

[0042] The asset storage unit 23 stores asset data to be transmitted to each user device 12. The distribution management unit 20 reads asset data required for the user device 12 to execute an application from the asset storage unit 23 and transmits the asset data to the user device 12 as update data.

[0043] FIG. 5 shows an example of the user management information 24 recorded in the user management information storage unit 22. The user management information 24 is stored for each user. The user management information 24 includes a user ID (identifier), which is user identification information, and possession information. The possession information indicates value possessed by the user 13 for acquiring items and requesting the display of gift objects in an application or a platform that provides the application. The value includes, for example, coins. Coins can be used to acquire items, pay for lotteries (gacha), or pay for requesting the display of gift objects. Coins can be purchased by the user 13. The possession information may also include accumulated points awarded based on the user 13's activities. Activities include streaming and viewing videos, participating in events, etc. Points are awarded based on the user 13's activities, and the points are added to the accumulated points. The possession information may also include information regarding electronic money or cryptocurrency that can be used in the real world.

[0044] FIG. 6 shows an example of attribute information 25 stored in the recorded image storage unit 19. The attribute information 25 is stored for each recorded image. The attribute information 25 functions as a tag used when searching for a recorded image. The attribute information 25 includes the date and time and the distribution space. The attribute information 25 may also include viewing information and distribution information.

[0045] The date and time indicates the date when the recorded image was generated. The information on the distribution space is the identification information of the distribution room or world used when the recorded image was generated. Since a distribution room is individually set up for each distribution user 13A, in the case of a distribution room, the identification information of the distribution user 13A may be used. Examples of the identification information of a world are "Exhibition World" and "Fireworks Festival Venue." Distribution rooms and worlds will be described later.

[0046] The information about the distribution space may be identification information of an area within a distribution room or identification information of an area within a world. For example, a distribution room is divided into multiple areas such as a "room" or a "garden." A world is divided into multiple areas such as exhibition booths or a stage where users 13 perform live shows. These pieces of identification information indicate the area where the avatar object is located when the recorded image is generated.

[0047] The information on the distribution space may include information on an object included in the recorded image. For example, if the recorded image includes a "bookshelf" object, the information on the distribution space may include identification information of the "bookshelf" object.

[0048] The information on the distribution space may include information on an avatar object included in the recorded image. For example, a world is a space in which each user 13 can participate. When an avatar object is included in the recorded image, the information on the distribution space may include identification information of the avatar object or the user ID of the user 13 corresponding to the avatar object.

[0049] The viewing information includes the behavior of the viewing users 13B during video distribution. An example of information indicating the behavior is the identification information of a gift object displayed in the scene where the recorded image was generated. That is, the identification information of the gift object included in the recorded image. An example of the identification information of the gift object is a "stuffed rabbit." The viewing information may include the number of viewing users 13B when the recorded image was generated, or the number of times the viewing users 13B gave a favorable rating (the number of times the "Like" button was pressed). The viewing information may also include messages sent from the viewing users 13B during video distribution.

[0050] The distribution information includes information about collaboration distribution and information about events. Collaboration distribution is distribution by multiple avatar objects in the same distribution room or world. Information about collaboration distribution indicates a combination of distribution users 13A or a combination of avatar objects. An example of information indicating a combination of distribution users 13A is "distributor user A and broadcaster user B." Information about events indicates identification information of an event held on a platform that provides an application, etc.

[0051] If the corresponding viewing information and distribution information are not detected when generating the recorded image, the attribute information 25 does not need to include them. Additionally, the attribute information 25 may also include other items such as information about the games played by the distribution user 13A during the video distribution, and information about the objects worn by the avatar object.

[0052] The generation unit 15 of the broadcast user device 12A acquires the above information when, for example, the broadcast user 13A performs an operation to instruct the broadcast user 13A to take a screenshot. The generation unit 15 then stores the attribute information 25 in the recorded image storage unit 19 in association with the recorded image. For example, suppose that "Broadcaster A" is performing a collaborative broadcast with "Broadcaster B" in "Broadcast Room A" on "December 25, 2022," when a gift object of a "stuffed rabbit" is provided by the viewing user 13B. Also, suppose that the broadcast user device 12A receives an instruction to take a screenshot at this time. In this case, the generation unit 15 includes five items of information, "December 25, 2022," "Broadcast Room A," "Broadcaster Rabbit Stuffed Toy," and "Broadcaster User A and Broadcaster User B," in a single attribute information 25.

[0053] <Video distribution and viewing methods> Next, the video distribution and viewing method will be explained. When the management server 11 receives a distribution request and identification information of the specified virtual space from the broadcast user device 12A, it generates a distribution slot. At this time, the broadcast user 13A can select his / her own broadcast room (first virtual space) or world (second virtual space). The world may be selected from multiple worlds with different themes.

[0054] The distribution slots in which videos are currently being distributed are displayed on a list screen displayed on each user device 12. The viewing user 13B can start viewing the video by selecting the distribution slot they want to view from the list screen. The management server 11 transmits asset data and the like for displaying the virtual space to the user device 12.

[0055] Video distribution and viewing methods can be divided into client rendering, browser rendering, video distribution, and server distribution, depending on the rendering device and method. Other methods may be used as long as either the user device 12 or the management server 11 can perform image processing. In this embodiment, the client rendering method is used.

[0056] In the client rendering method, each user device 12 renders objects by executing a program. User device 12 stores 3D model data of avatar objects and objects other than avatar objects. Objects other than avatar objects are also called physical objects. Physical objects include gift objects. 3D model data of avatar objects includes data on body parts that make up the avatar object itself and on attachment parts that can be attached to the avatar object. Data on body parts includes polygon data, skeletal data (bones) for expressing the movement of the avatar object, texture data, etc. Attachment parts include texture data, etc. Broadcasting user 13A can set attachment parts for the avatar object corresponding to him / her according to his / her preferences. Model data of physical objects other than avatar objects includes polygon data, texture data, etc.

[0057] The output control unit 17 performs drawing using video data including motion data and identification information (part IDs) indicating parts of each object. When distributing a video, the output control unit 17 of the broadcasting user device 12A transmits video data including motion data, avatar object and virtual camera coordinates, etc. to the management server 11. The output control unit 17 also transmits part IDs indicating each part of the avatar object to the management server 11 at a predetermined timing (for example, at the start of distribution). The management server 11 transmits the video data to the viewing user device 12B, etc. The viewing user device 12B receives this data from the management server 11. The viewing user device 12B then uses the received data to draw images of the avatar object and physical object of the broadcasting user 13A.

[0058] The output control unit 17 also transmits audio data based on the audio collected by the microphone to the management server 11. The management server 11 transmits the audio data to the viewing user device 12B. A timestamp is attached to the motion data, audio data, etc.

[0059] The output control unit 17 of the viewing user device 12B receives video data and audio data for displaying video from the management server 11. Then, the output control unit 17 uses the video data to draw an image including an avatar object corresponding to the broadcast user 13A.

[0060] In addition, the output control unit 17 of the broadcasting user device 12A and the viewing user device 12B receives data such as messages posted by the viewing user 13B from the management server 11. In addition, the output control unit 17 receives from the management server 11 the identification information and display request of the gift object sent by the viewing user 13B.

[0061] The output control unit 17 generates image output data by combining rendered images of avatar objects and other objects with posted messages, GUI (Graphical User Interface) elements, etc., and outputs the image output data to the display 109. The output control unit 17 also synchronizes audio data with the image output data based on a timestamp and outputs the audio data from the speaker.

[0062] The broadcast user device 12A stores information at the time of broadcast in the storage 104. The information at the time of broadcast may be temporarily stored and deleted or transmitted to the management server 11 after a predetermined time has elapsed. The information at the time of broadcast includes the date and time, identification information of the broadcast room or world, information about the gift object, the number of viewers, etc. Furthermore, when a collaborative broadcast is performed, the broadcast user device 12A stores identification information of other broadcast users 13A who participated in the collaborative broadcast. Note that the management server 11 may store such information at the time of broadcast in the user management information storage unit 22.

[0063] The broadcasting user device 12A and the viewing user device 12B may display video in different ways. For example, the broadcasting user device 12A may display video in a client rendering manner, while the viewing user device 12B may display video in a browser rendering manner. The user 13 may also be able to select the video display manner. For example, some of the viewing user devices 12B may display video in a client rendering manner, while others may display video in a browser rendering manner.

[0064] <Virtual Space> The virtual space 30 in this embodiment will be described with reference to Fig. 7. The virtual space 30 includes a plurality of objects. The virtual space 30 includes an avatar object 51 and a physical object.

[0065] The user 13 can use a broadcasting room 31 and a world as the virtual space 30. The broadcasting room 31 is a space individually associated with the broadcasting user 13A. The broadcasting user 13A can place any physical object selected from a group of objects in the broadcasting room 31.

[0066] A world differs from a distribution room 31 in that it is not a space associated with an individual user 13, but is a space in which each user 13 can participate. A world may be a space in which only distribution users 13A can participate. Alternatively, a world may be a space in which all users 13, including viewing users 13B, can participate. Worlds are often formed around a theme. For example, they may be virtual spaces that represent facilities that exist in the real world, exhibitions that display products from companies or individuals, or live venues where performers appear.

[0067] A three-dimensional coordinate system 32 is set in the virtual space 30 to indicate the positions of objects, etc. The three-dimensional coordinate system 32 is a coordinate system having an X-axis, a Y-axis, and a Z-axis. The position of each object can be specified by the coordinates of the three-dimensional coordinate system 32. An object coordinate system (not shown) is set for each object. The object coordinate system is a coordinate system having an X-axis, a Y-axis, and a Z-axis. The orientation, movement, etc. of the avatar object 51 can be specified by the coordinates or rotation coordinates of this coordinate system.

[0068] 7 is defined by a floor 33 and walls 34. The distribution room 31 may be further defined by a ceiling 38. Physical objects 35 are placed in the distribution room 31. Examples of the physical objects 35 are furniture such as shelves or chairs, or miscellaneous items such as photo frames. The physical objects 35 also include image display objects 36. A recorded image is displayed on the image display object 36. The image display object 36 has a display area for displaying an image. The output control unit 17 displays a predetermined image in the display area. In the example of FIG. 7, the image display objects 36 are objects of an album, a calendar, and a photo frame. The image display object 36 may also be other objects such as a photo frame or a poster.

[0069] A virtual camera (not shown) is positioned in the distribution room 31 or the world. A camera coordinate system (not shown) is set for the virtual camera, similar to the object. The virtual camera changes its field of view based on an input operation by the distribution user 13A. The field of view of the virtual camera may be changeable by an input operation by the viewing user 13B. For example, the virtual camera may follow the avatar object 51 so as to maintain a constant distance from the avatar object 51.

[0070] The output control unit 17 performs rendering based on the coordinates of objects included in a predetermined field of view (for example, a field of view cone having an approximately conical shape) based on the virtual camera of the broadcasting user device 12A. The output control unit 17 then displays the image generated by rendering on the display 109. The rendering here refers to a drawing process that includes obtaining the position of the virtual camera, perspective projection, hidden surface removal based on depth information associated with each object, shading, texture mapping, etc.

[0071] The output control unit 17 changes the field of view of the virtual camera based on the operation of the broadcast user 13A, thereby displaying the image display object 36 on the screen. 7, the output control unit 17 may receive an operation from the broadcast user 13A to select an image display object 36, and display the image display object 36 on the screen. At this time, the output control unit 17 may adjust the recorded image displayed on the image display object 36 to a predetermined size, and display the virtual space 30 in front of the rendered image.

[0072] <Operation> The operation of the information processing system 10 will now be described. (Generation of recorded images) Here, a case will be described in which the broadcast user 13A issues a command to take a screenshot while broadcasting a video. The broadcast user device 12A accepts the command to take a screenshot from the broadcast user 13A. The command to take a screenshot is given by input to the input device 110 or by voice input.

[0073] The generation unit 15 of the broadcast user device 12A generates a recorded image by reading (capturing) the information displayed on the screen based on the screenshot instruction. Furthermore, the generating unit 15 generates attribute information 25 based on the distribution status. The generating unit 15 stores the generated attribute information 25 in the recorded image storage unit 19 in association with the recorded image.

[0074] (Selection of recorded images) The selection process for selecting images to be recorded during the period when a video is being distributed will be described. The selection unit 16 may set selection conditions for the recorded images in advance. These selection conditions are predetermined by a platform administrator or the like. Furthermore, as a step prior to the selection process, when the selection unit 16 newly stores the recorded images and attribute information 25 in the recorded image storage unit 19, the selection unit 16 may use the attribute information 25 to set selection conditions for the recorded images corresponding to each item of the attribute information 25. In other words, the selection conditions are set so that the generated recorded images can be called up at appropriate scenes in future video distribution.

[0075] For example, for the "distribution space" included in the attribute information 25, initial information is set such as "the name of a specific space or event appears in the broadcasting user's conversation" or "broadcasting will be performed in the same world or event as in past broadcasts." When the selection unit 16 newly stores attribute information 25 including the recorded image and information on "exhibition" in the recorded image storage unit 19, the selection unit 16 uses the initial information to set a new selection condition such as "the word 'exhibition' appears in the broadcasting user's conversation," or "broadcasting will be performed in the same 'exhibition' as in past broadcasts." The "exhibition" may be a proper name held on a predetermined schedule.

[0076] For the "viewing information," initial information such as "a specific gift object was provided" and "a message containing a specific keyword was sent" is set. When the selection unit 16 stores new attribute information 25 including the recorded image and the identification information for the "rabbit object" in the recorded image storage unit 19, the selection unit 16 uses the initial information to set a new selection condition of "a 'rabbit object' was provided." The selection unit 16 also sets a new selection condition of "a message containing 'rabbit' was sent."

[0077] When the selection unit 16 newly stores the attribute information 25 including the recorded image and the identification information of "exhibition" in the recorded image storage unit 19, it uses the initial information to set a new selection condition of "a message including 'exhibition' was sent." Alternatively, the initial information may be "a stamp associated with a predetermined keyword was provided."

[0078] For example, initial information is set for "distribution information" such as "collaboration distribution will be performed by distribution users including the same combination as in past distributions." When the selection unit 16 newly stores the recorded image and attribute information 25 including identification information for "distributor A and distributor B" in the recorded image storage unit 19, the selection unit 16 uses the initial information to newly set a selection condition such as "collaboration distribution will be performed by a group including 'distributor A and distributor B'."

[0079] As shown in FIG. 8, the selection unit 16 acquires the distribution status from the start of video distribution to the end of video distribution (step S1). Specifically, the selection unit 16 acquires identification information of the distribution room 31 or world, information about the event participated in, the voice recognition results of the voice spoken by the distribution user 13A, the message sent by the viewing user 13B, identification information of the stamp provided by the viewing user 13B, and identification information of the gift object. This information may be received from the management server 11 or stored in the device itself as a distribution log (distribution record). Alternatively, the selection unit 16 may acquire the information by automatically reading the video and analyzing the video using the output control unit 17, etc. The distribution status acquired by the selection unit 16 corresponds to the items of attribute information 25 associated with the recorded image.

[0080] The selection unit 16 determines whether the selection conditions set based on the attribute information 25 are satisfied based on the acquired distribution status (step S2). At this time, the selection unit 16 may determine that the selection conditions are satisfied when the distribution status and the selection conditions completely match. Alternatively, the selection unit 16 may determine that the selection conditions are satisfied when the similarity between the distribution status and the selection conditions is at or above a predetermined level.

[0081] An example of the selection of a recorded image will be described. During collaboration distribution, the selection unit 16 determines that the selection condition is satisfied when the combination of distributor users 13A and the combination of distributor users 13A indicated by the attribute information 25 match in part or in whole.

[0082] The selection unit 16 may select the recorded image when a keyword included in the speech recognition result of recognizing the speech of the broadcast user 13A is included in the attribute information 25. The selection unit 16 may select the recorded image when the keyword recognized by the speech and the keyword included in the attribute information 25 partially match.

[0083] When a distribution user 13A participates in an event, if the attribute information 25 contains information about a past event associated with that event, the selection unit 16 may select a recorded image 60 associated with that attribute information 25.

[0084] When the broadcasting user 13A broadcasts again in a virtual space such as a broadcasting room, world, or area within such a space in which the broadcasting user 13A broadcast in the past, the selection unit 16 may select the recorded image 60 associated with the attribute information 25. Alternatively, when the broadcasting user 13A broadcasts again in a virtual space that includes the same objects, such as furniture, as those located in the virtual space in which the broadcasting user 13A broadcast in the past, the selection unit 16 may select the recorded image 60 associated with the attribute information 25.

[0085] The selection unit 16 may select a recorded image 60 associated with the attribute information 25 when an avatar object 51 that existed in a virtual space where the broadcasting user 13A previously broadcasted is present again in the virtual space where the broadcasting user 13A is currently broadcasting or will start broadcasting.

[0086] The selection unit 16 may select the recorded image 60 associated with the attribute information 25 when a keyword included in the attribute information 25 is associated with the message or stamp sent by the viewing user 13B.

[0087] When a gift object previously provided by the viewing user 13B is provided again, the selection unit 16 may select the recorded image 60 associated with the attribute information 25 of the gift object. In this way, when the selector 16 determines that any one of the selection conditions is satisfied (step S2: YES), it acquires the recorded image 60 whose attribute information 25 satisfies the selection condition (step S3).

[0088] On the other hand, if the selection unit 16 determines that the selection condition is not satisfied (step S2: NO), the process returns to step S1 and acquires the distribution status. When the recorded image 60 is selected, the output control unit 17 identifies the image display object 36 (step S4). Note that the selection unit 16 may identify the image display object 36. When one image display object 36 is set in the distribution room 31 or the world, the output control unit 17 identifies the image display object 36. When multiple image display objects 36 are set in the distribution room 31 or the world, the output control unit 17 identifies the image display objects 36 in the order assigned to the image display objects 36. When the recorded image 60 is already displayed on the image display object 36 assigned "No. 1," the output control unit 17 identifies the next image display object 36 assigned "No. 2" as the display target for the recorded image 60.

[0089] The order is assigned so as to match the order in which the image display objects 36 that the broadcasting user 13A can freely arrange are placed in the broadcasting room 31, the order of the size of the display region 37, etc. For example, if a recorded image 60 includes a large number of avatar objects 51, the recorded image 60 may be preferentially assigned to the image display object 36 with the largest size of the display region 37, i.e., the image display object 36 with the highest order.

[0090] The output control unit 17 displays the recorded image 60 on the identified image display object 36 (step S5). Furthermore, the broadcasting user device 12A transmits the identification information of the image display object 36 and the data of the recorded image 60 to the viewing user device 12B via the management server 11. The output control unit 17 of the viewing user device 12B displays the recorded image 60 on the image display object 36 using the received identification information of the image display object 36 and the data of the recorded image 60. As a result, the recorded image 60 stored in the broadcasting user device 12A is also displayed on the viewing user device 12B. Once the recorded image 60 has been displayed, the process returns to step S1. In other words, the recorded image 60 is displayed each time the selection condition is met.

[0091] <Display mode of image display object> Next, an example of a screen displayed on the display 109 of the user device 12 will be described. 9 shows an example of a virtual space screen 50 of the broadcasting user device 12A. The virtual space screen 50 includes an avatar object 51 and a gift object 55. In the example of FIG. 9, the gift object 55 is a "stuffed rabbit." The virtual space screen 50 also includes a message 56 sent by the viewing user 13B.

[0092] The virtual space screen 50 also includes multiple GUI elements 52 and icons (not shown). One of the GUI elements 52 is a capture button 53. The capture button 53 may be displayed on the virtual space screen 50 in response to an operation by the broadcasting user 13A. When the generation unit 15 receives an operation of the capture button 53 by the broadcasting user 13A, it generates a recorded image 60. The generation unit 15 may process the recorded image 60. For example, the generation unit 15 may generate a recorded image 60 of the entire screen, and then extract a predetermined area of ​​the recorded image 60. Alternatively, the output control unit 17 may change at least one of the brightness, luminance, and color of the recorded image 60, or add effects to the recorded image 60.

[0093] When the generation unit 15 receives an operation on the capture button 53 by the distribution user 13A, it generates a recorded image 60 by taking a screenshot of the virtual space screen 50. The generation unit 15 also includes in the attribute information 25 the date and time, identification information of the distribution room 31 or world, text data indicating the content of the message 56, and identification information of the gift object 55.

[0094] FIG. 10 shows an example of an image display object 36. The image display object 36 is a photo frame object. A recorded image 60 is displayed in the display area 37 of the image display object 36. When the recorded image 60 is displayed, the broadcasting user 13A may be notified by changing the display mode of the image display object 36 or by displaying a pop-up. For example, the broadcasting user 13A displays the image display object 36 by changing the field of view of the virtual camera. That is, the broadcasting user 13A moves the avatar object 51 close to the image display object 36 so that the image display object 36 is displayed in the field of view, thereby causing the virtual camera to follow the avatar object 51, or performs an operation to change at least one of the position and direction of the virtual camera. Alternatively, the output control unit 17 may accept an operation to select the image display object 36 and enlarge the recorded image 60.

[0095] The three image display objects 36 illustrated in FIG. 10 are assigned the numbers "1" to "3." For example, if broadcast user 13A utters a keyword related to the event "fireworks festival," selection unit 16 determines that the selection condition "keywords including 'fireworks festival' appear in the broadcast user's conversation" is met. Then, selection unit 16 acquires recorded image 60 associated with attribute information 25 including information about the "fireworks festival" event.

[0096] The output control unit 17 displays the acquired recorded image 60 in the display area 37 of the image display object 36. The recorded image 60 may be displayed continuously until the distribution is completed. Alternatively, the recorded image 60 may be displayed for only a predetermined time and then deleted before the distribution is completed.

[0097] If the output control unit 17 determines that a new selection condition is satisfied while the recorded image 60 is being displayed on the image display object 36, the output control unit 17 displays the new recorded image 60 on the image display object 36.

[0098] After the recorded image 60 is displayed on all the image display objects 36, if any of the recorded images 60 is deleted, a new recorded image 60 is displayed on the image display object 36 from which the recorded image 60 has been deleted.

[0099] Fig. 11 shows an example in which the image display object 36 is a calendar object. This image display object 36 is capable of displaying a plurality of recorded images 60 associated with the broadcast user 13A. The image display object 36 shown in Fig. 11 has the same number of display areas 37 as the number of dates on the calendar.

[0100] The output control unit 17 displays the recorded image 60 in the display area 37 of the image display object 36 for the date corresponding to the date and time, based on the date and time information included in the attribute information 25 of the recorded image 60. For example, if the date and time included in the attribute information 25 of the recorded image 60 is "January 10th," the recorded image 60 is displayed in the display area 37 of the image display object 36 corresponding to "January 10th." The recorded image 60 may be displayed in a reduced or cropped state to fit the display area 37. The recorded image 60 is displayed in a manner that allows it to accept operations from the broadcast user 13A. When the recorded image 60 is selected by the broadcast user 13A, the output control unit 17 enlarges and displays the recorded image 60. Text data can also be displayed in the image display object 36. The broadcast user 13A selects one of the multiple display areas 37 set in the image display object 36 and inputs text data into that display area 37 using the input device 110. For example, as illustrated in the display area 37 corresponding to "January 31" of the image display object 36 in FIG. 11, a distribution schedule (e.g., "Distribution Schedule") can be displayed on the distribution schedule date registered by the distribution user 13A.

[0101] While the broadcasting user 13A and the viewing user 13B can simply enjoy looking at the recorded image 60, the recorded image 60 can also be a catalyst for interaction between users. For example, the broadcasting user 13A can use the recorded image 60 as a topic of conversation. For example, the broadcasting user 13A can explain the content of the broadcast episode in which the recorded image 60 was recorded, or ask the viewing user whether they have watched that broadcast episode. A viewing user 13B who is watching a video of the broadcasting user 13A for the first time can understand the trends in the videos of the broadcasting user 13A by viewing the recorded image 60. A viewing user 13B who repeatedly watches the videos of the broadcasting user 13A can obtain more information about their favorite broadcasting user 13A by viewing the recorded image 60.

[0102] Furthermore, when displaying a recorded image 60 from a past broadcast, the broadcast user 13A may select the recorded image 60 they want to display from a list of images. However, in video broadcasting, the broadcast user 13A needs to pay attention to their own performance and the gift objects and messages provided by the viewing user 13B. For this reason, as described above, the output control unit 17 automatically displays a recorded image 60 corresponding to the current broadcast situation on the image display object 36, thereby providing a highly convenient function for the broadcast user 13A. It is also possible to display a recorded image 60 that the broadcast user 13A does not remember. Furthermore, since the recorded image 60 is displayed on an image display object 36 that resembles furniture or miscellaneous goods, it can blend in with other physical objects in the virtual space 30.

[0103] The effects of the above embodiment will be described. (1) By automatically displaying a recorded image 60 generated in the past in the image display object 36 when distributing a video, it is possible to prevent the video distributed by the distribution user 13A from becoming monotonous without increasing the burden on the distribution user 13A.

[0104] (2) Since the recorded image 60 is selected according to the distribution status of the video, it is possible to display the recorded image 60 that is appropriate for that distribution. Therefore, since the recorded image 60 can be changed for each distribution, it is possible to distribute a wide variety of videos.

[0105] (3) When a combination of multiple broadcast users 13A in a collaborative broadcast matches, in part or in whole, a combination of broadcast users 13A in a past broadcast, a recorded image 60 of the past broadcast is displayed on the display 109. This allows broadcast users 13A to look back on past broadcasts.

[0106] (4) If a keyword recognized from the voice of broadcast user 13A is included in attribute information 25, select a recorded image 60 associated with that attribute information 25. This makes it possible to display a recorded image 60 that matches the intention of broadcast user 13A without the effort of searching.

[0107] (5) When the broadcast user 13A participates in an event, if the attribute information 25 contains information about a past event associated with that event, the broadcast user 13A selects the recorded image 60 associated with that attribute information 25. This allows the broadcast user 13A to look back on past events.

[0108] (6) If the attribute information 25 contains identification information for the broadcasting room 31 or world in which the broadcasting user 13A broadcasts, the recorded image 60 associated with that attribute information 25 is selected. This allows the broadcasting user 13A to look back on past broadcasts.

[0109] (7) If the message or stamp sent by the viewing user 13B contains a keyword included in the attribute information 25, a recorded image 60 associated with the attribute information 25 is selected. Therefore, a recorded image 60 that meets the desires of the viewing user 13B can be displayed.

[0110] (8) If the gift object 55 requested to be displayed by the viewing user 13B matches the identification information of the gift object 55 included in the attribute information 25, the recorded image 60 associated with the attribute information 25 is selected. This allows the distribution user 13A to look back on past distributions.

[0111] The above-described embodiments can be modified as follows: The above-described embodiments and the following modifications can be combined with each other within the scope of technical compatibility. (Selection conditions) In the above embodiment, when the selection condition is satisfied (step S2: YES), the selection unit 16 selects the recorded image 60. Alternatively, the selection unit 16 may select the most appropriate recorded image 60 when it is determined that it is time to acquire the recorded image 60.

[0112] The timing for acquiring the recorded image 60 may be when the video distribution starts, when a predetermined time has elapsed since the start of the video distribution, when the number of comments by the distribution user 13A decreases, or when the number of viewing users 13B or the number of messages decreases. The selection unit 16 compares the distribution status acquired in step S1 with the attribute information 25 and identifies attribute information 25 that has a high similarity to the distribution status. For example, when the selection unit 16 determines to acquire the recorded image 60, it calculates a feature amount (feature vector) that quantifies the distribution status according to an evaluation axis included in the attribute information 25. The evaluation axis includes the distribution space, objects included in the recorded image 60, and the avatar object 51 appearing in the recorded image 60. Similarly, the selection unit 16 (or the generation unit 15) calculates the feature amount (feature vector) of each recorded image 60 based on the attribute information 25 of the recorded image 60. The selection unit 16 then compares the feature amount of the distribution status with the feature amount of the recorded image 60 and acquires the one with a high similarity. The similarity may be determined, for example, by a determination using Euclidean distance or cosine similarity. According to this, for example, when a broadcast is performed in a newly created world with a predetermined theme such as "Aquarium," even if there is no past broadcast history in the same world, a recorded image 60 generated during a broadcast in a world highly similar to the world (e.g., a "Zoo" world) is selected. Furthermore, if a gift object that is a "stuffed rabbit" is provided by the viewing user 13B during a broadcast, a recorded image 60 including an image of a gift object highly similar to the gift object (e.g., a "stuffed bear" gift object) is selected. Furthermore, for example, when the broadcasting user 13A is performing a collaborative broadcast with "User 1" in a collaborative broadcast, a recorded image 60 including an image of an avatar object 51 of another user highly similar to "User 1" is selected. "Other users highly similar" include users who have a friendly relationship with "User 1" (users who have registered each other as "friends," etc.), users who have the same broadcasting trends as "User 1," and users who wear the same items as "User 1."Furthermore, the selection unit 16 may select a recorded image 60 that is highly similar to the playing situation at the time of generating the recorded image 60 when the user 13 plays a game, not limited to video distribution including the avatar object 51. For example, when the user 13 defeats an enemy by performing a predetermined action (technique), the generation unit 15 may select a recorded image 60 that includes, as attribute information 25, the performance of the predetermined action or an action similar to the predetermined action.

[0113] (movie) In the above embodiment, the broadcasting user device 12A broadcasts a video including an avatar object 51. The video does not necessarily have to include the avatar object 51. In this case, the broadcasting user 13A changes the field of view of the virtual camera to explore a virtual space such as a world. Each user device 12 outputs objects in the virtual space by including them on the virtual space screen. The broadcasting user device 12A may also broadcast a video of a game being played. In this case, the viewing user device 12B obtains video data and audio data for rendering game images from the management server 11. The viewing user device 12B renders the game images based on the video data.

[0114] (Selection section) In the above embodiment, the selection unit 16 is provided in the user device 12. Alternatively, the management server 11 may have some or all of the functions of the selection unit 16.

[0115] (Output control section) In each of the above embodiments, the output control unit 17 of the user device 12 distributes and views videos using the client rendering method. Alternatively, distribution and viewing may be performed using any of the above methods other than the client rendering method. The browser rendering method is a method in which the distributor user device 12A and the viewer user device 12B display videos using independent web browsers stored in the storage 104. In this case, the distributor user device 12A transmits video data and the like to the management server 11. The user device 12 displaying the videos receives web data from the management server 11, which is written in a markup language such as HTML (Hyper Text Markup Language) and uses CSS (Cascading Style Sheets), JavaScript (registered trademark), and the like. The web browser program launched on the user device 12 renders avatar objects and other objects using JavaScript running on the browser. In this method, model data of avatar objects and the like is not stored in the user device 12. The browser rendering method does not require an application to be installed on the user device 12. Alternatively, the user device 12 may have an application including an in-app browser installed.

[0116] In the browser rendering method, the file that displays the virtual space screen can be generated by rendering using at least one of the following well-known methods: static rendering (Static Server-Side Rendering), server-side rendering (Server-Side Rendering), client-side rendering (Client-Side Rendering), pre-rendering (CSR with Prerendering), hydration rendering (SSR with (Re)Hydration), etc. Rendering here refers to interpreting data written in a markup language or the like and calculating the layout of image elements and text to be displayed on the virtual space screen.

[0117] In static rendering, a web server stores static HTML files in advance. The web server may be included in the management server 11. The user device 12 downloads the static HTML files from the web server. The user device 12 displays the rendered HTML files on the display 109 to display a virtual space screen. In server-side rendering (SSR), the web server performs rendering and other operations in response to a request from the user device 12 and transmits the rendered HTML file to the user device 12. The user device 12 displays the downloaded HTML file on the display 109 to display a virtual space screen. The web server also includes an application server. In client-side rendering (CSR), the user device 12 performs rendering by executing JavaScript (registered trademark). At this time, the user device 12 downloads a simple HTML file from the web server, as well as necessary JavaScript (registered trademark), CSS files, etc. In pre-rendering, which is a type of CSR, the user device 12 generates and stores some HTML files in advance and downloads and renders the necessary content from the web server upon request. In hydration rendering, the web server generates and stores HTML files that have not yet been rendered. The user device 12 downloads the HTML file from the web server and downloads the necessary data based on the HTML file for rendering. Note that in each of the above methods, the user device 12 may obtain the necessary data from the management server 11 instead of the web server.

[0118] The video distribution method is a method in which the broadcaster user device 12A generates video data. Specifically, the broadcaster user device 12A draws objects using motion data and the like, almost similar to the image processing in the client rendering method, and generates video data. The broadcaster user device 12A also transmits the encoded video data and audio data to the management server 11. The management server 11 transmits the video data and audio data to the viewing user device 12B. The viewing user device 12B displays video on the display 109 based on the video data received from the broadcaster user device 12A.

[0119] The server distribution method is a method in which the management server 11 generates video data. The management server 11 receives motion data and audio data from the broadcasting user device 12A. The management server 11 then generates output data for outputting video based on this data. The management server 11 transmits the generated output data to the broadcasting user device 12A and the viewing user device 12B. The broadcasting user device 12A and the viewing user device 12B output video to the display 109 based on the received output data.

[0120] In this way, either the user device 12 or the management server 11 is responsible for generating data for outputting moving images. The data to be generated differs depending on the distribution method. The device responsible for generating the data can be changed depending on the distribution method.

[0121] The virtual space 30 may be a space where events such as live performances or matches are held, as long as the virtual space screen is displayed. Alternatively, the virtual space 30 may be a virtual store or an entertainment facility such as an amusement park. Alternatively, the virtual space 30 may be a space where communication between users 13 is possible.

[0122] (Avatar Object) In the above embodiments, the user device 12 renders the avatar object 51 corresponding to the broadcast user 13A using tracking data or emote data that detects the broadcast user 13A's movements. This allows the movements of the broadcast user 13A or the movements selected by the broadcast user 13A to be reflected in the avatar object 51. Alternatively or additionally, the user device 12 may temporarily or continuously render the avatar object 51 without using tracking data or emote data. The output control unit 17 controls the avatar object 51 based on an input operation by the user 13. The avatar object 51 performs an action in response to an instruction input by the user 13 to the input device 110. This aspect reduces the rendering processing load on the user device 12 when rendering a large number of avatar objects 51 in the virtual space 30.

[0123] (User) In the above embodiments, the broadcasting user 13A refers to a user who transmits at least one of information related to images (video) and information related to audio. For example, the broadcasting user 13A is a user who broadcasts videos independently (first embodiment) or a user who performs collaborative broadcasting in which multiple people can participate (second embodiment). Alternatively, the broadcasting user 13A may be a user who hosts or organizes a video chat or voice chat in which multiple people can participate and / or watch, or an event (party, etc.) in a virtual space in which multiple people can participate and / or watch. In other words, the broadcasting user 13A can also be referred to as a host user, organizer user, or organizer user.

[0124] On the other hand, the viewing user 13B refers to a user who receives at least one of information related to video and information related to audio. However, the viewing user 13B not only receives the information but also reacts to the information. For example, the viewing user 13B is a user who watches a video broadcast or a collaboration broadcast. Alternatively, the viewing user 13B may be a user who participates in and / or watches a video chat, a voice chat, or an event. Therefore, the viewing user 13B can also be referred to as a guest user, a participating user, a listener, a viewing user, a supporting user, etc.

[0125] (Information Processing System) In the above embodiment, the information processing system was described as one in which a broadcasting user 13A broadcasts a video. Alternatively, the information processing system may provide a multiplayer game in which multiple users 13 participate online, such as a multiplayer online role-playing game (MORPG). In this embodiment, the recorded image 60 selected by the user device 12 or the management server 11 can be displayed on an image display object 36 located in the virtual space, allowing the recorded image 60 to be shared among the users 13 participating in the game.

[0126] For example, the recorded image 60 may be generated when one user 13 receives an object for assistance from the other user 13. The recorded image 60 may be generated when the user 13 accomplishes a task, wins or loses against an opponent, or performs a predetermined action (technique), etc.

[0127] Furthermore, for example, in a mode in which a group consisting of multiple users 13 is formed and the group plays a game, the recorded images 60 may be shared during periods other than game play (non-play periods). For example, when a virtual space (lobby or plaza) is displayed on the user device 12 before the game starts, the recorded images 60 may be displayed on an image display object 36 located in the virtual space. The recorded images 60 may be still images or videos of past matches, still images of a group of game media (deck) consisting of items or multiple game media used in past matches, still images showing the positions (roles) of each user 13 in past matches, etc. The recorded images 60 may be accumulated and displayed in chronological order. For example, the recorded images 60 may be displayed in order from the newest to the oldest, or in reverse order. The user 13 may perform a predetermined operation to pause, fast-forward, or fast-rewind the recorded images 60, which are videos. By checking the shared recorded image 60, the user 13 can plan a strategy for the game to be started later or review past match history, thereby increasing motivation to participate in the online multiplayer game. Game media include any media such as cards, items, virtual currency (medals or coins), tickets, characters, avatar objects, etc. The manner of use of the game media is not limited to those explicitly stated in this specification.

[0128] The management server 11 may be a single server device equipped with multiple virtual servers, or may be a set of multiple physical servers. For example, the management server 11 may be a combination of a server having a distribution management unit 20, a server having a user information management unit 21, a server having a user management information storage unit 22, and a server having an asset storage unit 23.

[0129] In the above embodiment, the user device 12 stores and selects the recorded image 60. Alternatively or additionally, the management server 11 may request the generation of the recorded image 60 and select it.

[0130] When the management server 11 generates the recorded image 60, if the management server 11 determines that the conditions for taking a screenshot have been met, it sends a request to take a screenshot to the broadcast user device 12A. For example, the conditions for taking a screenshot may be that the number of viewing users 13B currently viewing has increased to a predetermined number, that a predetermined number of gift objects have been provided within a predetermined period of time, or that the cumulative broadcast time of the broadcast user 13A within a predetermined period of time has reached a predetermined time. Alternatively, the condition for taking a screenshot may be that the broadcast user 13A has completed a predetermined task while broadcasting a video. The task may be an in-game task (mission). Upon receiving the request to take a screenshot, the broadcast user device 12A may output a notification to the display 109 prompting the broadcast user 13A to take a screenshot. After confirming the notification, the broadcast user 13A performs an operation to take a screenshot. Alternatively, upon receiving the request to take a screenshot, the broadcast user device 12A may automatically take a screenshot without any operation by the broadcast user 13A. For example, when the generation unit 15 receives a request to execute a screenshot transmitted from the management server 11, the generation unit 15 acquires the above information and stores the attribute information 25 in the recorded image storage unit 19 in association with the recorded image.

[0131] In an aspect in which the management server 11 stores the recorded image 60, when the user device 12 generates the recorded image 60, it transmits the user ID, the recorded image 60, and the attribute information 25 to the management server 11. The management server 11 stores the recorded image 60 and the attribute information 25 in association with the user ID. The management server 11 also acquires the distribution status (step S1). At this time, the management server 11 may acquire the distribution status by analyzing the video, or may acquire the distribution status from the user device 12. The management server 11 then determines whether the distribution status satisfies the selection conditions, and if the selection conditions are satisfied, acquires the recorded image 60 and transmits it to each user device 12.

[0132] In the above embodiment, the broadcast user device 12A stored the recorded image 60 in the recorded image storage unit 19. Alternatively or in addition to this, the viewing user device 12B may store the recorded image 60 in the recorded image storage unit 19 in association with the identification information of the broadcast user 13A. When the viewing user 13B is watching a video broadcast by the broadcast user 13A, the selection unit 16 of the viewing user device 12B may compare the broadcast status with the recorded image 60 associated with the identification information of the broadcast user 13A. The selection unit 16 may then acquire the recorded image 60 whose broadcast status satisfies the selection condition. The selection unit 16 transmits the acquired recorded image 60 to the broadcast user device 12A via the management server 11.

[0133] In the above embodiments, the user device 12 is an information processing device such as a smartphone, a mobile phone, a tablet terminal, a personal computer, a console game machine, or a wearable computer such as a head-mounted display. Alternatively, the information processing system may be a system installed in a video distribution studio. The information processing system includes an information processing device, a server, a sensor unit worn on the body of the distribution user 13A, a tracking system that detects the position of the sensor unit, an operation unit, a speaker, a display, etc. The tracking system may include a multi-axis laser emitter that emits pulsed laser light for synchronization. The sensor unit includes a sensor that detects laser light and detects its own position and orientation while synchronizing with the synchronization pulse.

[0134] Next, the technical ideas that can be understood from the above-described embodiment and other examples will be described below. [1] An output control unit that displays a video of a virtual space distributed by a user on a display device; a generating unit that associates a recorded image generated using at least a part of the moving image with attribute information of the recorded image and stores the associated recorded image in a storage unit; a selection unit that uses the attribute information of the recorded image to select the recorded image from the recorded images stored in the storage unit based on a selection condition, The output control unit displays the selected recorded image on an image display object located in the virtual space during the period in which the video is being distributed.

[0135] [2] The information processing system according to [1], wherein the selection unit selects the recorded image based on the selection condition that the distribution status of the video matches the attribute information. [3] The output control unit displays avatar objects corresponding to the plurality of users who distribute the video, including the video; The information processing system described in [1] or [2], wherein the selection unit selects the recorded image based on the selection condition that the user combination and the user combination indicated by the attribute information match in part or in whole.

[0136] [4] An information processing system according to any one of [1] to [3], wherein the output control unit recognizes the user's voice, and the selection unit selects the recorded image based on the selection condition that a keyword from the voice recognition result is included in the attribute information.

[0137] [5] An information processing system described in any one of [1] to [4], wherein the selection unit selects the recorded image based on the selection condition that the attribute information contains information about a past event associated with the event in the virtual space.

[0138] [6] An information processing system described in any one of [1] to [5], wherein the selection unit selects the recorded image based on the selection condition that the user distributing the video will again distribute in the virtual space where the user has previously distributed.

[0139] [7] An information processing system described in any one of [1] to [6], which selects the recorded image based on the selection condition that a keyword included in the attribute information is associated with a message or stamp sent by a viewing user watching a video.

[0140] [8] An information processing system described in any one of [1] to [7], wherein the selection unit selects the recorded image based on the selection condition that a gift object that was previously provided in the user's video is provided again. [Explanation of symbols]

[0141] 10...Information Processing Systems 11...Administration server 12...User device

Claims

1. an output control unit that displays, on a display device, virtual space images synchronized among a plurality of user devices; a generation unit that associates a recorded image generated using at least a part of the virtual space image with attribute information of the recorded image and stores the associated recorded image in a storage unit; a selection unit that uses the attribute information of the recorded image to select the recorded image from the recorded images stored in the storage unit based on a selection condition, The output control unit displays the selected recorded image on an image display object located in the virtual space during a period in which the virtual space image is displayed.

2. the output control unit displays the video of the virtual space distributed by the user on the display device; The information processing system according to claim 1 , wherein the selection unit selects the recorded image based on the selection condition that the distribution status of the video matches the attribute information.

3. the output control unit displays avatar objects corresponding to a plurality of users who distribute the video, which is the virtual space image, by including the video in the video; The information processing system according to claim 1 , wherein the selection unit selects the recorded image based on the selection condition that the combination of users and the combination of users indicated by the attribute information match in part or in whole.

4. the output control unit recognizes the user's voice; The information processing system according to claim 1 , wherein the selection unit selects the recorded image based on the selection condition that a keyword included in the speech recognition result is included in the attribute information.

5. The information processing system according to claim 1 , wherein the selection unit selects the recorded image based on the selection condition that the attribute information includes information about a past event associated with the event in the virtual space.

6. The information processing system according to claim 1, wherein the selection unit selects the recorded image based on the selection condition that the user distributing the video, which is the virtual space image, will again distribute the video in the virtual space where the user previously distributed the video.

7. The information processing system of claim 1, wherein the recorded image is selected based on the selection condition that a keyword included in the attribute information is associated with a message or stamp sent by a user watching the video, which is the virtual space image.

8. The information processing system according to claim 1 , wherein the selection unit selects the recorded image based on the selection condition that a gift object previously provided by a user watching the video that is the virtual space image is provided again.

9. One or more computers A process of displaying a virtual space image synchronized among a plurality of user devices on a display device; a process of storing a recorded image generated using at least a part of the virtual space image and attribute information of the recorded image in a storage unit in association with each other; using the attribute information of the recorded image, execute a process of selecting the recorded image from the recorded images stored in the storage unit based on a selection condition; An information processing method, wherein in the process of displaying the virtual space image on the display device, the selected recorded image is displayed on an image display object located in the virtual space during the period in which the virtual space image is displayed.

10. one or more computers, an output control unit that displays, on a display device, virtual space images synchronized among a plurality of user devices; a generation unit that associates a recorded image generated using at least a part of the virtual space image with attribute information of the recorded image and stores the associated recorded image in a storage unit; a selecting unit that uses the attribute information of the recorded image to select the recorded image from the recorded images stored in the storage unit based on a selection condition; The output control unit displays the selected recorded image on an image display object located in the virtual space during a period in which the virtual space image is displayed.

Images