Information processing system and information processing method
The system enhances user engagement and entertainment quality by generating and displaying virtual spaces before, during, and after content performance, offering interactive experiences and item purchases, thus improving the overall experience.
Patent Information
- Application Number
- JP2023566025
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-12-10
- Publication Date
- 2024-11-27
- Estimated Expiration
- 2041-12-10
AI Technical Summary
Existing image display systems for head-mounted displays fail to enhance user engagement and entertainment quality before, during, and after content consumption, such as concerts or live performances.
An information processing system and method that generates and sequentially displays virtual spaces before, during, and after content performance, allowing users to interact and engage through avatars, purchase items, and communicate nonverbally, enhancing the overall experience.
Improves user engagement and entertainment quality by providing a immersive experience similar to real-space events, promoting item sales and maintaining group cohesion through consistent virtual interactions.
Smart Images

Figure 0007591163000001 
Figure 0007591163000002 
Figure 0007591163000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a data processing technology, and more particularly to an information processing system and an information processing method. [Background technology]
[0002] Image display systems that allow users wearing head-mounted displays to view a target space from any viewpoint are becoming widespread. For example, electronic content that realizes Virtual Reality (VR) by displaying a virtual three-dimensional space as the display target and displaying an image on the head-mounted display according to the user's line of sight is known. Using a head-mounted display can also increase the sense of immersion in the video and improve the operability of applications such as games.
[0003] Currently, there are services being provided that distribute video content such as concerts (for example, live performances or film concerts) to users' terminals and allow the users to view the video content. Summary of the Invention [Problem to be solved by the invention]
[0004] In order to increase the satisfaction of users who view content, the inventors considered that while the quality of the content is of course important, it is also important to improve the entertainment value of the content from before to after it is performed.
[0005] The present invention has been made in view of the above problems, and one object of the present invention is to provide a technique for improving the entertainment value of content from before to after the performance. [Means for solving the problem]
[0006] In order to solve the above problems, an information processing system of one aspect of the present invention includes a first generation unit that generates an image of a first room, which is a virtual space where multiple users gather to watch a content before the performance of a specified content, a second generation unit that generates an image of a second room, which is the virtual space where the content is performed, a third image generation unit that generates an image of a third room, which is a virtual space where multiple users gather after the content is performed, and an output unit that sequentially displays the image of the first room, the image of the second room, and the image of the third room on a display unit.
[0007] Another aspect of the present invention is an information processing method, which includes the steps of: generating an image of a first room, which is a virtual space where a plurality of users who will watch a content gather, before a performance of a predetermined content, generating an image of a second room, which is a virtual space where the content is performed, generating an image of a third room, which is a virtual space where a plurality of users gather, after the performance of the content, and displaying the image of the first room, the image of the second room, and the image of the third room sequentially on a display unit, by a computer.
[0008] In addition, any combination of the above components, and any expression of the present invention converted between an apparatus, a computer program, a recording medium on which a computer program is readably recorded, a data structure, etc. are also valid as aspects of the present invention. Effect of the Invention
[0009] According to the present invention, it is possible to improve the entertainment value of content from before the performance to after the performance. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram showing a configuration of a content viewing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the appearance of the HMD shown in FIG. [Diagram 3] 2 is a diagram showing the internal circuit configuration of the HMD in FIG. [Figure 4]FIG. 2 is a block diagram showing functional blocks of the HMD shown in FIG. [Diagram 5] 2 is a block diagram showing functional blocks of the server in FIG. 1. [Figure 6] 4 is a flowchart showing an operation of the content viewing system. [Figure 7] FIG. 13 is a diagram showing an example of pre-performance lobby footage. [Figure 8] FIG. 1 is a diagram showing an example of group arrangement in a live concert venue. [Figure 9] FIG. 1 is a diagram showing an example of group arrangement in a live concert venue. [Figure 10] FIG. 13 is a diagram showing an example of a video of a live concert venue. [Figure 11] FIG. 13 is a diagram showing an example of a video of a live concert venue. [Figure 12] FIG. 13 is a diagram showing an example of a video of a live concert venue. [Figure 13] FIG. 13 is a diagram showing an example of post-performance lobby footage. [Figure 14] FIG. 13 shows examples of photographs provided in the lobby after the performance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] The subject of the system, device, or method in the present disclosure includes a computer. The computer executes a program to realize the function of the subject of the device or method in the present disclosure. The computer includes a processor that operates according to a program as a main hardware configuration. The type of the processor is not important as long as it can realize the function by executing the program. The processor may be composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or an LSI. The electronic circuits may be integrated into one chip or may be provided on multiple chips. The multiple chips may be integrated into one device or may be provided on multiple devices. The program may be recorded in a non-transitory recording medium such as a computer-readable ROM, an optical disk, or a hard disk drive. The program may be stored in the recording medium in advance or may be supplied to the recording medium via a wide area communication network including the Internet. In this specification or claims, terms such as "first", "second", etc. do not indicate any order or importance, but are intended to distinguish one configuration from another configuration, unless otherwise specified.
[0012] In the embodiment, a content viewing system is proposed that provides a novel user experience by providing a user with a lead that spans the period before, during, and after a performance of electronic content, and is similar to an event viewing in real space. In the content viewing system of the embodiment, video content is distributed to head mounted displays (hereinafter also referred to as "HMDs") worn on the heads of multiple users, and the multiple users are allowed to view the video content simultaneously.
[0013] Fig. 1 shows the configuration of a content viewing system 10 according to an embodiment. The content viewing system 10 includes a plurality of HMDs 100 (HMDs 100a to 100f are shown in Fig. 1) used by a plurality of users, and a server 12. These devices are connected via a communication network 14 which may include a LAN, a WAN, the Internet, or the like.
[0014] The server 12 is an information processing device that distributes video content to the HMD 100 and also controls various effects related to the viewing of the video content. The video content distributed by the server 12 may include videos of various categories and genres, but in the embodiment, the server 12 distributes video content filmed from a concert (hereinafter referred to as "live video"). A detailed description of the server 12 will be given later.
[0015] The HMD 100 is a head-mounted display device that is worn on the user's head and displays VR images. In the VR space displayed by the HMD 100, live video provided by the server 12 is displayed, and avatars (which can also be called characters) of multiple users are displayed over the period before, during, and after the performance of the live video.
[0016] Fig. 2 shows an example of the appearance of the HMD 100 in Fig. 1. The HMD 100 is composed of an output mechanism 102 and a wearing mechanism 104. The wearing mechanism 104 includes a wearing band 106 that is worn by the user and goes around the head to secure the device.
[0017] The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 100, and includes a display panel inside so as to face the eyes when the user wears the HMD 100. The display panel of the HMD 100 in the embodiment is not transparent. In other words, the HMD 100 in the embodiment is a light-opaque head-mounted display.
[0018] The housing 108 may further include an eyepiece lens that is positioned between the display panel and the user's eyes when the HMD 100 is worn, and expands the user's viewing angle. The HMD 100 may further include speakers and earphones at positions that correspond to the user's ears when the HMD 100 is worn. The HMD 100 also includes a motion sensor that detects the translational and rotational movements of the head of the user wearing the HMD 100, and thus the position and posture at each time.
[0019] The HMD 100 also includes a stereo camera 110 on the front of the housing 108. The stereo camera 110 captures video of the surrounding real space in a field of view that corresponds to the user's line of sight. If the captured image is displayed instantly, the user can see the real space in the direction in which he or she is facing, which is known as video see-through. Furthermore, if a virtual object is drawn on the image of a real object captured in the captured image, augmented reality (AR) can be realized.
[0020] Fig. 3 shows the internal circuit configuration of the HMD 100 in Fig. 1. The HMD 100 includes a CPU 120, a GPU 121, a main memory 122, a storage 123, a display unit 124, and an audio output unit 126. These units are connected to each other via a bus 128. An input / output interface 130 is further connected to the bus 128. To the input / output interface 130, a communication unit 132 including an interface for wireless communication, a motion sensor 134, a microphone 136, and a stereo camera 110 are connected.
[0021] The CPU 120 processes information acquired from each unit of the HMD 100 via the bus 128, and also supplies video data and audio data acquired from the server 12 to the display unit 124 and audio output unit 126. The GPU 121 executes image processing based on instructions from the CPU 120. For example, the GPU 121 generates data of a VR image showing a VR space to be displayed on the display unit 124.
[0022] The main memory 122 stores programs and data necessary for the processing in the CPU 120 and the GPU 121. The storage 123 stores an application program for viewing live video distributed from the server 12 (hereinafter, also referred to as a "live viewing App").
[0023] The display unit 124 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image in front of the eyes of a user wearing the HMD 100. The display unit 124 may achieve stereoscopic vision by displaying a pair of stereo images in areas corresponding to the left and right eyes. The display unit 124 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the HMD 100 is worn, and that expand the user's viewing angle.
[0024] The audio output unit 126 is composed of speakers or earphones provided at positions corresponding to the user's ears when the HMD 100 is worn, and allows the user to hear sounds.
[0025] The communication unit 132 includes a peripheral device interface such as USB or IEEE1394, and a network interface such as a wired LAN or wireless LAN. The communication unit 132 transmits and receives data to and from the server 12 via an access point (not shown). For example, the communication unit 132 may connect to an access point using a known wireless communication technology such as Wi-Fi (registered trademark), and communicate with the server 12 via the access point. The communication unit 132 also communicates with a controller (not shown) held by the user's hand using a known wireless communication technology.
[0026] The motion sensor 134 includes a gyro sensor and an acceleration sensor, and acquires the angular velocity and acceleration of the HMD 100. The microphone 136 receives sounds around the user and sounds emitted by the user, and converts the sounds into electrical signals (also called "audio data").
[0027] As shown in FIG. 2, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints in a field of view corresponding to the user's viewpoint. An image captured by the stereo camera 110 and showing the space around the user is also referred to as a "camera image" below. The camera image can also be said to be an image showing an object existing in the user's line of sight (typically in front of the user). Measurement values by the motion sensor 134 and data on the captured image (camera image) by the stereo camera 110 are transmitted to the server 12 via the communication unit 132 as necessary. Note that there is no limit to the number of cameras mounted on the HMD 100, and one camera or three or more cameras may be mounted.
[0028] Fig. 4 is a block diagram showing the functional blocks of the HMD 100 in Fig. 1. The multiple functional blocks shown in the block diagram in this specification can be realized in terms of hardware by a configuration of a computer processor (CPU, GPU, etc.), memory, storage, etc., and in terms of software by a computer program that implements the functions of the multiple functional blocks. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.
[0029] The HMD 100 includes a control unit 20, a storage unit 22, and a communication unit 132. The control unit 20 may be realized by a CPU 120 and a GPU 121, and executes various data processing. The control unit 20 transmits and receives data to and from an external device (e.g., a controller or a server 12) via the communication unit 132. The storage unit 22 may be realized by a main memory 122 and a storage 123, and stores data referenced or updated by the control unit 20.
[0030] The storage unit 22 includes a room data storage unit 24, an item data storage unit 26, and a user data storage unit 28. The room data storage unit 24 stores data (also called "room data") relating to a plurality of virtual spaces from before the performance of the live video to after the performance. The room data includes data that specifies, for example, the layout, shape, pattern, color, etc. of each virtual space.
[0031] In the embodiment, the multiple virtual spaces include a pre-performance lobby as a first room, a live venue as a second room, and a post-performance lobby as a third room. The pre-performance lobby is a virtual space where multiple users who watch live footage gather before the performance of the live footage. For example, in the pre-performance lobby, users may voice chat with their groupmates while reviewing how to enjoy the live performance (such as choreography and cheering methods).
[0032] The live venue is a virtual space where live footage is shown and multiple users watch the live footage. The post-performance lobby is a virtual space where multiple users who have watched the live footage gather after the performance of the live footage. For example, in the post-performance lobby, users share their impressions of the live performance with their friends in the same group through voice chat, enjoying the afterglow of the live performance. The pre-performance lobby and the post-performance lobby may be the same virtual space as the lobby, and may have the same layout, shape, pattern, color, etc.
[0033] Furthermore, multiple avatars corresponding to multiple users appear in each of the pre-performance lobby, the live venue, and the post-performance lobby. In the embodiment, the actions that the user's avatar can perform in the pre-performance lobby are different from the actions that the user's avatar can perform in the post-performance lobby. Specifically, in the pre-performance lobby, the user's avatar can purchase items for cheering on the live performance in response to the user's operation, and can practice choreography and cheering for the live performance using handclaps and items. Meanwhile, in the post-performance lobby, the user's avatar can view and purchase photos of the live performance in response to the user's operation.
[0034] The item data storage unit 26 stores data (also called "item data") on a plurality of items that are virtual items that can be purchased by the user and that can be used by the user's avatar. The item data includes, for example, data defining the price and presentation (effects, etc.) of the item, and data defining the performer that the item supports.
[0035] The user data storage unit 28 stores data (also referred to as "user data") related to the user who wears the HMD 100. The user data includes, for example, identification information of the user, identification information of a group to which the user belongs, and data of items owned (purchased) by the user. In this embodiment, a maximum of four users can belong to one group.
[0036] The control unit 20 includes an operation reception unit 30, a position and orientation detection unit 32, a room switching unit 34, an avatar operation determination unit 36, a purchase processing unit 38, a video generation unit 40, a video acquisition unit 48, a video output control unit 50, an audio reception unit 52, a server cooperation unit 54, an audio output control unit 56, and a photo storage unit 58. The functions of these multiple functional blocks may be implemented in a live viewing App. The processor (CPU 120 and GPU 121) of the HMD 100 may fulfill the functions of the multiple functional blocks of the control unit 20 by reading the live viewing App into the main memory 122 and executing it.
[0037] The operation reception unit 30 receives data indicating an operation input by a user to the controller, the data being transmitted from the controller, and passes the data indicating the user's operation to each functional block of the control unit 20.
[0038] The position and orientation detection unit 32 detects the position and orientation of the head of the user wearing the HMD 100, based on the angular velocity and acceleration of the HMD 100 detected by the motion sensor 134 and the image of the surroundings of the HMD 100 captured by the stereo camera 110. The position and orientation detection unit 31 further detects the position of the user's viewpoint and the line of sight, based on the position and orientation of the user's head.
[0039] The room switching unit 34 is a virtual space to be displayed, in other words, executes a process of switching the virtual space to be image-generated. For example, the room switching unit 34 instructs the image generating unit 40 to generate a pre-performance lobby image before the timing of the live image is reached. Also, in response to an instruction from the server 12, the room switching unit 34 instructs the image generating unit 40 to generate a live venue image in place of the pre-performance lobby image. Also, in response to an instruction from the server 12, the room switching unit 34 instructs the image generating unit 40 to generate a post-performance lobby image in place of the live venue image.
[0040] The avatar movement determination unit 36 determines the posture, movement, and behavior that the user's avatar should take based on the user's operation accepted by the operation acceptance unit 30 and the position and posture of the user's head detected by the position and posture detection unit 32.
[0041] The purchase processing unit 38 executes the purchase processing of items based on the user's operation accepted by the operation acceptance unit 30. For example, the purchase processing unit 38 may cooperate with an external payment system to settle the cost of items purchased by the user in the pre-performance lobby using payment method information (e.g., a credit card number) that the user has registered in advance.
[0042] The video generation unit 40 generates video data of the virtual space. The video generation unit 40 includes a first generation unit 42, a second generation unit 44, and a third generation unit 46. The first generation unit 42 generates video data of the pre-performance lobby using data related to the pre-performance lobby stored in the room data storage unit 24. The second generation unit 44 generates video data of the live venue using data related to the live venue stored in the room data storage unit 24. The third generation unit 46 generates video data of the post-performance lobby using data related to the post-performance lobby stored in the room data storage unit 24.
[0043] Video output control unit 50 sequentially outputs the pre-performance lobby video data generated by first generation unit 42, the live venue video data generated by second generation unit 44, and the post-performance lobby video data generated by third generation unit 46 to display unit 124. Video output control unit 50 sequentially causes display unit 124 to display the pre-performance lobby video, the live venue video, and the post-performance lobby video.
[0044] The moving image acquiring unit 48 receives moving image data of the live video transmitted from the server 12. The second generating unit 44 sets the moving image data of the live video received by the moving image acquiring unit 48 on a screen unit provided at the live venue.
[0045] The voice receiving unit 52 receives voice data input from the microphone 136. The server linking unit 54 transmits and receives data to and from the server 12. Specifically, the server linking unit 54 transmits the user's voice data input from the microphone 136 to the server 12. The server linking unit 54 further transmits data indicating the posture, movement, and behavior of the user's avatar determined by the avatar movement determination unit 36 to the server 12.
[0046] Furthermore, the server cooperation unit 54 accepts voice data of other users (in the embodiment, other users in the same group) transmitted from the server 12, and passes it to the voice output control unit 56. The voice output control unit 56 causes the voice output unit 126 to output the voice indicated by the voice data of the other users transmitted from the server 12.
[0047] Furthermore, the server cooperation unit 54 receives data indicating the posture, movement, and behavior of the avatar of the other user transmitted from the server 12, and passes it to the image generation unit 40. Each of the first generation unit 42, the second generation unit 44, and the third generation unit 46 of the image generation unit 40 reflects the posture, movement, and behavior of the avatar of the user determined by the avatar movement determination unit 36 in the VR image, and also reflects the posture, movement, and behavior of the avatar of the other user transmitted from the server 12 in the VR image. In this way, the user's avatar is moved according to the posture, movement, and operation of the user (user of the local environment), and the avatar of the other user is moved according to the posture, movement, and operation of the other user (user of the remote environment).
[0048] The photo storage unit 58 stores data of photos of the live performance or the live performance venue in the memory unit 22 (for example, a photo memory unit) in response to a user operation.
[0049] Fig. 5 is a block diagram showing functional blocks of the server 12 in Fig. 1. The server 12 includes a control unit 60, a storage unit 62, and a communication unit 64. The control unit 60 executes various data processing. The storage unit 62 stores data referenced or updated by the control unit 60. The communication unit 64 communicates with an external device according to a predetermined communication protocol. The control unit 60 transmits and receives data to and from the HMD 100 via the communication unit 64.
[0050] The storage unit 62 includes a video data storage unit 66 and a user data storage unit 68. The video data storage unit 66 stores various video data (including live video data in this embodiment) that the server 12 distributes to the HMD 100.
[0051] The user data storage unit 68 stores data on a plurality of users across a plurality of groups, which are the destinations of the video content. The data on each user includes, for example, the user's identification information, the identification information of the group to which the user belongs, and the identification information of the video content that the user views.
[0052] The control unit 60 includes a performance execution unit 70, a placement unit 72, a video distribution unit 74, a behavioral data reception unit 76, a behavioral data provision unit 78, a voice data reception unit 80, and a voice data provision unit 82. The functions of these multiple functional blocks may be implemented in a computer program and installed in the storage of the server 12. The processor (CPU, etc.) of the server 12 may fulfill the functions of the multiple functional blocks of the control unit 60 by reading the computer program into a main memory and executing it.
[0053] The performance execution unit 70 determines the performance in the pre-performance lobby, the live performance venue, and the post-performance lobby. The performance execution unit 70 transmits data indicating the determined performance content to the HMD 100 of each of the multiple users viewing the live video.
[0054] The placement unit 72 determines the position of each of the multiple users watching the live video based on the group to which each user belongs stored in the user data storage unit 68. In other words, the placement unit 72 determines the position of each user in the virtual space (the live venue in this embodiment) for each user group. The placement unit 72 transmits data indicating the position of each user to the HMD 100 of each user.
[0055] The video distribution unit 74 transmits the video data of the live video stored in the video data storage unit 66 to the HMD 100 of each of the multiple users who are watching the live video. The video data of the live video may be transferred and played back by streaming.
[0056] The behavioral data receiving unit 76 receives data indicating the posture, movement, and behavior of the avatar of each of the multiple users watching the live video, transmitted from the HMD 100. The behavioral data providing unit 78 transmits data indicating the posture, movement, and behavior of the avatar of a certain user, received by the behavioral data receiving unit 76, to the HMD 100 of the other users.
[0057] The audio data receiving unit 80 receives audio data transmitted from the HMD 100 of each of a plurality of users viewing live video. The audio data providing unit 82 refers to the data stored in the user data storage unit 68 and identifies the group to which the user who transmitted the audio data received by the audio data receiving unit 80 belongs. The audio data providing unit 82 transmits audio data of a user belonging to a certain group to the HMD 100 of another user belonging to the same group, but does not transmit the data to the HMD 100 of a user belonging to another group.
[0058] The operation of the content viewing system 10 having the above configuration will be described. 6 is a flowchart showing the operation of the content viewing system 10. The following mainly describes the operation of the HMD 100a used by user a and the server 12 when user a views live video, but other HMDs 100 also operate in the same manner as HMD 100a. User a belongs to group A, which further includes user b using HMD 100b, user c using HMD 100c, and user d using HMD 100d. User e using HMD 100e and user f using HMD 100f belong to group B, which is different from group A.
[0059] User a inputs an operation to the HMD 100a to instruct the launch of the live viewing App, and the control unit 20 of the HMD 100a launches the live viewing App. When the live viewing App is launched (Y of S10), user a selects a live show to be viewed from among multiple live shows that can be viewed (in other words, multiple live venues). User a also selects a group to join (here, group A), enters his / her name, and logs in. The server linking unit 54 of the HMD 100a transmits identification information of the live show and group selected by the user to the server 12. The user data storage unit 68 of the server 12 stores the identification information of the live show and group selected by the user in association with the user's identification information.
[0060] The room switching unit 34 of the HMD 100a instructs the image generating unit 40 to generate a pre-performance lobby image. The image generating unit 40 (first generating unit 42) of the HMD 100a generates data for the pre-performance lobby image, and the image output control unit 50 causes the display unit 124 to display the pre-performance lobby image (S11).
[0061] FIG. 7 shows an example of a pre-performance lobby image. An avatar 202 of user a is rendered in pre-performance lobby image 200. A mirror 204 is also placed in the pre-performance lobby. An avatar image 206 of user a entering the pre-performance lobby and an avatar image 208 of user b are rendered on mirror 204. The posture, movement, and behavior of user a's avatar 202 and avatar image 206 change according to the posture, movement, and behavior of user a in the real world. Similarly, the posture, movement, and behavior of user b's avatar (not shown) and avatar image 208 change according to the posture, movement, and behavior of user b in the real world.
[0062] In addition, only users b, c, and d who belong to the same group A as user a can enter the pre-performance lobby that user a enters. On the other hand, users e and f can enter the pre-performance lobby that is reserved for group B.
[0063] In the pre-performance lobby, user a practices the cheering choreography (claps here) while waiting for the live video to start. When user a claps in the real world, the operation reception unit 30 of the HMD 100 detects the clapping based on the movement of the controller or the images captured by the stereo camera 110. The first generation unit 42 of the video generation unit 40 generates a pre-performance lobby video 200 in which the avatar 202 and the avatar image 206 clap, and also generates the pre-performance lobby video 200 including effects 210 that are associated in advance with the clapping. The effects 210 include, for example, an effect in which a ring of light spreads and an effect in which star-shaped objects (which can also be called particles) fly.
[0064] In the pre-performance lobby, user a can purchase desired items to use during the live performance or to support performers from among multiple virtual items stored in item data storage unit 26. In response to user a's operation, first generation unit 42 of HMD 100a generates an image showing user a's purchase of an item as a pre-performance lobby image. Operation reception unit 30 of HMD 100a receives a user's operation to instruct the purchase of an item. Purchase processing unit 38 of HMD 100a settles the price of the item and stores data of the purchased item in user data storage unit 28 in association with user a's identification information.
[0065] In the pre-performance lobby, the user can practice cheering and choreography using the purchased items. In response to the operation of user a, the first generation unit 42 of the HMD 100a generates, as the pre-performance lobby video 200, a video in which the avatar 202 and avatar image 206 of user a cheer using the purchased items. In the embodiment, the purchasable items include a frying pan, and in the pre-performance lobby, the user can practice swinging the frying pan, etc. The pre-performance lobby video 200 continues to be displayed on each user's HMD 100 until the performance time of the live video (in other words, the timing of switching to the live venue video) (N of S12).
[0066] When the time for the live video to be shown arrives (Y in S12), the placement unit 72 of the server 12 determines the position of each of the multiple users watching the live video in the live venue based on the group of each user (S13). The position of each user in the live venue is the position where each user's avatar is placed in the live venue.
[0067] FIG. 8 shows an example of group arrangement at a live venue. The figure shows the group arrangement assigned to users in group A (user a, user b, user c, user d). FIG. 9 also shows an example of group arrangement at a live venue. The figure shows the group arrangement assigned to users in group B (user e, user f). The screen on which the live video is displayed is placed in front of the group in the front row (in front of groups E and F in FIG. 8, and in front of groups F and G in FIG. 9).
[0068] As shown in Fig. 8 and Fig. 9, in the placement data to be assigned to users of group A, the placement unit 72 places group A at a predetermined fixed position in the live venue, and places other groups at other positions in the live venue. In addition, in the placement data to be assigned to users of group B, the placement unit 72 places group B at the above-mentioned fixed position in the live venue, and places the other groups at other positions in the live venue. In the embodiment, the above-mentioned fixed position is the center of the second row, and is hereinafter also referred to as the "viewing position." In this way, the placement unit 72 assigns a fixed position suitable for viewing the live video to each of multiple users across multiple groups who watch the live video.
[0069] The placement unit 72 also places at least one other group between the viewing position in the live venue (in this embodiment, the center of the second row) and the position where the live video is being presented (in this embodiment, the position of the screen on which the live video is projected). In the example of FIG. 8, groups E and F are placed between the viewing position of group A and the screen. In the example of FIG. 9, groups F and G are placed between the viewing position of group B and the screen. This allows each user to enjoy the artist's performance while checking the behavior of the avatars of the other groups.
[0070] Furthermore, the arrangement unit 72 maintains the positional relationship between group A and group B between the arrangement data assigned to users belonging to group A and the arrangement data assigned to users belonging to group B. That is, the arrangement unit 72 aligns the relative positions of the groups between the arrangement data assigned to each group. For example, in both the group arrangement shown in FIG. 8 and the group arrangement shown in FIG. 9, group B is to the right of group A, and group F is diagonally in front of group A to the right and diagonally in front of group B to the left. This makes it easier to maintain the consistency of communication (described later) between the groups.
[0071] Once the position of each group has been determined by the placement unit 72, the performance execution unit 70 of the server 12 transmits a switching instruction to the live venue, including placement data for each group, to the HMD 100 of each user. Based on the instruction from the server 12, the room switching unit 34 of the HMD 100a inputs a switching instruction to the live venue, including placement data for group A, to the video generation unit 40. The video generation unit 40 (second generation unit 44) generates data of the live venue video in which the avatars of each group are placed at the positions indicated by the placement data for group A (e.g., FIG. 8). The video output control unit 50 causes the live venue video to be displayed on the display unit 124 (S14).
[0072] Fig. 10 shows an example of a live venue image. An avatar 202 of user a is drawn in live venue image 220. A screen 222 on which the live video is set is also placed in the live venue. An avatar 224 is an avatar of a user who belongs to a different group from user a (for example, group F shown in Fig. 8).
[0073] At the live venue, user a can communicate nonverbally with users in other groups. Nonverbal communication includes waving at the avatars of users in other groups and throwing objects (which can also be called particles). For example, in FIG. 10, user a's avatar 202 throws a hard object 226 at the avatar 224 of a user in the other group. Such actions of user a's avatar 202 are transmitted to the HMDs 100 of the other users via the server 12 and are also reflected in the live venue images 220 on the HMDs 100 of the other users.
[0074] As described above, by aligning the relative positions of the groups between the placement data assigned to each group, it is possible to align nonverbal communication between the groups. For example, when the object 226 is thrown to the right by a user of group A (i.e., toward group B), the user of group B can perceive that the object 226 has come from the left (i.e., toward group A).
[0075] Although not shown in FIG. 10, the live venue image 220 also depicts avatars of other users (users B, C, and D) who belong to the same group A as user a. User a can perform both non-verbal and verbal communication with other users in the same group. Verbal communication is specifically voice chat. The voice data providing unit 82 of the server 12 transmits the voices of users a, b, c, and d to the HMD 100 of other users in group A, but does not transmit them to the HMD 100 of users other than group A. In addition, the voice data providing unit 82 transmits the voices of users e and f to the HMD 100 of other users in group B, but does not transmit them to the HMD 100 of users other than group B. In this way, by blocking voice transmission across different groups, it is possible to prevent the voices of other groups from being perceived as a nuisance by the user.
[0076] 11 also shows an example of a live venue video. User a cheers on the artist by clapping while watching the live video. The avatar 202 of user a in the live venue video 220 claps in sync with the movements of user a. Effects 210 (particles, etc.) are displayed in the live venue video 220 in time with the clapping of the avatar 202. The behavioral data providing unit 78 of the server 12 notifies the HMD 100a of the movements of the avatars of other users, and the second generating unit 44 of the HMD 100a also displays the movements of the avatars 224 of other users in the live venue video 220.
[0077] 12 also shows an example of a live venue image. In response to an operation by user a, the second generation unit 44 of the HMD 100a generates, as live venue image 220, an image showing that an item (item 228 in this case) purchased in the pre-performance lobby is used by the avatar 202 of user a.
[0078] As described above, the items purchased in the pre-performance lobby are associated with people or characters (herein referred to as "performers") appearing in the live video. When an item purchased in the pre-performance lobby is used by a user's avatar, the second generation unit 44 of the HMD 100a generates a live venue video 220 including an action (which can also be called a reaction or feedback) toward the user from the performer associated with the item.
[0079] Specifically, the performance execution unit 70 of the server 12 identifies the item used by each user's avatar at the live venue based on data indicating the movement of each user's avatar notified from each user's HMD 100. In other words, the performance execution unit 70 identifies the user using each item. For each item, the performance execution unit 70 determines by lottery from among multiple users who are using the item, one user who will be the target of a reaction performance (which can also be called a gratitude performance) by a performer associated with the item. The performance execution unit 70 transmits a reaction performance instruction to each user's HMD 100, specifying the target user for the reaction performance determined for each item.
[0080] The second generating unit 44 of the HMD 100 generates a live venue image 220 showing a reaction from a performer to the avatar of the user indicated by the reaction performance instruction. The live venue image 220 in Fig. 12 shows that a reaction object 230 showing a performer associated with the item 228 has been delivered to the avatar 202 of the user a using the item 228 as a reaction performance. When the target user of the reaction performance to the item 228 is determined to be another user, the live venue image 220 of the user a shows that the reaction object 230 has been delivered to the avatar of the other user.
[0081] Such reaction effects can encourage users to purchase items in the lobby after the performance, and can also promote competition among users, making live viewing even more enjoyable. The reaction effects are periodically repeated. The target of multiple reaction effects may be the same user, or may be different users.
[0082] The live venue image 220 continues to be displayed on each user's HMD 100 until the performance of the live video ends (N in S15). When the performance of the live video ends (Y in S15), the production execution unit 70 of the server 12 sends an instruction to switch to the post-performance lobby to each user's HMD 100. Based on the instruction from the server 12, the room switching unit 34 of the HMD 100a inputs an instruction to switch to the post-performance lobby to the image generation unit 40. The image generation unit 40 (third generation unit 46) generates data for the post-performance lobby image. The image output control unit 50 causes the display unit 124 to display the post-performance lobby image (S16).
[0083] FIG. 13 shows an example of a post-performance lobby image. User a's avatar 202 is depicted in post-performance lobby image 240. As in the pre-performance lobby, a mirror 204 is also placed in the post-performance lobby. An avatar image 206 of user a and an avatar image 208 of user b who are both in the post-performance lobby are depicted on mirror 204. Only users b, c, and d who belong to the same group A as user a can enter the post-performance lobby that user a enters. Users e and f can enter the post-performance lobby that is reserved for group B.
[0084] In the lobby after the performance, User A can have the avatar 202 hold and check multiple photos 242 capturing multiple scenes from the video content (here, the live performance) while talking about his / her impressions of the live video with his / her friends in Group A through voice chat. Data for the photos 242 may be generated by the performance execution unit 70 of the server 12 and provided from the server 12 to the HMD 100 of each user. The third generation unit 46 of the HMD 100 may display the photos 242 provided by the server 12 in the post-performance lobby video 240.
[0085] FIG. 14 shows an example of a photo provided in the lobby after the performance. The photo 242 provided in the lobby after the performance may include a photo showing the avatar 202 of the user a at the live venue, or may include a photo showing both the performer and the avatar 202 at the live venue. The photo storage unit 58 of the HMD 100a may store data of photos taken by a virtual camera provided in advance at the live venue or a virtual camera installed at the live venue by the user a in the memory unit 22 (e.g., a photo memory unit). The user a may input a predetermined operation to the virtual camera to cause the virtual camera to take a range including the user's avatar, the avatars of other users, and the screen. The third generation unit 46 of the HMD 100a may display the photo taken by the operation of the user a stored in the memory unit 22 (e.g., a photo memory unit) in the post-performance lobby image 240.
[0086] In the post-performance lobby, user a can purchase any of a number of photographs 242 (including a photograph showing user a's avatar). In response to user a's operation, the third generation unit 46 of the HMD 100a generates an image showing user a's purchase of the photograph 242 as a post-performance lobby image. In response to the user's operation to instruct the purchase of the photograph 242, the purchase processing unit 38 of the HMD 100a settles the price of the photograph 242, and stores data relating to the purchased photograph 242 (such as identification information and image data) in the user data storage unit 28 in association with user a's identification information.
[0087] In addition, the actions that a certain user can perform in the lobby after the performance may change based on the behavior of the user at the live venue. For example, the production execution unit 70 of the server 12 may provide the HMD 100a with photos taken of the live performance or the live venue when the lobby after the performance is displayed, on the condition that the number of cheers (for example, the total number of clappings and the number of item usages) by the avatar of user a at the live venue is equal to or greater than a predetermined threshold. This allows a user who cheers a lot at the live venue to be given the privilege of being able to view photos in the lobby after the performance. In addition, the production execution unit 70 of the server 12 may provide the HMD 100a with more photos when the lobby after the performance is displayed, the more the number of cheers by the avatar of user a at the live venue is. This allows the cheering behavior of users at the live venue to be promoted.
[0088] The display of the post-performance lobby image 240 continues until User A inputs a predetermined end operation (N in S17). When User A inputs the end operation (Y in S17), the process in this figure ends. If the live viewing App is not started on the HMD 100a (N in S10), the process in S11 and after is skipped and the process in this figure ends.
[0089] According to the content viewing system 10 of the embodiment, the pre-performance lobby image 200, the live venue image 220, and the post-performance lobby image 240 are sequentially provided to the user as a guide for the user throughout the period before, during, and after the performance of the video content. This allows the user to have an experience close to that of viewing an event in real space, and improves the entertainment value throughout the period before, during, and after the performance of the video content.
[0090] The present invention has been described above based on the embodiments. These embodiments are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process, and that such modifications are also within the scope of the present invention.
[0091] A first modified example will be described. In at least one of the virtual spaces of the pre-show lobby, the live performance venue, and the post-show lobby, a special user (herein called a "guide") who is assigned the role of guiding a plurality of general users who are watching the video content may be provided, and an avatar of the guide may be placed therein. In addition, when a pre-show lobby and a post-show lobby are provided for each group, a guide avatar may be placed in the pre-show lobby and the post-show lobby for all groups.
[0092] The guide may lecture each user on how to cheer on a live show or how to use items through voice chat or gestures. The voice data providing unit 82 of the server 12 may transmit data of the voice uttered by the guide, i.e., voice data transmitted from the guide's HMD 100, to the HMD 100 of users in multiple groups, or to the HMD 100 of users in all groups. The voice data providing unit 82 of the server 12 may also transmit voice data from all users to the guide's HMD 100, regardless of the group to which each user belongs.
[0093] In this variant, an operation by the guide may trigger a video switching instruction (switching from the pre-performance lobby to the live venue, and / or switching from the live venue to the post-performance lobby) to be sent from the guide's HMD 100 to all users' HMDs 100.
[0094] A second modified example will be described. The video generating unit 40 may further include a fourth generating unit that generates a video of a My Room, which is a virtual space assigned to each user and is a fourth virtual space in which items that the user's avatar can use are collected. The My Room can be said to be a virtual space linked to each user, and can be said to be a virtual space in which items purchased by each user are collected. The fourth generating unit of the video generating unit 40 may generate a video of the My Room in which one or more purchased items stored in the user data storage unit 28 are arranged, as a video of the virtual space to be displayed first when the live viewing App is started. The video output control unit 50 may cause the display unit 124 to display the video of the My Room.
[0095] In My Room, the user can freely view items that have already been purchased. In addition, information on multiple video contents that can be viewed in the future is displayed in My Room. When the user inputs an operation to select the video content to be viewed in My Room, the room switching unit 34 of the HMD 100 instructs the video generating unit 40 to display the pre-performance lobby corresponding to the selected video content. In addition, the server linking unit 54 of the HMD 100 may transmit data (distribution request data) indicating the video content selected by the user to the server 12. Thereafter, the processing from S11 onward in the flowchart of FIG. 6 may be executed.
[0096] A third modified example will be described. In the above embodiment, live video of a concert or the like is distributed from the server 12 to the HMD 100, but the server 12 may distribute other types of video content to the HMD 100. The distributed video content may be, for example, live video of a sports event such as baseball or soccer, or video of a movie or drama. When distributing video of a movie or drama, it is preferable that the voice chat can be switched on / off according to a user's operation. For example, the voice data providing unit 82 of the server 12 may switch on / off the voice chat within a group according to a user's setting, that is, the transfer of voice data may be suppressed even between users in the same group.
[0097] A fourth modified example will be described. A part of the functions of the HMD 100 of the embodiment may be provided in the server 12. A part of the functions of the server 12 of the embodiment may be provided in the HMD 100. A part of the functions of the HMD 100 of the embodiment and / or a part of the functions of the server 12 of the embodiment may be provided in a user-side information processing device (e.g., a game console, etc.) connected to the HMD 100.
[0098] Any combination of the above-mentioned examples and modifications is also useful as an embodiment of the present disclosure. A new embodiment resulting from the combination has the effects of each of the combined examples and modifications. In addition, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be realized by each component shown in the examples and modifications alone or in cooperation with each other.
[0099] The technical ideas based on the description of the embodiments and modifications can be expressed in the aspects shown in the following items. The information processing system of each of the following items can also be expressed as an information processing device or a head mounted display. [Item 1] a first generation unit that generates an image of a first room, which is a virtual space in which a plurality of users gather to watch a predetermined content, before the content is played; a second generation unit that generates an image of a second room, which is a virtual space in which the content is performed; a third image generating unit that generates an image of a third room, which is a virtual space in which the plurality of users gather, after the performance of the content; an output unit that sequentially displays the image of the first room, the image of the second room, and the image of the third room on a display unit; An information processing system comprising: This information processing system sequentially provides the user with a first room before the content is played, a second room during the content, and a third room after the content is played, providing the user with a flow similar to that of watching an event in real space, and offering a novel user experience. [Item 2] The actions that the user's avatar can perform in the first room are different from the actions that the user's avatar can perform in the third room. Item 1. An information processing system according to item 1. According to this information processing system, in the first room, the user's avatar can be made to perform appropriate actions before the content is played, and in the third room, the user's avatar can be made to perform appropriate actions after the content is played, thereby providing the user with an experience similar to watching an event in real space. [Item 3] The first generation unit generates, in response to an operation by a user, a video showing a user purchasing an item as the video of the first room; the second generation unit generates, in response to an operation by a user, a video in which the item purchased in the first room is used by an avatar of the user, as the video of the second room. 3. The information processing system according to item 1 or 2. According to this information processing system, by enabling the use of an item while viewing content, it is possible to promote sales of the item and to increase the interest of viewing content. [Item 4] The item purchased in the first room is associated with a person or character appearing in the content; the second generation unit generates, when an item purchased in the first room is used by an avatar of a user, a video including an action toward the user from a person or character associated with the item; Item 3. An information processing system according to item 3. According to this information processing system, by providing a special privilege of receiving an action from a person or character appearing in the content, it is possible to further promote sales of goods and to further increase the interest of viewing the content. [Item 5] An action that a user can perform in the third room is changed based on an action of the user in the second room. 5. An information processing system according to any one of items 1 to 4. According to this information processing system, by obtaining feedback on the user's behavior while viewing the content after the content has been viewed, it is possible to increase the user's interest during and after viewing the content. [Item 6] In the third room, an image showing the user's avatar captured in the second room can be purchased. 6. An information processing system according to any one of items 1 to 5. According to this information processing system, it is possible to enhance interest after viewing the content. [Item 7] A placement unit that determines a position of each user in the second room based on a group of users, 7. An information processing system according to any one of items 1 to 6. According to this information processing system, by arranging users of the same group in close proximity to each other, it is possible to increase interest while viewing content. [Item 8] the placement unit places a first group at a predetermined fixed position in the second room and assigns first placement data in which another group is placed at another position in the second room to a user of the first group, and places a second group at the fixed position in the second room and assigns second placement data in which another group is placed at another position in the second room to a user of the second group, such that a positional relationship between the first group and the second group is maintained between the first placement data and the second placement data. Item 8. An information processing system according to item 7. According to this information processing system, by arranging each group in a fixed position suitable for content viewing, while aligning the relative positions of the groups between the arrangement data assigned to each group, it becomes easier to maintain consistency in communication between groups. [Item 9] the placement unit places a first group at a predetermined fixed position in the second room, assigns first placement data in which another group is placed at another position in the second room to a user of the first group, places a second group at the fixed position in the second room, assigns second placement data in which another group is placed at another position in the second room to a user of the second group, and places at least one other group between the fixed position in the second room and a position where the content is performed; Item 8. An information processing system according to item 7. According to this information processing system, each group is placed in a fixed position suitable for viewing the content, and at least one other group is placed between that fixed position and the position where the content is presented, so that the user can check the behavior of other groups in relation to the content while viewing the content. [Item 10] a voice providing unit that provides data of a voice uttered by a user belonging to a certain group to a device of another user belonging to the same group, but does not provide the data to a device of a user belonging to another group; 10. An information processing system according to any one of items 1 to 9. According to this information processing system, it is possible to suppress the provision of audio that is unnecessary for the user. [Item 11] At least one of the first room, the second room, and the third room is provided with a special user for guiding the plurality of users; The voice providing unit provides data of the voice uttered by the special user to devices of a plurality of groups of users. Item 11. An information processing system according to item 10. According to this information processing system, it is possible to suppress provision of voices that are unnecessary for the user, while exceptionally providing voices that are useful across multiple groups to multiple groups. [Item 12] A fourth generation unit is further provided for generating an image of a fourth room, which is a virtual space assigned to each individual user and in which items usable by the user's avatar are collected; When a user selects a content to be viewed in the fourth room, the first generation unit generates a video of the first room corresponding to the selected content. 12. An information processing system according to any one of items 1 to 11. According to this information processing system, a fourth room can be provided to the user as his or her own room in which the user can select content to view and check collected items, etc., thereby providing the user with an experience close to the real world. [Item 13] generating an image of a first room, which is a virtual space where a plurality of users gather to view a predetermined content, before the content is played; generating an image of a second room, which is a virtual space in which the content is performed; generating an image of a third room, which is a virtual space where the plurality of users gather, after the performance of the content; displaying the first room image, the second room image, and the third room image on a display unit in sequence; An information processing method implemented by a computer. According to this information processing method, by sequentially providing the user with a first room before the content is played, a second room during the content is played, and a third room after the content is played, it is possible to realize a user flow similar to that of watching an event in real space, providing a novel user experience. [Item 14] A function of generating an image of a first room, which is a virtual space where a plurality of users gather to watch a predetermined content, before the content is played; A function of generating an image of a second room, which is a virtual space in which the content is performed; a function of generating an image of a third room, which is a virtual space where the plurality of users gather, after the performance of the content; a function of sequentially displaying the image of the first room, the image of the second room, and the image of the third room on a display unit; A computer program that enables a computer to realize the above. According to this computer program, by sequentially providing the user with a first room before the content is played, a second room during the content is played, and a third room after the content is played, it is possible to realize a user flow similar to that of watching an event in real space, providing a novel user experience. [Industrial Applicability]
[0100] The technology disclosed herein can be applied to an information processing system or an information processing device. [Explanation of symbols]
[0101] 10 content viewing system, 12 server, 40 video generation unit, 42 first generation unit, 44 second generation unit, 46 third generation unit, 50 video output control unit, 72 placement unit, 82 audio data providing unit, 100 HMD, 124 display unit.
Claims
1. a first generation unit that generates an image of a first room, which is a virtual space in which a plurality of users who will view a predetermined content gather, before the performance of the content; a second generation unit that generates an image of a second room, which is a virtual space in which the content is performed; a third image generating unit that generates an image of a third room, which is a virtual space in which the plurality of users gather, after the performance of the content; an output unit that sequentially displays the image of the first room, the image of the second room, and the image of the third room on a display unit; An information processing system comprising:
2. An action that the user's avatar can perform in the first room is different from an action that the user's avatar can perform in the third room. The information processing system according to claim 1 .
3. The first generation unit generates, in response to an operation by a user, an image of the first room showing a purchase of an item by the user; the second generation unit generates, in response to an operation by a user, a video in which an item purchased in the first room is used by an avatar of the user, as the video of the second room.
3. The information processing system according to claim 1 or 2.
4. the item purchased in the first room is associated with a person or character appearing in the content; the second generation unit generates, when an item purchased in the first room is used by an avatar of a user, a video including an action toward the user from a person or character associated with the item; The information processing system according to claim 3 .
5. an action that a user can perform in the third room is changed based on an action of the user in the second room; 5. An information processing system according to claim 1.
6. In the third room, an image showing the user's avatar captured in the second room can be purchased.
6. An information processing system according to claim 1.
7. A placement unit that determines a position of each user in the second room based on a group of users.
7. An information processing system according to claim 1.
8. the placement unit places a first group at a predetermined fixed position in the second room and assigns first placement data in which another group is placed at another position in the second room to the first group of users, and places a second group at the fixed position in the second room and assigns second placement data in which another group is placed at another position in the second room to the second group of users, such that a positional relationship between the first group and the second group is maintained between the first placement data and the second placement data. The information processing system according to claim 7.
9. the placement unit places a first group at a predetermined fixed position in the second room, assigns first placement data in which another group is placed at another position in the second room to a user of the first group, places a second group at the fixed position in the second room, assigns second placement data in which another group is placed at another position in the second room to a user of the second group, and places at least one other group between the fixed position in the second room and a position where the content is performed. The information processing system according to claim 7.
10. a voice providing unit that provides data of a voice uttered by a user belonging to a certain group to devices of other users belonging to the same group, but does not provide the data to devices of users belonging to other groups; 10. An information processing system according to claim 1.
11. At least one of the first room, the second room, and the third room is provided with a special user for guiding the plurality of users; The voice providing unit provides data of the voice uttered by the special user to devices of a plurality of groups of users. The information processing system according to claim 10.
12. a fourth generation unit that generates an image of a fourth room, the fourth room being a virtual space associated with an individual user and containing items that can be used by the user's avatar; When a user selects viewing of a specific content in the fourth room, the first generation unit generates a video of the first room corresponding to the content.
12. An information processing system according to claim 1.
13. generating an image of a first room, which is a virtual space where a plurality of users gather to watch a predetermined content, before the performance of the content; generating an image of a second room, which is a virtual space in which the content is performed; generating an image of a third room, which is a virtual space where the plurality of users gather, after the performance of the content; displaying the image of the first room, the image of the second room, and the image of the third room on a display unit in sequence; An information processing method implemented by a computer.
14. a function of generating an image of a first room, which is a virtual space where a plurality of users who will view a predetermined content gather, before the content is presented; A function of generating an image of a second room, which is a virtual space in which the content is performed; a function of generating an image of a third room, which is a virtual space in which the plurality of users gather, after the performance of the content; a function of sequentially displaying the image of the first room, the image of the second room, and the image of the third room on a display unit; A computer program that enables a computer to realize the above.
Citation Information
Patent Citations
Image processing system and image processing method
JP2019102091A
Information processing apparatus, information processing method, and computer program
JP2019139673A
Purchased commodity management system, user terminal, server, purchased commodity management method, and program
JP2021047722A
Game program, game method, and terminal device
JP2021153901A
Method and apparatus for selecting events to be displayed at virtual venues and social networking
US20090013263A1
