Program, information processing method, and information processing system

Abstract

To provide various pieces of information to a user in a virtual space without deteriorating usability.SOLUTION: A first output control unit configured to output a virtual space image of a three dimensional virtual space viewed from a virtual camera to a display visually recognized by a user, a second output control unit configured to output a projection screen operable by the user into the virtual space image, an input coordinate acquisition unit configured to acquire input coordinates of the user to an input device, and an operation determination unit configured to determine whether or not an input operation of the user is an operation to an input area corresponding to the projection screen based on the acquired input coordinates, when the operation determination unit determines that the input operation is not the operation on the input area, the first output control unit performs first display control processing related to the three dimensional virtual space other than the control on the projection screen, and when the operation determination unit determines that the input operation is the operation on the input area, the second output control unit performs second display control processing on the projection screen according to the input coordinates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing system. [Background technology]

[0002] A service is already available that displays a space such as an event venue created in a virtual space on a user device used by the user. For example, in this virtual space, the user can collect information by viewing exhibits (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-160065 Summary of the Invention [Problem to be solved by the invention]

[0004] In some virtual spaces, the user device can accept instructions regarding exhibits based on user input operations, such as selecting or enlarging an exhibit. However, the user device must also accept user instructions regarding the virtual space, which are instructions regarding things other than exhibits. The user device must distinguish between these instructions depending on the input operation. Increasing the number of instructions that can be accepted complicates the input operation. This may impair usability, such as ease of operation. Note that similar issues exist not only in virtual spaces of event venues where exhibits are placed, but also in other virtual spaces where objects are placed that display information for user viewing. [Means for solving the problem]

[0005] A program for solving the above problem causes one or more computers to function as a first output control unit that outputs a virtual space image of a three-dimensional virtual space seen from a virtual camera to a display viewed by a user, a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image, an input coordinate acquisition unit that acquires input coordinates to the user's input device, and an operation determination unit that determines, based on the acquired input coordinates, whether the user's input operation is an operation on an input area corresponding to the projection screen; when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process for the three-dimensional virtual space other than control of the projection screen; and when the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates.

[0006] An information processing method that solves the above problem includes one or more computers performing a first output control process that outputs a virtual space image of a three-dimensional virtual space seen from a virtual camera to a display viewed by a user, a second output control process that outputs a projection screen that can be operated by the user within the virtual space image, an input coordinate acquisition process that acquires input coordinates to the user's input device, and an operation determination process that determines whether the user's input operation is an operation on an input area corresponding to the projection screen based on the acquired input coordinates, and if the operation determination process determines that the input operation is not an operation on the input area, a first display control process for the three-dimensional virtual space other than control of the projection screen is performed, and if the operation determination process determines that the input operation is an operation on the input area, a second display control process for the projection screen is performed according to the input coordinates.

[0007] An information processing system that solves the above problem includes a first output control unit that outputs a virtual space image of a three-dimensional virtual space seen from a virtual camera to a display viewed by a user, a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image, an input coordinate acquisition unit that acquires input coordinates to the user's input device, and an operation determination unit that determines whether the user's input operation is an operation on an input area corresponding to the projection screen based on the acquired input coordinates, wherein if the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process for the three-dimensional virtual space other than control of the projection screen, and if the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a variety of information to users in a virtual space without reducing usability. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating hardware of a device such as a user device according to the first embodiment. [Figure 3] FIG. 2 is a block diagram illustrating functions of a user device according to the first embodiment. [Figure 4] FIG. 2 is a block diagram illustrating the functions of a management server according to the first embodiment. [Figure 5] FIG. 2 is a diagram schematically illustrating user management information according to the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a typical example of view tally information according to the first embodiment. [Figure 7] FIG. 3 is a diagram schematically illustrating performance information according to the first embodiment. [Figure 8]FIG. 2 is a diagram schematically illustrating a virtual space according to the first embodiment. [Figure 9] 5A to 5E are diagrams illustrating examples of input operations accepted by the user device of the first embodiment. [Figure 10] 5 is a flowchart showing a processing procedure of a user device according to the first embodiment. [Figure 11] 5 is a flowchart showing a processing procedure of a user device according to the first embodiment. [Figure 12] FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. [Figure 13] FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. [Figure 14] FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. [Figure 15] FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. [Figure 16] 10A and 10B are diagrams illustrating transmission and reception of input operation data in the second embodiment. [Figure 17] FIG. 10 is a diagram illustrating a virtual space screen displayed on a user device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] [First embodiment] A first embodiment of a program, an information processing method, and an information processing system will be described below.

[0011] <Information Processing System> 1, an information processing system 11 will be described. The information processing system 11 includes a management server 12 and a user device 20.

[0012] The information processing system 11 provides a service of outputting an image of a three-dimensional virtual space (hereinafter referred to as virtual space) to a user device 20. In the virtual space, the virtual position of the user 10 is movable. The virtual space is also called a world or a field. The information processing system 11 is a system that distributes a video of the virtual space displayed by one user 10 to other users 10, or a system in which the user device 20 outputs an image of the three-dimensional virtual space without distributing such a video. In this embodiment, the information processing system 11 will be described as a system that distributes a video.

[0013] In one embodiment, the management server 12 is managed by a PF provider that provides a platform (PF) for distributing and viewing videos in a virtual space. The web server 13 is managed by a publisher. The publisher uses the platform to provide resources, such as information on products or services, to users 10. When the publisher uses the platform, the resources are pre-associated with the virtual space in which the resources are published. When one publisher is associated with one virtual space (world), the resources published in a predetermined area of ​​the virtual space are associated with this publisher. When multiple publishers are associated with one virtual space (world), the resources published in a predetermined area of ​​the virtual space are associated with the multiple publishers. By providing their own information to users 10, publishers have the advantage of being able to advertise their own products, services, etc. The PF provider may receive compensation from the publisher for making the platform available to the publisher. Alternatively, the PF provider may be the publisher.

[0014] The management server 12 comprises one or more devices. In other words, the management server 12 may be a group of servers. The management server 12 is connected to user devices 20 via a network such as the Internet. The user devices 20 are devices on which a program for distributing and viewing videos is installed, or devices used by users 10 who have registered with a video providing service for distributing and viewing videos. The management server 12 relays data related to video distribution between the user devices 20.

[0015] FIG. 1 shows two users 10, i.e., users 10A and 10B. User 10A is a user who distributes videos using user device 20A, i.e., a distribution user, and user 10B is a user who watches the distributed videos on user device 20B, i.e., a viewing user. One user 10 can be a distribution user 10A at one time and a viewing user 10B at another time. That is, user 10 is a distribution user 10A when distributing videos and a viewing user 10B when watching videos. Furthermore, user device 20A that distributes videos is referred to as a distribution user device, and user device 20 that outputs videos for viewing by users is referred to as a viewing user device 20B. User device 20 can be switched between a mode for distributing videos, i.e., a distribution mode, and a mode for outputting videos for viewing, i.e., a viewing mode. In this embodiment, when describing without distinguishing between distribution user 10A and viewing user 10B, they are simply referred to as users 10. Furthermore, when the broadcasting user device 20A and the viewing user device 20B are not to be distinguished from each other, they will simply be referred to as user devices 20.

[0016] <Hardware> The hardware of the user device 20 will be described with reference to FIG. 2. The user device 20 may be a smartphone (multifunction phone), a tablet device, a personal computer, a console game machine, a wearable computer, or any other information processing device capable of playing videos. The wearable computer may be, for example, a head-mounted display worn on the user's head or a glasses-type wearable device, as long as it has a screen on which the user can watch videos. The user device 20 may be a standalone device, or a set of multiple devices connected so that various data can be transmitted and received between them. An example of the latter user device 20 is a system that performs outside-in tracking.

[0017] The user device 20 includes a processing circuit 100, a storage 104, a sensor unit 112, a display 109, an input device 110, and peripheral devices 111. The user device 20 illustrated in Fig. 2 includes a sensor unit 112 that detects the movement of the broadcast user 10A. Note that the sensor unit 112 may be a separate device from the user device 20 that is communicatively connected to the user device 20.

[0018] The processing circuit 100 includes a CPU 101, which is a processor that executes one or more of the control processes disclosed in this specification, and a memory 102. Processing data and instructions may be stored in the memory 102. The processing data and instructions may also be stored in a storage 104, which is a storage medium disk such as a hard drive (HDD) or a portable storage medium, or in a storage medium provided separately from the processing circuit 100. The storage 104 is connected to the CPU 101 via a storage controller 103 and a bus 108. In this embodiment, an application is stored in the storage 104. The application includes a program for distributing and viewing videos and various data required to execute the program.

[0019] Each function disclosed herein may be implemented using circuitry, which may include, but is not limited to, a CPU 101, a general-purpose processor, a special-purpose processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), other conventional circuitry, and / or a combination thereof, configured or programmed to perform the disclosed functions. A processor is a processing circuit or circuit because it includes transistors and other circuits therein. A processor may be a programmed processor that executes a program stored in a memory. In this disclosure, a unit or means, such as a processing circuit, is hardware that performs or is programmed to perform the described functions. The hardware may be any hardware disclosed herein or otherwise known that is programmed or configured to perform the described functions.

[0020] Furthermore, the processor is not limited by the form of computer-readable medium on which the process instructions are stored. For example, the instructions may be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk, or other non-transitory computer-readable medium of an information processing device with which processing circuit 100 communicates, such as a server or computer. The process may also be stored in network-based storage, cloud-based storage, or other mobile-accessible storage and executable by processing circuit 100.

[0021] The flowchart descriptions or blocks disclosed herein can be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logical functions or steps in a process. Also, the flowchart descriptions or blocks disclosed herein can perform two or more functions or steps in an order different from that shown or described in the specification, such as substantially simultaneously or in reverse order.

[0022] The hardware elements for implementing the processing circuit 100 may be implemented by various circuit elements. Furthermore, each function disclosed herein may be implemented by a circuit including one or more processing circuits.

[0023] The processing circuit 100 also includes a network controller 106 for connecting to a network NW. The network NW may be a public network such as the Internet, a private network such as a local area network (LAN) or a wide area network (WAN), or any combination thereof, and may also include a public switched telephone network (PSTN) or an integrated services digital network (ISDN®) or a sub-network. The network NW may also be a wired network such as an Ethernet® network or a universal serial bus (USB) cable, or a wireless network such as a cellular network including 3G, 4G, and 5G wireless cellular systems. The wireless network may also be Wi-Fi®, wireless LAN, Bluetooth®, or other known forms of wireless communication. Furthermore, the network controller 106 may be compliant with other direct communication standards such as Bluetooth®, near field communication (NFC), infrared, etc.

[0024] The processing circuit 100 further includes a display controller 105 and an input / output interface 107. The display controller 105, the network controller 106, and the input / output interface 107 are connected to a bus 108. The display controller 105 is connected to a display 109. The input / output interface 107 is connected to an input device 110 and peripheral devices 111.

[0025] The sensor unit 112 will now be described. The sensor unit 112 is one or more sensors that detect data indicating the movement of the user 10. The movement of the user 10 includes facial motion, which indicates a change in the user's facial expression, and body motion, which indicates a change in the relative position of the user's body with respect to the sensor unit 112. Facial motion includes movements such as blinking and opening and closing of the mouth. A known sensor can be used as the sensor unit 112. The sensor unit 112 has a sensor that measures the distance to a measurement object by irradiating the measurement object with light. Examples of such sensors include true depth sensors or "LIDAR" (Light Detection and Ranging, or Laser Imaging Detection and Ranging) sensors. For example, the light-emitting section of the sensor unit 112 projects a dot pattern consisting of tens of thousands of invisible dots onto the user's face or the like using a dot projector. The sensor unit 112 detects and analyzes the reflected light from the dot pattern to form a depth map of the face and capture an infrared image of the face or the like. This allows the sensor unit 112 to capture accurate facial data. The arithmetic processing unit of the sensor unit 112 generates various information based on the depth map and the infrared image, compares the information with registered reference data, and calculates the depth of each point on the face (the distance between each point and the near-infrared camera) and positional deviations other than in the depth direction. In another example, the sensor unit 112 includes a ToF sensor that measures the time of flight (time of flight) from when light is irradiated toward a measurement target, such as the user's face, until it is reflected back, or the phase difference between the irradiated light and the reflected light, using a distance measurement method other than the above-mentioned method. Alternatively or in addition, the sensor unit 112 may include a camera that captures an image of the user's face and an image processing unit that processes the data captured by the camera. The ToF sensor calculates the distance to the measurement target based on the time of flight or phase difference.

[0026] The sensor unit 112 outputs facial motion and body motion as tracking data to the processing circuit 100. The tracking data is included in the motion data. Motion data generally refers to data for moving an avatar object corresponding to the user 10. For example, the motion data includes emote data in addition to tracking data. The emote data is data for making the avatar object perform a pre-registered predetermined motion, such as "clap." Hereinafter, this pre-registered motion of the avatar object will be referred to as an "emote."

[0027] Furthermore, the sensor unit 112 may have a function of tracking not only the user's face but also their hands (hand tracking). The sensor unit 112 may also include a sensor that detects the position or orientation of a human body part other than the hand. The sensor unit 112 may further include a sensor that detects speed or acceleration, a sensor that detects direction or orientation (such as a gyro sensor), etc. The sensor unit 112 may have a spatial mapping function that recognizes objects in the real space where the user exists based on the detection results of the above-mentioned various sensors and maps the recognized objects on a spatial map, that is, creates a map of the positions of the recognized objects in the real space.

[0028] The input device 110 may be a touch panel, a keyboard, a mouse, a controller operated by the user's hands (e.g., a game controller such as a game pad or joystick), or operation buttons provided on the housing of the user device 20. The controller may incorporate various known sensors, such as an acceleration sensor, a gyroscope, or an inertial measurement unit (IMU). In another example, the input device 110 may be a tracking device that identifies the user's hand movements, eye movements, head movements, gaze direction, and the like. In this aspect, for example, the user's instructions can be determined based on the user's hand movements, and various operations can be performed, such as starting or ending video distribution, rating messages or videos, and displaying predetermined objects.

[0029] The peripheral device 111 includes a microphone that collects sounds in the real world, such as the voice of the user 10, and a speaker that outputs sound effects and the voices of other users 10, etc. The following describes a program included in an application stored in storage 104. The program includes a native unit implemented in user device 20. The native unit runs on the OS (Operating System) of user device 20 and can access devices (physical resources) such as storage 104 and a camera of user device 20. The native unit displays video on display 109 based on video data received from management server 12.

[0030] In this embodiment, the program includes an in-app browser. The in-app browser is a component, such as WebView, that is incorporated into a program and realizes browser functions. The WebView is, for example, an API (Application Programming Interface), such as SFSafariView or WKWebView. The in-app browser includes an analysis unit and an output control unit. The in-app browser acquires resources. The resources are, for example, web data. The web server 13 provides resources associated with the virtual space to the user device 20. The in-app browser can acquire resources from the web server 13 via a network. The web data is data necessary to display a browser screen on the display 109. The web data may include, for example, data written in HyperText Markup Language (HTML), data written in JavaScript (registered trademark), or the like, a CSS (Cascading Style Sheets) file, or the like. Like an independent web browser, the analysis unit of the in-app browser interprets HTML, CSS, or the like. Furthermore, the output control unit of the in-app browser executes JavaScript (registered trademark).

[0031] In another aspect, the program may launch an independent web browser (i.e., a web browser installed on the user device 20 separately from the in-app browser of the application) when a predetermined operation by the user is received. In this embodiment, the program will be described as having an in-app browser.

[0032] The management server 12 has the same processing circuit 100 and storage 104 as the user device 20. In the management server 12, the input device 110, the peripheral device 111, the sensor unit 112, the display 109, and the like can be omitted.

[0033] <User device> The functions of the user device 20 will be described with reference to Fig. 3. The processing circuit 100 executes a program recorded in the storage 104, whereby the user device 20 functions as an input coordinate acquisition unit 21, an operation determination unit 22, a first output control unit 23, and a second output control unit 24.

[0034] The first output control unit 23 outputs a virtual space image, which is a virtual space viewed from a virtual camera, to a display 109 that is visually recognized by the user. The second output control unit 24 outputs a projection screen that can be operated by the user within the virtual space image. The projection screen displays the resources obtained from the web server 13.

[0035] The input coordinate acquisition unit 21 acquires input coordinates, which are coordinates of an input made by a user to the input device 110. When the input device 110 is a touch panel, the input coordinate acquisition unit 21 acquires a series of input coordinates corresponding to the trajectory of a touch operation on the touch panel. When the input device 110 is a mouse, the input coordinate acquisition unit 21 acquires a series of input coordinates corresponding to the trajectory of a drag of the mouse and input coordinates corresponding to a click.

[0036] Based on the acquired input coordinates, the operation determination unit 22 determines whether the input operation by the user 10 is an operation on an input area corresponding to the projection screen. The input area is a predetermined area within the virtual space screen displayed on the display 109. When the operation determination unit 22 determines that the input operation is not an operation on the input area, the first output control unit 23 performs a first display control process related to the virtual space.

[0037] When the operation determination unit 22 determines that the input operation is an operation on the input area, the second output control unit 24 performs a second display control process on the projection screen in accordance with the acquired coordinates. Furthermore, when the operation determination unit 22 determines that the acquired series of coordinates changes continuously along the first axis, i.e., that the input operation corresponds to a first scroll operation, the second output control unit 24 scrolls the projection screen in the direction of the first scroll operation.

[0038] When the operation determination unit 22 determines that the input operation is a combination of a predetermined designation operation and a second scroll operation, the second output control unit 24 scrolls the projection screen in the direction of the second scroll operation. The second scroll operation is an input operation in which the acquired series of coordinates continuously changes along the second axis.

[0039] When the operation determination unit 22 determines that the input operation includes only the second scroll operation, the second output control unit 24 performs a first display control process. When the operation determination unit 22 determines that the input operation is a selection operation for selecting an arbitrary position on the projection screen, the second output control unit 24 executes processing according to the position of the selection operation.

[0040] <Administration Server> The management server 12 will be described with reference to Fig. 4. The management server 12 functions as a distribution management unit 30, a browsing history information receiving unit 32, and a performance recording unit 33 by the processing circuit 100 thereof executing a program recorded in the storage 104. The management server 12 also includes a user management information storage unit 34, a browsing aggregation information storage unit 36, and a performance information storage unit 37.

[0041] The distribution management unit 30 transmits video data for outputting a virtual space screen including a virtual space image to the display 109 to the user device 20. The distribution management unit 30 transmits asset data for drawing objects in the virtual space to the user device 20. The distribution management unit 30 receives data for drawing avatar objects, including motion data, from the distribution user device 20A, and transmits the received data to other user devices 20.

[0042] The browsing history information receiving unit 32 receives browsing history information indicating that the user 10 has browsed a resource associated with the virtual space from the user device 20. The browsing history information receiving unit 32 associates the browsing history information with the user 10 and records it in the browsing aggregation information storage unit 36.

[0043] The result recording unit 33 calculates the number of times each resource has been viewed based on the viewing history information, and records the number of times each resource has been viewed in the result information storage unit 37 as the result information of that resource. <Data Structure> Each piece of information recorded in the management server 12 will be described with reference to FIGS.

[0044] FIG. 5 shows an example of user management information 40 stored in the user management information storage unit 34. The user management information 40 is recorded for each user. The user management information 40 includes a user ID (identifier), which is user identification information, accumulated points, and coins. The accumulated points are points awarded for each user 10 based on activities on an application or a platform that provides the application, such as streaming and viewing videos. The accumulated points can be used as consideration for a lottery (gacha) that can be run on the platform, gift objects, or items. A gift object is an object that a viewing user 10B provides to a broadcasting user 10A, etc. Items include items worn by an avatar object and items used in games provided by the platform. Accumulated points may also be exchangeable for currency used in the real world. Coins differ from accumulated points in that they can be purchased by the user 10. Coins may be awarded based on the user 10's activity (streaming status or viewing status). Like accumulated points, coins may also be used as consideration for a lottery, gift objects, etc.

[0045] FIG. 6 shows an example of the view tally information 42 recorded in the view tally information storage unit 36. In this embodiment, the view tally information 42 is updated based on the view history information transmitted from the user device 20. The view tally information 42 includes a user ID, a URL (Uniform Resource Locator), a room ID, a view start time, a view end time, and a behavior history. The view start time is the time when the user device 20 connected to the web server 13 or the time when the user device 20 displayed a resource associated with the virtual space. The view end time is the time when the user device 20 disconnected from the web server 13 or the time when the user device 20 finished displaying the resource. The behavior history indicates the behavior performed by the user 10 on the displayed resource. Examples of the behavior history include requesting materials from the resource, purchasing a product, etc. from the resource, etc.

[0046] FIG. 7 shows an example of performance information 43 recorded in the performance information storage unit 37. Performance information 43 is recorded for each publisher. A publisher publishes resources. Multiple resources may be associated with one publisher. Publishers can advertise products and services to users through resources displayed in the virtual space.

[0047] The performance information 43 includes a publisher identifier, a URL, user stay time, questionnaire response information, and number of views. The publisher identifier is an identifier assigned to the publisher. The user stay (view) time indicates the time each user 10 spent viewing a resource, or statistical information on those viewing times. This time can be calculated from the view start time and view end time in the view summary information 42. The user stay time may be recorded as the time each user spent viewing a resource. Alternatively, the user stay time may be the user's average stay time, i.e., the average value of those individually recorded times. The survey response information indicates the content of the survey answered by the user 10 in the virtual space, i.e., the user's 10's responses to the survey in the virtual space. The number of views is the number obtained by aggregating the viewing history information and is the total number of times the resource was displayed.

[0048] <Video distribution and viewing methods> The management server 12 receives a video distribution request from the user device 20 of the user 10. At this time, the user 10 can specify a virtual space. Upon receiving the distribution request and identification information of the specified virtual space, the management server 12 generates a room (distribution slot) for the virtual space specified by the user 10. The management server 12 also transmits data (asset data) for displaying the virtual space to the user device 20.

[0049] The distribution slots in which videos are currently being distributed are displayed on a list screen displayed on each user device 20. The viewing user 10B can start viewing the video by selecting the distribution slot that he or she wants to view from the list screen.

[0050] The video delivery and viewing method can be one of a client rendering method, a browser rendering method, a video delivery method, and a server delivery method. In this embodiment, the client rendering method is used.

[0051] In the client rendering method, each user device 20 executes a program to render objects. User device 20 stores 3D model data, etc., of avatar objects and objects other than avatar objects (hereinafter referred to as physical objects). The 3D model data of avatar objects includes video data of body parts that make up the avatar object itself and attachment parts that can be attached to the avatar object. The data of body parts includes polygon data, skeletal data (bones) for expressing the movement of the avatar object, texture data, etc. The attachment parts include texture data, etc. Broadcast user 10A can set attachment parts for the avatar object corresponding to him / her according to his / her preferences. The model data of physical objects other than avatar objects includes polygon data, texture data, etc.

[0052] The first output control unit 23 performs drawing using video data including motion data and identification information (part IDs) indicating parts of each object. The first output control unit 23 also transmits video data including motion data, coordinates of the avatar object, etc. to the management server 12. The first output control unit 23 also transmits part IDs indicating each part of the avatar object to the management server 12 at a predetermined timing (for example, at the start of distribution). The management server 12 transmits the video data to the viewing user device 20B, etc. The viewing user device 20B receives this data from the management server 12. The viewing user device 20B then uses the received data to draw images of the avatar object and physical object of the broadcast user 10A.

[0053] The first output control unit 23 also transmits audio data based on the audio collected by the microphone to the management server 12. The management server 12 transmits the audio data to the viewing user device 20B. A timestamp is attached to the motion data, audio data, etc.

[0054] The first output control unit 23 of the viewing user device 20B receives video data and audio data for displaying video from the management server 12. Then, the first output control unit 23 uses the video data to draw an image including an avatar object corresponding to the broadcast user 10A.

[0055] The first output control unit 23 of the broadcasting user device 20A and the viewing user device 20B receives data such as messages posted by other users from the management server 12. The first output control unit 23 generates video data by combining rendered images of avatar objects and other objects with posted messages, notifications, etc., and outputs the generated video data to the display 109. The first output control unit 23 also synchronizes audio data with the video data based on a timestamp and outputs the audio data from the speaker.

[0056] In this embodiment, each user device 20 generates video using the client rendering method. Alternatively, the broadcasting user device 20A and the viewing user device 20B may display video using different methods. For example, the broadcasting user device 20A may display video using the client rendering method, while the viewing user device 20B may display video using the browser rendering method. Alternatively, the user may be able to select the method for displaying video. For example, some of the viewing user devices 20B may display video using the client rendering method, while others may display video using the browser rendering method.

[0057] <Virtual Space> A virtual space 50 in this embodiment will be described with reference to Fig. 8. The virtual space 50 includes a plurality of objects 51. The objects 51 located in the virtual space 50 include an avatar object 52 and other physical objects 53.

[0058] A world coordinate system 90 is set in the virtual space 50, indicating positions in the virtual space. The world coordinate system is a coordinate system having an X-axis, a Y-axis, and a Z-axis. The world coordinate system 90 is also called a global coordinate system. The position of the object 51 can be specified by coordinates in the world coordinate system 90. The coordinates specified in the world coordinate system 90 include coordinates (X, Y, Z) expressed in a Cartesian coordinate system. These coordinates may also include rotation coordinates (roll, pitch, yaw) relative to each coordinate axis. Each object 51 has its own object coordinate system 91 set. The object coordinate system 91 is a coordinate system having an X-axis, a Y-axis, and a Z-axis. The orientation, movement, etc. of the avatar object 52 can be specified by coordinates in the Cartesian coordinate system or rotation coordinates.

[0059] A virtual camera 54 associated with each broadcast user 10A is positioned in the virtual space 50. A camera coordinate system 92 is set for the virtual camera 54. For example, the camera coordinate system 92 includes coordinates (X, Y, Z) specified in a three-dimensional Cartesian coordinate system, as well as rotational coordinates (Xθ, Yθ, Zθ) centered on each coordinate axis.

[0060] The first output control unit 23 performs rendering based on the coordinates of objects included in a predetermined field of view (for example, a field of view cone having an approximately conical shape) centered on the virtual optical axis of the virtual camera 54. The first output control unit 23 then outputs the two-dimensional image generated by rendering to the display 109. The rendering here refers to a drawing process that includes obtaining the position of the virtual camera 54, perspective projection, hidden surface removal based on depth information associated with each object, and the like. The rendering may be at least one of these processes, or may further include processes such as shading and texture mapping.

[0061] The position of the virtual camera 54 can be changed by at least one of the distributor user 10A and the viewing user 10B. The distributor user 10A can select a first-person perspective (avatar position) or a third-person perspective as a setting for the position of the virtual camera 54. In the first-person perspective, the first output control unit 23 performs rendering with the avatar's eye position aligned or nearly aligned with the position of the virtual camera 54. In this case, the line of sight extending from the front of the avatar object 52 nearly aligns with the optical axis of the virtual camera 54. As a result, an image is displayed on the display 109 as if the virtual space 50 were being viewed from the avatar object 52, i.e., through the avatar's eyes. In the third-person perspective, the first output control unit 23 sets the virtual camera 54 at a position around the avatar object 52. Setting the virtual camera 54 in the third-person perspective allows an image of the avatar object 52 to be displayed on the screen. The first output control unit 23 can change the position of the virtual camera 54 in the third-person perspective. Alternatively, images may be generated using a multi-view camera that selects one (or more) of virtual cameras 54 positioned at three or more positions, including a first-person viewpoint, a third-person viewpoint, and one or more other positions.

[0062] The camera coordinate system 92 specifies the camera orientation and operations such as panning and tilting. In the case of a third-person perspective, the first output control unit 23 changes at least one of the position and direction of the virtual camera 54 based on an operation by the user 10. In the case of a first-person perspective, the first output control unit 23 changes at least one of the position and direction of the virtual camera 54 based on an operation by the user 10 to move the avatar object 52.

[0063] In this embodiment, in the case of a third-person perspective, virtual camera 54 may be associated with avatar object 52 so as to maintain a positional relationship with avatar object 52. For example, first output control unit 23 causes virtual camera 54 to follow the movement of avatar object 52 during a period when no camera operation is being performed. For example, first output control unit 23 acquires the movement direction instructed by broadcast user 10A as a first direction, converts the first direction into a second direction as a movement direction in world coordinate system 90, and moves avatar object 52 in the second direction. Furthermore, first output control unit 23 moves virtual camera 54 in the second direction while maintaining a constant relative distance between avatar object 52 and virtual camera 54. This prevents avatar object 52 from moving away from virtual camera 54, thereby preventing avatar object 52 from leaving the screen or being displayed so small that it is difficult to see. Even when the virtual camera 54 is caused to follow the avatar object 52, the user 10 can change at least one of the position and direction of the virtual camera 54 by an input operation. In the present embodiment, the first output control unit 23 changes the position of the virtual camera 54 based on a slide operation by the user 10.

[0064] A browser screen 57 is output as a projection screen in the virtual space 50. The browser screen 57 is a screen projected onto a display object 58. The display object 58 is an object having coordinates in the world coordinate system in the virtual space 50. Only one display object 58 may be provided in the virtual space 50, or multiple display objects 58 may be provided.

[0065] For example, if an exhibition is held in the virtual space 50, a display object 58 may be provided for each of a plurality of exhibition booths. The display object 58 is an object for displaying resources made public by the exhibitor of the exhibition booth. The display object 58 is, for example, a two-dimensional object with no thickness, or a three-dimensional object with a display surface. Note that an exhibition object 51 other than the display object 58 may be set as an exhibit in the exhibition booth.

[0066] The second output control unit 24 loads web data identified by a predetermined URL using an in-app browser and outputs a browser screen 57 based on the web data to a display object 58 using a technique such as texture mapping. The browser screen 57 may include still images in JPEG, GIF, or PNG format, or images such as video based on video data such as MPEG4. The browser screen 57 is displayed in the same manner as if it were displayed by a web browser separate from the in-app browser of the application. Note that details of the browser screen 57 are omitted from FIG. 8.

[0067] The first output control unit 23 converts coordinates P1 in the display area 97 displayed on the screen output to the display 109 into coordinates P2 on the browser screen 57 (see arrow 95 in FIG. 8). An example of this coordinate conversion process will be described. An input coordinate system 93 is set in the virtual space 50. The input coordinate system 93 is a two-dimensional Cartesian coordinate system that specifies coordinates on the screen of the display 109. The Y axis of the input coordinate system 93 corresponds to the first axis. The X axis corresponds to the second axis.

[0068] A UI area 55 and a browser area 56 are set in the virtual space 50. The browser area 56 corresponds to an input area. Coordinates in these areas are specified in an input coordinate system 93. The first output control unit 23 or the second output control unit 24 detects the position and direction of the input operation by acquiring the coordinates of the input operation by the user 10 relative to these areas. Furthermore, the first output control unit 23 or the second output control unit 24 determines the content of the user instruction corresponding to the input operation according to at least one of the position and direction of the input operation, and performs control according to the user instruction.

[0069] The UI area 55 is an area (layer) that accepts operations on GUI (Graphical User Interface) elements such as buttons, text boxes, and icons. FIG. 8 shows UI areas 55A to 55D as the UI area 55. The UI area 55 itself is hidden, but GUI elements (not shown in FIG. 8) are displayed within the UI area 55. That is, while the GUI elements within the UI area 55 are displayed on the virtual space screen, the UI area 55 itself is not displayed. In the example of FIG. 8, a GUI element such as a setting button is displayed within the UI area 55A. GUI elements such as a button for moving the avatar object 52 and a button for jumping are displayed within the UI areas 55B and 55C, respectively. A GUI element such as a text box into which a message can be input is displayed within the UI area 55D. Note that the UI areas 55A to 55D in FIG. 8 are simplified representations of actual screens.

[0070] Browser area 56 is an area (layer) that accepts input operations for browser screen 57. The coordinates of browser area 56 can be specified by input coordinate system 93. First output control unit 23 sets browser area 56 when the display conditions for browser screen 57 are satisfied. At this time, first output control unit 23 sets browser area 56 so that browser area 56 overlaps with browser screen 57 when viewed from virtual camera 54.

[0071] Furthermore, when at least one of the position and orientation of the virtual camera 54 is changed, the first output control unit 23 sets the browser area 56 to match the shape and size of the browser screen 57 as seen from the virtual camera 54. For example, when the browser screen 57 becomes larger on the screen as a result of the virtual camera 54 approaching the display object 58 in response to an operation by the user 10, the first output control unit 23 enlarges the browser area 56 to match the size of the browser screen 57. Similarly, when the browser screen 57 becomes smaller on the screen, the first output control unit 23 reduces the browser area 56 to match the size of the browser screen 57. Furthermore, when the relative angle between the virtual camera 54 and the browser screen 57 changes in response to an operation by the user 10, the first output control unit 23 deforms the browser area 56 to match the shape of the browser screen 57 on the screen.

[0072] The operation determination unit 22 identifies the coordinates of an input to the browser area 56. In other words, the operation determination unit 22 identifies an input trajectory corresponding to a trajectory of continuous input in the browser area 56. The operation determination unit 22 also converts the identified coordinates into coordinates on the browser screen 57. The coordinates on the browser screen 57 are identified in the browser screen coordinate system 94.

[0073] The second output control unit 24 receives a request for display control based on input coordinates, which are coordinates in the browser screen coordinate system 94, from the operation determination unit 22. The second output control unit 24 performs second output control processing based on a user instruction for controlling the browser screen 57. For example, when a series of input coordinates on the browser screen 57 are consecutive coordinates along the Y axis (positive Y direction and negative Y direction) of the browser screen coordinate system 94, the second output control unit 24 determines that an input operation for scrolling the browser screen 57 has been performed. Then, the second output control unit 24 scrolls the browser screen 57 in the same direction as the direction of the input operation.

[0074] Furthermore, when the operation determination unit 22 acquires, as input coordinates, coordinates in the input coordinate system 93 other than the UI area 55 and the browser area 56, it requests the first output control unit 23 to perform display control based on the acquired coordinates. The first output control unit 23 performs display control processing related to the virtual space 50.

[0075] <Input operation> Next, with reference to FIG. 9 , an input operation by the user 10 and the display control of the first output control unit 23 or the second output control unit 24 in response to the input operation will be described. Here, a case will be described in which the input device 110 is a touch panel. An input operation on the touch panel is performed by contact with a finger, a stylus, or the like. An input operation on the browser area 56 can be performed by at least one of the broadcasting user 10A and the viewing user 10B. In this embodiment, when the user device 20 is in a broadcasting mode in which a video is broadcast, the operation determination unit 22 of the user device 20 accepts an input operation on the browser area 56. Furthermore, when the user device 20 is in a viewing mode in which a video is viewed, the operation determination unit 22 of the user device 20 does not accept an input operation on the browser area 56. In other words, only the broadcasting user 10A can perform an input operation on the browser area 56.

[0076] 9(a) to 9(e) each show a virtual space screen 60 including a browser screen 57. The virtual space screen 60 includes a virtual space image 62 including other objects 51 (not shown in FIG. 9), and GUI elements (not shown in FIG. 9). Hereinafter, with respect to the virtual space screen 60, a direction parallel to the X axis of an input coordinate system 93 (positive X direction and negative X direction) will be referred to as a horizontal direction. The positive X direction is the direction in which the X coordinate increases, and the negative X direction is the direction in which the X coordinate decreases. Furthermore, a direction parallel to the Y axis of the input coordinate system 93 (positive Y direction and negative Y direction) and perpendicular to the X axis will be referred to as a vertical direction. The positive Y direction is the direction in which the Y coordinate increases, and the negative Y direction is the direction in which the Y coordinate decreases.

[0077] The first output control unit 23 determines the moving direction of the position of the virtual camera 54 based on predetermined conditions. The determination of the direction indicated by the user 10 may be performed using any known method, without any particular limitation. For example, when the user 10 performs an input operation diagonally relative to the X-axis and Y-axis, the first output control unit 23 compares the amount of change in the X-coordinate and the amount of change in the Y-coordinate of the input trajectory (vector). The first output control unit 23 then determines that the input operation is an input operation along the one of the two coordinate axes with the larger amount of change. For example, if the amount of change in the X-coordinate is larger than the amount of change in the Y-coordinate, the first output control unit 23 determines that the input operation is an input operation along the X-axis. Alternatively, the first output control unit 23 may determine the direction of the input operation based on the angle of the vector of the input trajectory.

[0078] 9(a), the user 10 can use a touch operation to specify an input start position PA outside the UI area (not shown in FIG. 9) and outside the browser area 56. At this time, the operation determination unit 22 acquires the input coordinates of the input start position PA.

[0079] 9(b), the user 10 may slide the touch position in the X-positive direction from the input start position PA to the input end position PB. The operation determination unit 22 acquires a series of input coordinates that continuously change from the input start position PA to the input end position PB in response to this slide operation. In this case, the operation determination unit 22 requests the first output control unit 23 to control the display based on the acquired input coordinates. The first output control unit 23 performs a first display control process. In the first display control process, the first output control unit 23 changes the position of the virtual camera 54. More specifically, the first output control unit 23 moves the position of the virtual camera 54 in a direction from the input start position PA toward the input end position PB (the X-positive direction in FIG. 9). The position of the virtual camera 54 may be moved parallel to the X-axis of the camera coordinate system 92, or may be moved in an arc so as to maintain a constant relative distance to a predetermined object 51 (e.g., an avatar object 52).

[0080] In addition, when the first output control unit 23 acquires a series of input coordinates that change continuously in the positive Y direction (or negative Y direction) from the input start position PA, the position of the virtual camera 54 may be moved in the positive Y direction (or negative Y direction). 9(a), the user 10 can specify an input start position PC within the browser area 56. At this time, the operation determination unit 22 acquires the input coordinates of the input start position PC.

[0081] As shown in FIG. 9(c), the user 10 may slide the touch position in the negative Y direction from the input start position PC to the input end position PD. At this time, the operation determination unit 22 acquires a series of input coordinates that continuously change from the input start position PC to the input end position PD in response to this slide operation. In this case, the operation determination unit 22 requests the second output control unit 24 to perform display control based on the acquired input coordinates. The second output control unit 24 scrolls the browser screen 57 in the negative Y direction of the browser screen coordinate system 94 (the same direction as the negative Y direction of the input coordinate system 93). In other words, the second output control unit 24 displays a range that is not displayed on the browser screen 57 shown in FIG. 9(a) and is located above the range that was displayed before the slide operation started. Note that even if the input end position PD is outside the browser area 56, the first output control unit 23 may determine that the operation is a vertical scroll operation on the browser screen 57 if part of the input trajectory of the vertical slide operation is included in the browser area 56.

[0082] 9(d), after the user 10 performs a long press on the input start position PC, i.e., continues the touch operation on the input start position PC for a predetermined time, the user 10 may perform a slide operation in the X-positive direction to the input end position PE without releasing the touch operation. That is, the operation determination unit 22 may determine that the input operation is a combination of a designation operation, which is a long press operation, and a slide operation in the horizontal direction. At this time, the operation determination unit 22 requests the second output control unit 24 to perform display control based on the acquired input coordinates.

[0083] The second output control unit 24 determines whether the input start position PC is included in a horizontally scrollable display range 61. The horizontally scrollable display range is a range on the screen in which the display content can be switched by moving it left and right, and is also called a carousel. When the second output control unit 24 determines that the input start position PC is included in the horizontally scrollable display range 61, it scrolls that range on the browser screen 57 in the direction from the input start position PC toward the input end position PE.

[0084] 9(e), the user 10 may perform a slide operation in the positive X direction from the input start position PC to the input end position PF without performing a long press operation. At this time, the operation determination unit 22 acquires a series of input coordinates that continuously change from the input start position PC to the input end position PF in response to this slide operation. In this case, the operation determination unit 22 requests the first output control unit 23 to perform display control based on the acquired input coordinates.

[0085] First output control unit 23 moves the position of virtual camera 54 in the positive X direction. That is, user 10 can move the position of virtual camera 54 by performing a horizontal slide operation outside UI area 55 and outside browser area 56, or a slide operation within browser area 56 that does not include a long press operation.

[0086] Furthermore, suppose that the user 10 taps a predetermined area of ​​the browser screen 57 (for example, the input start position PC shown in FIG. 9(a)). In this case, the operation determination unit 22 outputs a determination result including the input coordinates to the second output control unit 24. The second output control unit 24 performs display control according to the input coordinates acquired from the operation determination unit 22. For example, if the input coordinates are included in a video display area, the second output control unit 24 plays a stopped video or stops a video that is being played. The video display area is an area where thumbnails (small-sized images) of videos are displayed. Alternatively, if the input coordinates are included in an area that accepts a link selection operation, the second output control unit 24 acquires a resource specified by the link from the web server 13. Then, the second output control unit 24 displays the acquired link on the browser screen 57.

[0087] Furthermore, suppose that user 10 performs either a pinch-in or pinch-out operation (hereinafter referred to as a pinch operation) on virtual space screen 60. In this case, operation determination unit 22 determines that the operation is a pinch operation and outputs a determination result including the input coordinates to first output control unit 23. First output control unit 23 reduces or enlarges the entire virtual space image 62 regardless of whether the acquired input coordinates are included in browser area 56. In other words, even if user 10 performs a pinch-out operation within browser area 56, virtual space image 62 is enlarged. As a result, browser screen 57 included in virtual space 50 is also enlarged.

[0088] When the broadcast user device 20A performs display control in response to the input operation of the broadcast user 10A in this manner, the broadcast user device 20A transmits input operation data indicating the content of the input operation to the viewing user device 20B via the management server 12. The input operation data includes an operation instruction for the browser screen 57, identification information of the display object 58 to be operated, the operation amount, the position of the virtual camera 54, etc. The operation instruction may be vertical scrolling, horizontal scrolling, link selection, stopping or playing a video, etc. The operation amount may be the amount of scrolling, etc. The viewing user device 20B performs display control for the browser screen 57 and controls the virtual camera 54 based on the input operation data. This synchronizes the screen output to the broadcast user device 20A and the screen output to the viewing user device 20B. Alternatively, the broadcast user device 20A may transmit data indicating the display position of the browser screen 57 to the viewing user device 20B in order to synchronize the browser screens 57 between the broadcast user device 20A and the viewing user device 20B.

[0089] <Operation> 10 and 11, the processing of the user device 20 will be described. Here, it is assumed that the broadcast user device 20A is displaying a virtual space screen 60 including a browser screen 57 on the display 109. The display condition for the browser screen 57 is that the position of the avatar object 52 is within a predetermined area set for the display object 58, or that the broadcast user 10A has performed an operation to display the browser screen 57. When the second output control unit 24 determines that the display condition is met, it requests the web server 13 for a resource identified by the URL associated with the display object 58. When the second output control unit 24 receives the resource from the web server 13, it launches an in-app browser and outputs the browser screen 57 based on the resource to the display surface of the display object 58 in the virtual space 50.

[0090] Input coordinate acquisition unit 21 acquires input coordinates, which are the coordinates of an input made by broadcast user 10A to input device 110 (step S10). Operation determination unit 22 determines which area of ​​input coordinate system 93 the input coordinates represent. For example, operation determination unit 22 determines whether the acquired input coordinates are included within UI area 55 (step S11). If operation determination unit 22 determines that the input coordinates are included within UI area 55 (step S11: YES), it requests first output control unit 23 to perform display control processing corresponding to that UI area 55. First output control unit 23 performs display control processing corresponding to that UI area 55 (step S12). For example, if first output control unit 23 determines that a GUI element for moving avatar object 52 has been operated, it moves avatar object 52 in the direction indicated by broadcast user 10A.

[0091] When the operation determination unit 22 determines that the input coordinates are not included in the UI area 55 (step S11: NO), it determines whether the input coordinates are included in the browser area 56 (step S13). When the operation determination unit 22 determines that the input coordinates are included in the browser area 56 (step S13: YES), it requests the second output control unit 24 to perform display control based on the input coordinates. The second output control unit 24 performs operation determination processing based on the input coordinates (step S14).

[0092] When the operation determination unit 22 determines that the input coordinates are not included in either the UI area 55 or the browser area 56 (step S13: NO), it requests the first output control unit 23 to perform display control based on the input coordinates. The first output control unit 23 performs a first display control process for the virtual space 50 (step S15). Here, the first display control process is control of the virtual camera 54.

[0093] Next, the operation determination process (step S14) will be described. The second output control unit 24 converts the input coordinates into coordinates on the browser screen 57. Then, the second output control unit 24 determines the input operation corresponding to the converted coordinates.

[0094] When the second output control unit 24 determines that the input operation is a pinch-in operation, it reduces the virtual space image 62. When the second output control unit 24 determines that the input operation is a pinch-out operation, it enlarges the virtual space image 62. When the second output control unit 24 determines that the input operation is a selection operation, it transitions the screen, or stops or plays the video, depending on the input coordinates.

[0095] If the converted coordinates change continuously, the second output control unit 24 determines whether the user instruction is for vertical scrolling, horizontal scrolling, or control of the virtual camera 54.

[0096] 11, the second output control unit 24 determines whether the slide operation is a vertical operation (step S20). If the second output control unit 24 determines that the slide operation is a vertical operation (step S20: YES), the second output control unit 24 scrolls the browser screen 57 vertically (step S21). If the second output control unit 24 determines that the slide operation is a horizontal operation rather than a vertical operation (step S20: YES), the second output control unit 24 determines whether the input operation includes a designation operation (step S22).

[0097] If the second output control unit 24 determines that the input operation includes a designation operation (step S22: YES), it scrolls the horizontally scrollable range of the browser screen 57 in the direction in which the converted coordinates change (step S23).If the second output control unit 24 determines that the input operation does not include a designation operation (step S22: NO), it controls the virtual camera 54 (camera control) (step S24).

[0098] Next, the process of collecting the view aggregation information by the management server 12 will be described. When the broadcast user device 20A outputs the browser screen 57, it transmits browsing history information to the management server 12. The browsing history information includes the broadcast user's 10A user ID, URL, room ID, and viewing start time. Furthermore, each time an input operation is performed, the broadcast user 10A may transmit browsing information including the input operation as a behavior history to the management server 12. Furthermore, when the broadcast user device 20A no longer outputs the browser screen 57, it transmits browsing history information including the viewing end time to the management server 12. The browsing history information receiving unit 32 uses the browsing history information to record viewing tally information 42 in the viewing tally information storage unit 36.

[0099] The performance record unit 33 generates performance information 43 using the viewing count information 42. At this time, the number of views may be calculated based on the number of viewing users 10B. After generating the performance information 43, the performance record unit 33 records it in the performance information storage unit 37. The administrator of the management server 12 (e.g., a PF provider) can provide the performance information 43 to the publisher. The publisher can check the user response by checking the frequency with which the information they have published has been viewed and the responses to questionnaires, etc. For example, the administrator can receive compensation from the publisher according to the performance information.

[0100] <Screen display> The virtual space screen 60 (distribution screen) displayed by the broadcast user device 20A will be described with reference to Figures 12 to 15. Note that Figures 12 to 15 show the screen when the virtual camera 54 is positioned from a third-person perspective. Furthermore, the virtual space screen (viewing screen) displayed by the viewing user device 20B is substantially the same as the virtual space screen 60 displayed by the broadcast user device 20A, except for the GUI elements. However, in this embodiment, the viewing user 10B cannot operate the browser screen 57, as described above.

[0101] As shown in FIG. 12 , an avatar object 52 is displayed on a virtual space screen 60. For example, when the avatar object 52 moves to the vicinity of a display object 58 so that the display object 58 is included in the field of view of the virtual camera 54, the first output control unit 23 includes the display object 58 on the virtual space screen 60. When the first output control unit 23 determines that the display conditions are satisfied, it outputs a browser screen 57. The browser screen 57 displays a screen of a website identified by a URL associated with the display object 58. For example, if an exhibition is being held in the virtual space 50, a screen of a website related to the exhibition content is displayed. The website is a site that publishes information about the exhibitor, details of products or services, or the like.

[0102] Furthermore, a plurality of GUI elements 65 and icons are displayed on the virtual space screen 60. Furthermore, a movement button 66 and a jump button 67, which are GUI elements, are displayed on the virtual space screen 60. When the user 10 operates the movement button 66, the first output control unit 23 moves the avatar object 52. When the user 10 operates the jump button 67, the first output control unit 23 jumps the avatar object 52. This allows the avatar object 52 to be moved to any exhibition booth, for example.

[0103] When distribution user 10A performs a vertical slide operation on browser screen 57 (see arrow 96 in FIG. 12), second output control unit 24 scrolls browser screen 57 in the direction of the operation. On a website, actions similar to those performed on an independent browser are possible. For example, distribution user 10A can view information about products, etc., purchase products, etc., request materials, etc.

[0104] In this way, when broadcasting user 10A views information about a product or the like on browser screen 57, viewing user 10B who watches the distributed video will also see that information. Therefore, publishers can advertise products or the like by exhibiting them in virtual space 50 and having broadcasting user 10A distribute videos. Furthermore, if broadcasting user 10A can operate browser screen 57 within virtual space 50, more information can be provided to broadcasting user 10A and viewing user 10B than if a non-operable image is displayed in display object 58. Therefore, publishers can further increase the advertising effectiveness of the information they publish.

[0105] FIG. 13 shows a virtual space screen 60 in which the position of virtual camera 54 has been moved. When broadcast user 10A performs an input operation to move virtual camera 54, first output control unit 23 changes position 54A of virtual camera 54 to position 54B based on the direction of the input operation. At this time, as illustrated in FIG. 13, first output control unit 23 may move virtual camera 54 by a distance corresponding to the amount of input operation while keeping the virtual optical axis of virtual camera 54 facing avatar object 52. First output control unit 23 outputs to display 109 a virtual space image 62 included in field of view Z1 of virtual camera 54 at position 54B.

[0106] As shown in FIG. 14, when the distribution user 10A performs a designation operation and a horizontal slide operation successively in the horizontally scrollable display range 99 in the browser area 56, the second output control unit 24 scrolls the display range 99 horizontally (see arrow 98 in FIG. 14).

[0107] 15 shows the virtual space screen 60 when the broadcast user 10A pinches out in the browser area 56. The second output control unit 24 enlarges only the virtual space image 62 on the virtual space screen 60. As a result, the browser screen 57 is also enlarged and displayed.

[0108] The effects of the first embodiment will be described. (1-1) A browser screen 57 that can be operated by the user 10 through input operations is displayed on the virtual space screen 60. The user 10 can efficiently collect information that interests him or her while exploring the virtual space 50. Furthermore, the publisher can provide information to users 10 who have behavioral tendencies that differ from customers in the real world as potential customers through the browser screen 57. Furthermore, the operation determination unit 22 distinguishes between operations for controlling the display of the browser screen 57 and operations for controlling the virtual camera 54, depending on the input coordinates. This allows the user 10 to intuitively operate the browser screen 57 without performing complex operations.

[0109] (1-2) When the operation determination unit 22 determines that the input coordinates are a scroll operation that continuously changes along the first axis (vertical direction), the second output control unit 24 scrolls the browser screen 57 in the direction of the operation. This allows the user 10 to scroll the browser screen 57 along the first axis with an intuitive operation.

[0110] (1-3) When the operation determination unit 22 determines that the input operation is a combination of a predetermined designation operation and a scroll operation in which the input coordinates change continuously along the second axis (horizontal direction), the second output control unit 24 scrolls the horizontally scrollable display area 97 of the browser screen 57 in the direction of the operation. When the operation determination unit 22 determines that the input operation includes only a scroll operation in which the input coordinates change continuously along the second axis, the first output control unit 23 changes the position of the virtual camera 54 in the direction of the operation. This allows the user device 20 to distinguish between a slide operation that changes the position of the virtual camera 54 and an operation for scrolling along the second axis.

[0111] (1-4) When operation determination unit 22 determines that the input operation is not an input operation within UI area 55 or browser area 56, first output control unit 23 changes the position of virtual camera 54 in the direction of the operation. This allows user 10 to control virtual camera 54 by intuitive operation.

[0112] (1-5) When the operation determination unit 22 determines that the input operation is a selection operation in the browser area 56, the second output control unit 24 executes processing according to the position of the selection operation. This allows the user 10 to control the virtual camera 54 with an intuitive operation.

[0113] (1-6) The first output control unit 23 outputs into the virtual space an avatar object 52 corresponding to the user 10. As a result, through the avatar object 52, it is possible to stimulate at least one of interaction between the distributor user 10A and the viewing users 10B, and interaction between the distributor users 10A.

[0114] (1-7) The second output control unit 24 accepts input operations into the browser area 56 in the distribution mode, but does not accept input operations into the browser area 56 in the viewing mode. This allows the viewing user 10B to view the browser screen 57 viewed by the distribution user 10A on the viewing user device 20B, making it easier for them to interact with the distribution user 10A.

[0115] [Second embodiment] Hereinafter, a second embodiment of the program, the information processing method, the server, the information processing method for the server, and the information processing system will be described with reference to FIG.

[0116] In the first embodiment, only the broadcast user 10A performs input operations in the browser area 56. In the second embodiment, in addition to the broadcast user 10A, authorized viewing users 10B can also operate the browser screen 57. Furthermore, in the second embodiment, in addition to each broadcast user device 20 performing broadcasting independently, multiple broadcast users 10A perform collaborative broadcasting in which multiple broadcast user devices 20 perform broadcasting together. Even in collaborative broadcasting, each broadcast user 10A can independently perform input operations in the browser area 56. Hereinafter, parts similar to those in the first embodiment will be assigned the same reference numerals, and detailed description thereof will be omitted.

[0117] Collaborative distribution will now be described. Collaborative distribution refers to multiple users distributing videos in the same virtual space 50. Hereinafter, the distributor user 10A who requests the creation of a room will be referred to as the host user, and the distributor users 10A who participate in the room created by the host user will be referred to as the guest users. There may be multiple guest users.

[0118] The user 10 can participate in the collaboration distribution as a guest user by accepting the participation request from the host user, or by having the participation request sent by the user 10 approved by the host user.

[0119] The broadcast user device 20A used by the host user is referred to as the host user device 20H, and the broadcast user device 20A used by the guest user is referred to as the guest user device 20G. The host user device 20H outputs a virtual space screen 60 seen from a virtual camera 54 set by the host user. The guest user device 20G outputs a virtual space screen 60 seen from a virtual camera 54 set by the guest user. In other words, the host user device 20H and the guest user device 20G broadcast within the same virtual space 50, but the display ranges of their respective virtual space screens 60 may differ. Alternatively, if the position of the virtual camera 54 is fixed, each broadcast user 10A may output a virtual image of the same display range.

[0120] The second output control units 24 of the host user device 20H and the guest user device 20G each control the display of the browser screen 57 based on the input operation of the broadcast user 10A. The input operation on the browser screen 57 here includes operations to control the display state of the browser screen 57, such as operations to scroll vertically, operations to scroll horizontally, and selection operations, as in the first embodiment. Note that when multiple display objects 58 are located within the virtual space 50, the host user and the guest user may each operate the browser screen 57 displayed on a different display object 58.

[0121] 16 shows data being transmitted and received between a broadcasting user device 20A and a viewing user 20B. The management server 12 that relays data transmission and reception between the user devices 20 is not shown. Also, video data, gift display requests, message data, location information of the virtual camera 54, and the like transmitted between the user devices 20 are not shown.

[0122] One or more of the viewing users 10B have the authority to perform input operations on the browser screen 57, while the remaining viewing users 10B do not have the authority to perform input operations. Viewing users 10B who meet a predetermined condition have the authority to perform input operations. The predetermined condition may be, for example, that the viewing user 10B was previously associated with the broadcasting user 10A (following or being a friend) before watching the video, that the viewing user 10B made a request to the broadcasting user 10A to display a gift object (tipping), or that the viewing user 10B paid a fee to the broadcasting user 10A or the video distribution service. The viewing user device 20B of the viewing user 10B who meets the predetermined condition will accept input operations on the browser screen 57.

[0123] In this embodiment, the authority of distributor user 10A and viewing user 10B to operate browser screen 57, and the reflection of the operated browser screen 57 on other devices, is as follows. The host user device 20H accepts input operations on the browser screen 57, and transmits data for reflecting the display state of the operated browser screen 57 on other devices. The guest user device 20G accepts input operations on the browser screen 57 and transmits data for reflecting the display state of the operated browser screen 57 on other devices. The viewer user device 20B of the authorized viewer user 10B accepts input operations on the browser screen 57, but does not transmit data for reflecting the display state of the operated browser screen 57 on other devices.

[0124] When the host user device 20H accepts an input operation on the browser screen 57 of the host user, it transmits input operation data D1, including an input operation instruction, identification information of the display object 58 to be operated, and a URL, to the guest user device 20G and the viewing user device 20B. The input operation instruction includes a scroll direction and a scroll amount. The guest user device 20G and the viewing user device 20B output the browser screen 57 to the display object 58 to be operated by the host user, based on the input operation data D1. If the input operation data D1 does not include a URL, the guest user device 20G and the viewing user device 20B obtain a URL associated with the display object 58 to be displayed from the storage 104 or the management server 12 of their own devices. Then, they obtain a resource identified by the URL from the web server 13. The guest user device 20G and the viewing user device 20B scroll the browser screen 57 based on the input operation data D1. In other words, the guest user device 20G and the viewing user device 20B control the display of the browser screen 57 in accordance with the input operation of the host user. This allows the browser screens 57 to be synchronized between users 10 .

[0125] Similarly, when the guest user device 20G accepts an input operation on the browser screen 57 of the guest user, it transmits input operation data D1 to the host user device 20H and the viewing user device 20B. Based on the input operation data D1, the host user device 20H and the viewing user device 20B output the browser screen 57 to the display object 58 operated by the guest user.

[0126] On the other hand, when the viewing user device 20B of the authorized viewing user 10B accepts an input operation on the viewing user's browser screen 57, it controls the display of the browser screen 57 based on the input operation. However, this viewing user device 20B does not transmit the input operation data D1 to the broadcast user device 20A or other viewing user devices 20B. Furthermore, the viewing user device 20B of the authorized viewing user 10B accepts the input operation when it has not received the input operation data D1 from the broadcast user device 20A. In other words, when this viewing user device 20B receives the input operation data D1 from the broadcast user device 20A, it controls the browser screen 57 by prioritizing the received input operation data D1. When the viewing user 10B and the broadcast user 10A operate the browser screens 57 respectively output to different display objects 58, the display of each browser screen 57 is controlled by the input operation of the viewing user 10B and the input operation of the broadcast user 10A, respectively.

[0127] As described above, the viewing user device 20B of the unauthorized viewing user 10B does not accept input operations on the viewing user's browser screen 57. Based on the input operation data of each broadcasting user device 20A, this viewing user device 20B makes a resource transmission request D2 to the web server 13, and receives the resource D3 in response to the transmission request.

[0128] When one display object 58 is simultaneously displayed on both the display 109 of the host user device 20H and the guest user device 20G, and both the host user and the guest user perform input operations, the input operation that is received first may take priority. That is, for example, if the host user device 20H receives input operation data D1 transmitted from the guest user device 20G and then receives an input operation from the host user, the guest user's input operation takes priority. If the host user device 20H receives input operation data transmitted from the guest user device 20G and then receives an input operation from the host user, the host user's input operation takes priority. The same applies to the guest user device 20G. When the input operation on one browser screen 57 is completed, the other distribution user 10A and viewing user 10B can also perform input operations.

[0129] Furthermore, for example, when different display objects 58 are simultaneously operated by different broadcast users 10A, browser screens 57 corresponding to the respective input operations are output to the respective display objects 58.

[0130] FIG. 17 shows an example of a virtual space screen 60 output to display 109 of host user device 20H in collaborative distribution. Virtual space screen 60 includes avatar object 52A corresponding to the host user. If avatar object 52B of a guest user is included in the field of view of virtual camera 54, that avatar object 52B is drawn. Virtual space screen 60 also includes mini-avatar 52C, which is a simplified avatar object 52 corresponding to authorized viewing user 10B. The mini-avatar 52C has a smaller processing load for drawing than the avatar object 52 of distributor user 10A. For example, mini-avatar 52C is a two-dimensional image.

[0131] According to the second embodiment, in addition to the effects (1-1) to (1-7) of the first embodiment, the following effects can be obtained. (2-1) In the second embodiment, in addition to the distribution user 10A who performs collaboration distribution, authorized viewing users 10B can also perform input operations to control the browser screen 57. This allows the display content of the browser screen 57 to be provided to a larger number of users. This increases the advertising effect.

[0132] The above-described embodiments can be modified as follows: The above-described embodiments and the following modifications can be combined with each other within the scope of technical compatibility. <First output control unit> In each of the above embodiments, the first output control unit 23 of the user device 20 distributes and views videos using the client rendering method. Alternatively, distribution and viewing may be performed using any of the above methods other than the client rendering method. The browser rendering method is a method in which the distributor user device 20A and the viewer user device 20B display videos using independent web browsers stored in the storage 104. In this case, the distributor user device 20A transmits video data and the like to the management server 12. The user device 20 displaying the videos receives web data from the management server 12, which is written in a markup language such as HTML (Hyper Text Markup Language) and uses CSS (Cascading Style Sheets), JavaScript (registered trademark), and the like. The web browser program launched on the user device 20 draws avatar objects and other objects using JavaScript running on the browser. In this method, model data of avatar objects and the like is not stored in the user device 20. In the browser rendering method, an application does not need to be installed on the user device 20. Alternatively, an application including an in-app browser may be installed on the user device 20.

[0133] In the browser rendering method, at least one of well-known methods such as static rendering (Static Server-Side Rendering), server-side rendering (Server-Side Rendering), client-side rendering (Client-Side Rendering), pre-rendering (CSR with Prerendering), and hydration rendering (SSR with (Re)Hydration) can be used as a method for generating a file that displays the virtual space screen 60 by rendering. Rendering here refers to interpreting data written in a markup language or the like and calculating the layout of image elements and characters to be displayed on the virtual space screen 60.

[0134] In static rendering, the web server 13 stores static HTML files in advance. The user device 20 downloads the static HTML files from the web server 13. The user device 20 outputs the rendered HTML files to the display 109 to display the virtual space screen 60. In server-side rendering (SSR), the web server 13 performs rendering and other operations in response to a request from the user device 20 and transmits the rendered HTML files to the user device 20. The user device 20 outputs the downloaded HTML files to the display 109 to display the virtual space screen 60. The web server 13 also includes an application server. In client-side rendering (CSR), the user device 20 performs rendering by executing JavaScript (registered trademark). At this time, the user device 20 downloads a simple HTML file from the web server 13, as well as necessary JavaScript (registered trademark), CSS files, and the like. In pre-rendering, which is a type of CSR, the user device 20 generates and stores some HTML files in advance, and downloads and renders the necessary content from the web server 13 in response to a request. In hydration rendering, the web server 13 generates and records an unrendered HTML file. The user device 20 downloads the HTML file from the web server 13 and downloads and renders the necessary data based on the HTML file. In each of the above methods, the user device 20 may obtain the necessary data from the management server 12 instead of the web server 13.

[0135] In the video distribution method, the broadcast user device 20A generates video data. Specifically, the broadcast user device 20A uses motion data and the like to draw objects and generate video data. The broadcast user device 20A then transmits the encoded video data and audio data to the management server 12. The management server 12 then transmits the video data and audio data to the viewer user device 20B. The viewer user device 20B displays video on the display 109 based on the video data received from the broadcast user device 20A.

[0136] In the server distribution method, the management server 12 generates video data. The management server 12 receives motion data and audio data from the broadcasting user device 20A. The management server 12 then generates output data for outputting video based on this data. The management server 12 transmits the generated output data to the broadcasting user device 20A and the viewing user device 20B. The broadcasting user device 20A and the viewing user device 20B output video to the display 109 based on the received output data.

[0137] In this way, either the user device 20 or the management server 12 is responsible for generating data for outputting video. The data to be generated differs depending on the distribution method. The device responsible for generating the data can be changed depending on the distribution method. In the above embodiments, the virtual space 50 is exemplified as a space where an exhibition is held, but the present invention is not limited to this virtual space 50. The virtual space 50 may be any space where a projection screen is displayed, and may be a space where events such as live performances or matches are held. Alternatively, the virtual space 50 may be a virtual store or an entertainment facility such as an amusement park. Alternatively, the virtual space 50 may not hold any events, but may simply allow users 10 to explore or communicate with each other. There may not be any objects corresponding to exhibits within the virtual space 50.

[0138] <First display control process> In the above embodiment, the first display control process is a process for controlling the position of virtual camera 54. Alternatively or additionally, the first display control process may be a process for controlling the direction of virtual camera 54 without changing its position. For example, when the operation direction is the X-positive direction, the orientation of virtual camera 54 is moved in the X-positive direction while maintaining the position of virtual camera 54. Alternatively, the first display control process may be a process for moving object 51, such as avatar object 52, in a direction indicated by user 10.

[0139] <Second output control section> In the above embodiments, the first output control unit 23 outputs the browser screen 57 as the projection screen. Alternatively or additionally, the projection screen may be a screen on which information recorded in the storage 104 of the user device 20 or information acquired from an external server is output. This information may be, for example, image data including still images and videos, an image list, a document file, etc. Alternatively, this information may be video data acquired from an external server and playable on the user device 20, or call data for conversations using videos of other users. Alternatively, this information may be the top page of a video playback application. Specifically, this information may be image data such as photos or videos recorded using the album function of the storage 104, books such as manga, or data created using presentation software such as Microsoft PowerPoint (registered trademark). The user 10 can scroll this projection screen vertically, horizontally, transition between screens, and play or stop videos, just like the browser screen 57.

[0140] The second output control unit 24 scrolls the display area 97 of the browser screen 57 horizontally when the operation determination unit 22 determines that the input operation includes a combination of a designation operation and a horizontal sliding operation. Furthermore, the second output control unit 24 controls the virtual camera 54 when the operation determination unit 22 determines that the input operation does not include a designation operation and includes only a horizontal sliding operation. The second output control unit 24 may perform the reverse processing depending on these input operations. That is, the second output control unit 24 may control the virtual camera 54 when the input operation includes a designation operation and a horizontal sliding operation, and may control the display of the browser screen 57 when the input operation does not include a designation operation and includes only a horizontal sliding operation.

[0141] When the operation determination unit 22 determines that the input operation includes a lock operation, the second output control unit 24 may determine that input operations during the period between the input of the lock operation and the input of the unlock operation are operations on the browser screen 57. For example, the second output control unit 24 determines that all coordinates input between a long press operation on the input device 110 and the long press operation for unlocking are operations on the browser screen 57. The second output control unit 24 performs processing on the browser screen 57 based on input operations during the period from the lock operation to the input of the unlock operation for unlocking the lock operation. Therefore, the user 10 can scroll the display area 97 of the browser screen 57 horizontally by simply performing a horizontal sliding operation within the browser area 56, without performing a designation operation for designating horizontal scrolling of the browser screen 57.

[0142] The second output control unit 24 synchronizes the projection screens, such as the browser screen 57, by transmitting input operation data to the other user devices 20. Alternatively, the second output control unit 24 may transmit data (image data) capturing the browser screen 57 to the other user devices 20. The user devices 20 that receive this data project the received data onto the display object 58.

[0143] <Second display control process> In each of the above embodiments, the second output control unit 24 performs the second display processing of vertically scrolling the browser screen 57, horizontally scrolling the browser screen 57, selecting a link, and playing or pausing a video in response to an input operation by the user 10. The second output control unit 24 may be capable of performing at least one of these processes. Alternatively, the second output control unit 24 may perform processing other than the above-described processing as long as it is processing on a projection screen including the browser screen 57. For example, the second output control unit 24 may close or open the browser screen 57 in response to an input operation by the user 10. Furthermore, the second output control unit 24 may operate a GUI element, such as selecting one of the items displayed in a pull-down menu, in response to an input operation by the user 10.

[0144] <Input coordinate acquisition section> In each of the above embodiments, the input coordinate acquisition unit 21 acquires coordinates where contact of a finger, a stylus, or the like is detected on a touch panel, which is the input device 110. The input coordinate acquisition unit 21 acquires input coordinates according to the input device 110. If the input device 110 is a mouse or a controller, the input coordinate acquisition unit 21 acquires coordinates on the screen that the user 10 specifies using the mouse or controller. If the input device 110 includes a tracking sensor that detects the position of the pupils, the input coordinate acquisition unit 21 detects the direction of the user's gaze and detects the position on the screen specified by the user 10. If the input device 110 includes a tracking sensor that detects the position of a part of the user's 10's body, such as a finger, the input coordinate acquisition unit 21 detects the direction in which the user 10 points the finger and detects the position on the screen specified by the user 10.

[0145] <Operation judgment section> In each of the above embodiments, the operation determination unit 22 determines whether or not an input operation is an operation on the browser screen 57. Alternatively, the second output control unit 24 may determine whether or not an input operation is an operation on the browser screen 57 based on the input coordinates acquired from the operation determination unit 22.

[0146] <Administration Server> The management server 12 may be a single server device equipped with multiple virtual servers, or may be a set of multiple physical servers. For example, the management server 12 may be a set of a combination of a server having a distribution management unit 30, a server having a browsing history information receiving unit 32, and a server having a performance recording unit 33.

[0147] <Browsing history information receiving unit> The browsing history information receiving unit 32 may receive browsing information from the web server 13. The web server 13 records an access log of resources specified by URLs that can be registered by the user 10. The access log includes the connection date and time (browsing start time), the disconnection date and time (browsing end time), identification information of the connected user device 20, operation details, etc. The web server 13 transmits the access log to the management server 12. The browsing history information receiving unit 32 of the management server 12 records the received access log as browsing summary information 42 in the browsing summary information storage unit 36. The performance recording unit 33 records performance information 43 based on the access log in the performance information storage unit 37.

[0148] <Permission to operate the projection screen> In the first embodiment, only the broadcasting user 10A can operate the projection screen. In the second embodiment, the broadcasting users 10A, including the host user and guest user, and the authorized viewing users 10B can operate the projection screen, and only the operations of the broadcasting user 10A are reflected (synchronized) in the videos of the other users 10. The attributes of the users 10 who can operate the projection screen and the attributes of the users whose input operations are synchronized with other user devices 20 can be changed as appropriate. In other words, for each attribute of each user 10, whether or not the browser screen 57 can be operated is set, and if the browser screen 57 can be operated, whether or not the browser screen 57 can be synchronized with other devices is set. Note that the browser screen 57 may be synchronized with all other user devices 20 other than the user's own device, or the browser screen 57 may be synchronized with only some of the other user devices 20.

[0149] For example, the attributes of these users 10 may be in the following forms: [First aspect] (A) The host user device 20H accepts an input operation on the browser screen 57, and transmits data for reflecting the operated browser screen 57 on another device. (B) The guest user device 20G and the viewer user device 20B of the authorized viewer user 10B accept input operations on the browser screen 57, but do not transmit data for reflecting the operated browser screen 57 on other devices. [Second aspect] (C) The host user device 20H accepts an input operation on the browser screen 57 and transmits data for reflecting the operated browser screen 57 on another device. (D) The guest user device 20G and the viewing user device 20B of the authorized viewing user 10B accept input operations on the browser screen 57. When a synchronization instruction, which is an operation different from the input operation on the browser screen 57, is input to the user device 20 by these users 10, the user device 20 transmits data for reflecting the browser screen 57 on another device. The user device 20 does not transmit data for reflecting the browser screen 57 on another device unless a synchronization instruction is input. The synchronization instruction is input by operating a synchronization button or the like displayed on the virtual space screen 60.

[0150] The data to be reflected in the other devices is data generated by the user device 20 or the management server 12, and it is sufficient that the user device 20 receiving this data can determine the operation content of the user 10, who is the operator of the resource and the browser screen 57. For example, the data to be reflected in the other devices may include a URL that identifies the resource. Furthermore, the data to be reflected in the other devices may include data that indicates the operation content of the first user device 20 used by the user 10, who is the operator. Upon detecting an operation by the user 10, the first user device 20 transmits data indicating the operation content to the management server 12. The management server 12 generates an identifier as needed based on the data indicating the operation content. The management server 12 then transmits the identifier to the other devices. The second user device 20, which is the other device and receives the identifier, determines the operation content based on the identifier and displays the browser screen 57 according to the operation content.

[0151] When the input operation of the host user and the input operation of the user 10 of the user device 20 overlap, the input operation of the host user takes priority. Alternatively, the user device 20 may give priority to the input operation of the browser screen 57 of the user 10 of the user device 20.

[0152] In another embodiment, all users 10 may be able to operate the projected screens, and the projected screens may not be synchronized. Alternatively, another user 10 may be able to operate a browser screen 57 that a user 10 has launched (displayed) or that the user 10 has operated once. In other words, in one embodiment, only the host user may be able to operate the projected screen and synchronize the operations, while users 10 other than the host user may be able to operate the projected screen. For example, after a first user 10 launches a browser screen 57, if a predetermined condition is met, a second user 10 may be able to operate the browser screen 57. In this embodiment, the user device 20 of the second user 10 who has performed an operation midway transmits data for synchronizing the browser screen 57 to another device if the second user 10 has the authority to synchronize the browser screen 57. If the second user 10 does not have the authority to synchronize the browser screen 57, the user device 20 of the second user 10 does not transmit data for synchronizing the browser screen 57 to another device. The predetermined condition is that the host user is not performing an operation. Alternatively, the predetermined condition is that the user device 20 has accepted an operation from the user 10.

[0153] <Avatar Object> In each of the above embodiments, the user device 20 draws the avatar object 52 corresponding to the broadcast user 10A using tracking data or emote data that detects the movement of the broadcast user 10A. This allows the movement of the broadcast user 10A or the movement selected by the broadcast user 10A to be reflected in the avatar object 52. Alternatively or additionally, the user device 20 may temporarily or continuously draw the avatar object 52 without using tracking data or emote data. The first output control unit 23 controls the avatar object 52 based on an input operation by the user 10. The avatar object 52 performs an action in response to the input operation by the user 10. In this manner, the load of the drawing process on the user device 20 can be reduced when drawing a large number of avatar objects 52 in the virtual space 50.

[0154] In each of the above embodiments, the user device 20 renders an avatar object 52 corresponding to the broadcast user 10A. Alternatively, the user device 20 may render an image of the virtual space 50 from a first-person perspective, without rendering the avatar object 52.

[0155] <Availability of video streaming> In the above embodiments, it is assumed that the broadcast user device 20A broadcasts videos. Alternatively or additionally, the application implemented by the user device 20 may be an application that enables the user 10 to move around the virtual space 50 without broadcasting videos. In other words, the management server 12 does not perform processing for broadcasting videos, but instead provides asset data of the virtual space 50 in response to a request from the user device 20. The user device 20 outputs a virtual space screen 60. However, if an avatar object 52 corresponding to another user 10 is located in the same virtual space 50, that avatar object 52 may be displayed on the virtual space screen 60. In this embodiment, the user 10 launches an application and selects the desired virtual space 50. The user device 20 displays a virtual space screen of the virtual space 50 as seen from each virtual camera 54. In this embodiment, the user 10 can operate the projection screen in the same way as the broadcast user 10A in the first embodiment.

[0156] <user> In the above embodiments, the broadcasting user 10A refers to a user who transmits at least one of information related to images (video) and information related to audio. For example, the broadcasting user 10A is a user who broadcasts videos independently (first embodiment) or a user who performs collaborative broadcasting in which multiple people can participate (second embodiment). Alternatively, the broadcasting user 10A may be a user who hosts or organizes video chats or voice chats in which multiple people can participate and / or watch, or events (parties, etc.) in virtual spaces in which multiple people can participate and / or watch. In other words, the broadcasting user 10A can also be referred to as a host user, organizer user, or organizer user.

[0157] On the other hand, the viewing user 10B refers to a user who receives at least one of information related to video and information related to audio. However, the viewing user 10B not only receives the information but also reacts to the information. For example, the viewing user 10B is a user who watches a video broadcast or a collaboration broadcast. Alternatively, the viewing user 10B may be a user who participates in and / or watches a video chat, a voice chat, or an event. Therefore, the viewing user 10B can also be referred to as a guest user, a participating user, a listener, a viewing user, a supporting user, etc.

[0158] <Information Processing System> In each of the above embodiments, the management server 12 is managed by the PF provider, and the web server 13 is managed by the publisher. Alternatively or in addition to this, the user 10 may display an arbitrary web page on the browser screen 57, instead of a resource pre-associated with the virtual space 50 as in each of the above embodiments. For example, when the user 10 finds a display object 58 in the virtual space 50, the user 10 may input a search word or the like in a predetermined area and execute a search, thereby outputting an arbitrary browser screen 57. The predetermined area is, for example, a text box displayed in the UI area 55.

[0159] In the above embodiments, the user device 20 is an information processing device such as a smartphone, a mobile phone, a tablet terminal, a personal computer, a console game machine, or a wearable computer such as a head-mounted display. Alternatively, the information processing system may be a system installed in a video distribution studio. The information processing system includes an information processing device, a server, a sensor unit worn on the body of the distribution user 10A, a tracking system that detects the position of the sensor unit, an operation unit, a speaker, a display, etc. The tracking system may include a multi-axis laser emitter that emits pulsed laser light for synchronization. The sensor unit includes a sensor that detects laser light and detects its own position and orientation while synchronizing with the synchronization pulse.

[0160] By executing an application, the user device 20 functions as an input coordinate acquisition unit 21, an operation determination unit 22, a first output control unit 23, and a second output control unit 24. At least one of these functions may be executed by the management server 12 or another device in the information processing system 11. Furthermore, by executing a program, the management server 12 functions as a distribution management unit 30, a browsing history information receiving unit 32, and a performance recording unit 33. At least one of these functions may be executed by the user device 20 or another device in the information processing system 11. Furthermore, the management server 12 includes a user management information storage unit 34, a browsing aggregation information storage unit 36, and a performance information storage unit 37. At least one of these functions may be included in the user device 20 or another device in the information processing system 11.

Claims

1. one or more computers, a first output control unit that outputs a virtual space image of the three-dimensional virtual space viewed from a virtual camera to a display that is visually recognized by a user; a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image; an input coordinate acquisition unit that acquires input coordinates to the input device of the user; functioning as an operation determination unit that determines whether or not the input operation by the user is an operation on an input area corresponding to the projection screen, based on the acquired input coordinates; when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process regarding the three-dimensional virtual space other than control of the projection screen; A program in which, when the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates.

2. When the operation determination unit determines that the input operation is a scroll operation in which the input coordinates continuously change along a first axis, The program according to claim 1 , wherein the second output control unit scrolls the projection screen in the direction of the scroll operation.

3. When the operation determination unit determines that the input operation is a combination of a predetermined designation operation and a scroll operation in which the input coordinates continuously change along a second axis, The program according to claim 1 , wherein the second output control unit scrolls the projection screen in the direction of the scroll operation.

4. When the operation determination unit determines that the input operation includes only a scroll operation in which the input coordinate continuously changes along a second axis, 4. The program according to claim 1, wherein the first output control unit performs the first display control process.

5. 5. The program according to claim 1, wherein the first output control unit changes at least one of a position and a direction of the virtual camera based on the input coordinates as the first display control process.

6. The program according to any one of claims 1 to 5, wherein when the operation determination unit determines that the input operation is a selection operation for selecting any position on the projection screen, the second output control unit executes processing according to the position of the selection operation.

7. When the operation determination unit determines that the input operation in the input area includes a lock operation that specifies that the input operation is an operation on the projection screen, The program according to any one of claims 1 to 6, wherein the second output control unit performs processing on the projection screen based on the input operation within the period from the lock operation to the input of an unlock operation that unlocks the lock operation.

8. 8. The program according to claim 1, wherein the second output control unit outputs a browser screen displayed using a browser as the projection screen.

9. the first output control unit outputs an avatar object corresponding to the user to the three-dimensional virtual space; The program according to any one of claims 1 to 8, further comprising transmitting data for outputting a video including the avatar object and the projection screen to a user device used by another user.

10. The program described in any one of claims 1 to 9, wherein the second output control unit accepts input operations into the input area when the user is in a distribution mode in which the user distributes a video, and does not accept input operations into the input area when the user is in a viewing mode in which a video is output based on video data sent by another user.

11. One or more computers a first output control process for outputting a virtual space image of the three-dimensional virtual space viewed from a virtual camera to a display visually recognized by a user; a second output control process for outputting a projection screen operable by the user within the virtual space image; an input coordinate acquisition process for acquiring input coordinates to the input device of the user; and executing an operation determination process for determining whether or not the input operation of the user is an operation within an input area corresponding to the projection screen, based on the acquired input coordinates; performing a first display control process for the three-dimensional virtual space other than control for the projection screen when it is determined in the operation determination process that the input operation is not an operation on the input area; When it is determined in the operation determination process that the input operation is an operation on the input area, a second display control process is performed on the projection screen according to the input coordinates.

12. a first output control unit that outputs a virtual space image of the three-dimensional virtual space viewed from a virtual camera to a display that is visually recognized by a user; a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image; an input coordinate acquisition unit that acquires input coordinates to the input device of the user; an operation determination unit that determines whether or not the input operation by the user is an operation on an input area corresponding to the projection screen, based on the acquired input coordinates; when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process regarding the three-dimensional virtual space other than control of the projection screen; The operation determination unit determines that the input operation is an operation to the input area. When the determination is made, the second output control unit performs a second display control process on the projection screen according to the input coordinates.

Images