Server device and network control method

Abstract

The present invention relates to a server device and a network control method configured so that it is possible to reduce the traffic of object data constituting a virtual space. A live entertainment broker filters first object data, which is object data for a first object displayed in the virtual space, and second object data, which is object data for a second object, on the basis of determined filtering conditions, and transmits the object data after the filtering process to a rendering server. The present invention is applicable to, for example, a data processing system that provides a live entertainment service in which a live performance by an artist is provided in real time.

Description

[Technical field]

[0001] The present disclosure relates to a server device and a network control method, and more particularly to a server device and a network control method that are capable of reducing traffic of object data that constitutes a virtual space. [Background technology]

[0002] XR (Extended Reality) is being considered for application in various application domains such as entertainment, healthcare, education, etc. For example, in entertainment services, it is being considered to construct a virtual space in which a user can sit in a virtual stadium and watch a game while talking to a user who is in a different location in the real world but is sitting next to him in the stadium (for example, see Non-Patent Document 1).

[0003] As another example, in the entertainment service, a live entertainment service may be considered in which a live space including artists and an audience is created in a virtual space. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] 3GPP TR 26.928, “6.2.3 Viewport-dependent Streaming”, “A.22 Use Case 21: Immersive 6DoF Streaming with Social Interaction” Summary of the Invention [Problem to be solved by the invention]

[0005] When constructing a virtual space for the above live entertainment service, even if one server instance is assigned to a few audience members, if the audience is about 1 million people, a huge number of server instances corresponding to that number will be required. In addition, all live-action video data of the artists and audience members must be sent to each instance, which can cause an explosive increase in traffic between instances and lead to the system collapsing.

[0006] The present disclosure has been made in consideration of these circumstances, and makes it possible to reduce traffic of object data that constitutes a virtual space. [Means for solving the problem]

[0007] This technology No. 1 The server device includes a broker that filters first object data, which is object data of a first object, and second object data, which is object data of a second object, displayed in the virtual space, based on the determined filter conditions, and transmits the filtered object data to a rendering server. Based on the load status of the cloud or the traffic status between the cloud and a client device that displays an image based on the object data, a selection is made as to whether the rendering server should be placed in the cloud or in the client device for each type of object data. . The server device of the second aspect of the present technology is equipped with a broker that filters each of first object data, which is object data of a first object displayed in a virtual space, and second object data, which is object data of a second object, based on determined filter conditions and transmits the object data after the filtering process to a rendering server. When the rendering server is shared by a plurality of client devices that display images based on the object data, whether or not to share the object data among the plurality of client devices is selected for each type of object data.

[0008] This technology Third In the network control method of the aspect, a server device filters first object data, which is object data of a first object, and second object data, which is object data of a second object, displayed in a virtual space, based on a determined filter condition, and transmits the object data after the filtering process to a rendering server. Based on the load status of the cloud or the traffic status between the cloud and a client device that displays an image based on the object data, a selection is made as to whether the rendering server should be placed in the cloud or in the client device for each type of object data. . A network control method according to a fourth aspect of the present technology includes a server device filtering first object data, which is object data of a first object displayed in a virtual space, and second object data, which is object data of a second object, based on determined filter conditions, and transmitting the object data after the filtering process to a rendering server. When the rendering server is shared among a plurality of client devices that display images based on the object data, it is selected for each type of object data whether or not to share the object data among the plurality of client devices.

[0009] This technology 1st~4thIn one aspect, first object data, which is object data of a first object displayed in a virtual space, and second object data, which is object data of a second object, are each filtered based on the determined filter conditions, and the object data after the filtering process is transmitted to a rendering server. Furthermore, in the first and third aspects of the present technology, the placement of the rendering server is selected for each type of object data, whether to place the rendering server in the cloud or in the client device, based on the load status of the cloud or the traffic status between the cloud and a client device that displays an image based on the object data. Furthermore, in the second and fourth aspects of the present technology, when the rendering server is shared among a plurality of client devices that display images based on the object data, whether to share the rendering server among the plurality of client devices is selected for each type of object data.

[0010] The server device according to one aspect of the present technology can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting the program via a transmission medium or by recording the program on a recording medium.

[0011] The server device may be an independent device, or may be an internal block constituting a single device. [Brief description of the drawings]

[0012] [Figure 1] A diagram illustrating a live entertainment service realized in a data processing system to which the technology disclosed herein is applied. [Diagram 2] FIG. 2 is a block diagram of a data processing system for providing the live entertainment service of FIG. 1. [Diagram 3] 11 is a flowchart illustrating a process of generating object data. [Figure 4] FIG. 1 is a block diagram illustrating a configuration of a data processing system. [Diagram 5] FIG. 2 is a diagram illustrating the operation of the data processing system. [Figure 6] FIG. 2 is a diagram illustrating an object data subscription. [Figure 7] 11 is a flowchart illustrating a process until object data is published. [Figure 8] FIG. 13 is a diagram illustrating an example of an object data publication structure. [Figure 9]11 is a flowchart illustrating a transfer process of object data immediately before rendering is performed. [Figure 10] FIG. 11 is a diagram illustrating a rendering process. [Figure 11] 11 is a flowchart illustrating a rendering process in which the execution location of a renderer is dynamically changed. [Figure 12] FIG. 13 is a diagram illustrating a rendering data rounding process. [Figure 13] 13A to 13C are diagrams illustrating an example of a display image of the rendering data rounding process. [Figure 14] 13 is a flowchart illustrating a rendering process when a rendering data rounding process is performed. [Figure 15] 11A and 11B are diagrams illustrating a rendering data rounding process performed for each type of data. [Figure 16] 11A and 11B are diagrams illustrating a rendering data rounding process performed for each type of data. [Figure 17] FIG. 1 illustrates a local live entertainment broker. [Figure 18] 13 is a flowchart illustrating an object data transfer process using a local live entertainment broker. [Figure 19] FIG. 13 is a diagram illustrating variations of a filter condition. [Figure 20] FIG. 11 is a diagram illustrating an example of filter data. [Figure 21] FIG. 21 is a diagram conceptually showing a filtering process of the calculation formula in FIG. 20. [Figure 22] FIG. 13 is a diagram illustrating an example of subdivision setting of weighting parameters. [Diagram 23] 11A and 11B are diagrams illustrating examples of weight parameters of filter data and weight parameter calculation formulas. [Figure 24] FIG. 24 is a diagram conceptually showing a filtering process of the calculation formula of FIG. 23. [Diagram 25] 11A and 11B are diagrams illustrating differences in display in filtering processing. [Figure 26]FIG. 13 is a diagram illustrating the filtering process of a local live entertainment broker. [Figure 27] FIG. 11 is a diagram illustrating an example of filter data. [Figure 28] FIG. 2 is a block diagram illustrating an example of a functional configuration of a client device. [Figure 29] A block diagram showing the functional configuration of a server side renderer. [Diagram 30] FIG. 2 is a block diagram showing the functional configuration of an object data server. [Diagram 31] FIG. 2 is a block diagram showing a functional configuration of an object data publisher. [Diagram 32] FIG. 2 is a block diagram showing a functional configuration of an object data subscriber. [Diagram 33] FIG. 2 is a block diagram showing a functional configuration of an edge resource orchestrator. [Diagram 34] FIG. 2 is a block diagram showing the functional configuration of a live entertainment broker. [Diagram 35] FIG. 1 is a block diagram showing an example of the configuration of cloud computing to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a mode for carrying out the technology of the present disclosure (hereinafter, referred to as an embodiment) will be described with reference to the attached drawings. In this specification, the description of "and / or" means that both "and" and "or" are possible. In addition, in this specification and the drawings, components having substantially the same functional configuration are given the same reference numerals to avoid redundant explanation. The description will be given in the following order. 1. Overview of Live Entertainment Services 2. Example of data processing system configuration 3. Object Data Publication 4. Object data transfer process before rendering 5. Explanation of the rendering process 6. Rendering data rounding 7. Rounding of rendering data by data type 8. Combining Rendering Data Rounding with a Client-Side Renderer 9. Local Live Entertainment Brokers 10.Variations in Filter Conditions 11. Filtering Examples 12. Example of weighting parameter subdivision settings 13. Local Live Entertainment Broker Filtering 14. Block diagram of functional components 15. Cloud Computing Configuration Examples

[0014] <1. Overview of Live Entertainment Services> First, with reference to FIG. 1, a live entertainment service realized by a data processing system to which the technology of the present disclosure is applied will be described.

[0015] The data processing system 1 described in FIG. 2 and subsequent figures is a system that provides a live entertainment service in which a live venue including artists and audiences is constructed in a virtual space, and live performances of artists are provided in real time.

[0016] The live entertainment service provided by the data processing system 1 involves one artist and a large number of audience members, each of whom is located in a different place in the real world. Each of the large number of audience members is a live viewer. Of course, there may be more than one artist.

[0017] As shown in FIG. 1, if one audience member of interest is the viewer 11ME, the other audience members can be classified into nearby audience members 11AJ and non-neighboring audience members 11NJ based on their relationship with the viewer 11ME. The nearby audience members 11AJ and non-neighboring audience members 11NJ are classified, for example, by the distance from the viewer 11ME (Myself) based on the seating position in the live venue in the virtual space, their interest, and the like. The nearby audience members 11AJ and non-neighboring audience members 11NJ differ in the level of detail (LOD) expressed in the virtual space. When there is no particular distinction between the nearby audience members 11AJ and the non-neighboring audience members 11NJ, they are simply referred to as the audience members 11AU.

[0018] The virtual space of the live entertainment service is constructed by arranging 3D objects (hereinafter, simply referred to as objects) of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ in a virtual live venue 12. Each 3D object of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ is arranged in the virtual live venue 12, for example, by using a volumetric capture technique, reflecting the movement of the person in the real world in real time. The volumetric capture technique is a technique for generating a 3D object of a subject from a moving image captured from multiple viewpoints, and generating a virtual viewpoint image of the 3D object according to an arbitrary viewing position. In this embodiment, the method of generating and displaying the 3D object is not limited, and for example, a technique for generating an object of a subject using moving images captured from two different directions may be used.

[0019] The data of each object of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ is composed of the shape of the object in the virtual space at every moment, the position and movement in the virtual space, and the light (color), sound, smell, wind pressure, heat data, etc. generated by the object (observed from outside the object). However, for simplicity, in this embodiment, the data of each object will be described by focusing on visible and audible data. The visible data of each object is composed of video data such as mesh data, texture data, and motion data, and the audible data is composed of audio data.

[0020] In a live entertainment service, for example, when a viewer 11ME is looking in the direction 13A of the artist 11AR, objects of the artist 11AR present within the viewer's 11ME's viewpoint / visibility scope, as well as objects of nearby audience members 11AJ-1 and non-neighboring audience members 11NJ-1 in that direction 13A, are displayed on the viewer's 11ME's display.

[0021] For example, if the viewer 11ME changes his / her line of sight and looks in a direction 13B different from the direction 13A of the artist 11AR, nearby audience members 11AJ-2 and non-neighboring audience members 11NJ-2 that are within the viewpoint / visibility scope of that direction 13B will be displayed on the display.

[0022] <2. Example of data processing system configuration> FIG. 2 is a block diagram showing components of a data processing system to which the present technology is applied, which provides the above-mentioned live entertainment service.

[0023] In the data processing system 1, a client device 20 is provided for each of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighborhood audience 11NJ. That is, a client device 20 is provided for each of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighborhood audience 11NJ in a one-to-one correspondence. The client device 20 includes a sensor 21 and a display 22.

[0024] Also, a server device 30 is provided on the edge cloud corresponding to the client device 20. In this embodiment, for simplicity of explanation, it is assumed that the client device 20 and the server device 30 have a one-to-one relationship, but a one-to-many relationship, that is, a form in which one server device 30 manages multiple client devices 20, may also be used. The server device 30 includes an object data server 31, an object data publisher 32, an object data subscriber 33, and a server-side renderer 34.

[0025] The sensor 21 and the display 22 of the client device 20 do not have to be integrated into one device, and may be separate devices. Similarly, each of the object data server 31, the object data publisher 32, the object data subscriber 33, and the server-side renderer 34 of the server device 30 may be configured as two or more separate devices.

[0026] Furthermore, the data processing system 1 has a server device 40 on the center cloud that is shared by each server device 30 of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighborhood audience 11NJ.

[0027] The server device 40 has a live entertainment broker 41, a virtual space DB 42, and an object metadata DB 43, and the live entertainment broker 41 includes a subscription manager 51 and a filter 52. Each of the live entertainment broker 41, the virtual space DB 42, and the object metadata DB 43 may also be configured separately as two or more devices, rather than as a single integrated device.

[0028] The edge cloud is a cloud on the edge side close to the client device 20. For example, when the network is a mobile phone communication network, the edge cloud is composed of base stations, etc. The center cloud is a cloud such as a core network other than the edge cloud.

[0029] Clouds including edge clouds and center clouds are composed of multiple nodes and a network connecting them. The network is composed of a communication network or a communication path of any communication standard, such as the Internet, a public telephone line network, a wide area communication network for wireless mobile devices such as a so-called 4G line or 5G line, a wide area network (WAN), a local area network (LAN), a wireless communication network that communicates in accordance with the Bluetooth (registered trademark) standard, a communication path for short-distance wireless communication such as NFC (Near Field Communication), a communication path for infrared communication, a communication network for wired communication in accordance with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus), etc. Each node that constitutes the cloud is composed of a network connection device such as a sensor device, a router, a modem, a hub, a bridge, a switching hub, a base station control device, an exchange, and a server device. The server device as a node and the applications executed on the device realize the functions described below as an object data server 31, an object data publisher 32, an object data subscriber 33, a server-side renderer 34, a live entertainment broker 41, a virtual space DB 42, or an object metadata DB 43.

[0030] The sensor 21 is a device that acquires sensor data for generating objects of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, or the non-neighboring audience 11NJ. The sensor 21 is, for example, configured with an imaging sensor that generates an RGB image. The sensor 21 may be configured with the same or different sensor devices, such as an imaging sensor and a distance measurement sensor. The sensor 21 transmits the acquired sensor data to the object data server 31.

[0031] The display 22 receives rendering data transmitted from the server-side renderer 34 of the edge cloud and displays live video of the artist 11AR's objects performing at the virtual live venue 12. The live video displayed on the display 22 also includes objects of the nearby audience 11AJ and non-neighboring audience 11NJ according to the viewpoint / visible scope of the viewer 11ME. The live video displayed on the display 22 is a 2D image of objects such as the artist 11AR, nearby audience 11AJ, non-neighboring audience 11NJ, etc., generated according to the viewpoint / visible scope of the viewer 11ME.

[0032] The object data server 31 generates object data from the sensor data transmitted from the sensor 21 of the client device 20. The generated object data is temporarily stored in an internal storage unit, and then transferred to the object data publisher 32 or the object data subscriber 33.

[0033] The object data server 31 can generate not only current object data based on the sensor data successively received from the sensor 21, but also object data based on near-future predictions (e.g., future gaze predictions using gaze tracking).

[0034] The object data publisher 32 extracts information required for rendering an object from the object data supplied from the object data server 31 and transmits (transfers) it to the live entertainment broker 41 .

[0035] The object data subscriber 33 detects a change in viewpoint information of a subject (artist or audience) in the real world based on the object data supplied from the object data server 31. The viewpoint information includes the line of sight direction and the visible range.

[0036] The object data subscriber 33 negotiates with the subscription manager 51 of the live entertainment broker 41 regarding the display level of the object included in the viewpoint / visibility scope based on the viewpoint information of the detected subject. The display level of the object is the level of detail at which it is displayed. The object data subscriber 33 receives object data of the viewpoint / visibility scope from the filter 52 of the live entertainment broker 41 and supplies it to the server-side renderer 34.

[0037] The server-side renderer 34 generates rendering data based on the object data of the viewpoint / visible scope supplied from the object data subscriber 33. The server-side renderer 34 encodes the generated rendering data using a predetermined compression encoding method, and transmits the encoded stream data to the display 22 of the client device 20.

[0038] Data for each artist or audience object can be recorded in any data format. Geometry information (shape information) of an object can be stored, for example, as a set of points (point cloud) or in polygon mesh format represented by vertices and the connections between them. Color information of an object can be stored in UV texture format using a UV coordinate system or in multi-texture format stored as multiple 2D texture images.

[0039] Object rendering is a process of generating a 2D image (object image) of an object from the viewpoint of a virtual camera based on 3D model data of the object. The object image from the viewpoint of the viewer 11ME is generated by perspectively projecting the 3D model of the object onto the viewpoint / visibility scope of the viewer 11ME based on viewpoint information detected by the object data subscriber 33. As a compression encoding method for the object image, which is a generated 2D image, for example, the AVC (Advanced Video Coding) method, the HEVC (High Efficiency Video Coding) method, etc. can be adopted.

[0040] The subscription manager 51 of the live entertainment broker 41 negotiates with the object data subscriber 33 regarding the display level of the object. In other words, it determines the filter conditions for how to filter each object, such as the artist 11AR, the nearby audience 11AJ, the non-neighborhood audience 11NJ, etc., that exists within the viewpoint / visible scope of the viewer 11ME.

[0041] The filter 52 filters the object data of each object present within the viewpoint / visible scope of the viewer 11ME based on the filter conditions determined by the subscription manager 51. The filter 52 transmits the object data after the filtering process to the object data subscriber 33.

[0042] The virtual space DB 42 stores state information shared by each object. For example, the virtual space DB 42 stores seating information of a stadium or hall that is the virtual live venue 12, and information such as the shape, material, temperature, and humidity of the space.

[0043] The object metadata DB 43 stores attribute information of each object such as the artist 11AR, the nearby audience 11AJ, the non-neighborhood audience 11NJ, etc. The object metadata DB 43 stores query information for making it possible to refer to the attribute information of each object and query information for making it possible to refer to state information stored in the virtual space DB 42.

[0044] The object data generation process executed between the client device 20 and the server device 30 for each of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighborhood audience 11NJ will be described with reference to the flowchart in FIG.

[0045] First, in step S11, the sensor 21 of the client device 20 obtains sensor data obtained by observing a target subject, and transmits the sensor data to the object data server 31 of the corresponding server device 30.

[0046] In step S12, the object data server 31 receives the sensor data transmitted from the sensor 21 and generates object data based on the sensor data. The generated object data is stored in an internal storage unit. In generating the object data, state information stored in the virtual space DB 42 of the center cloud is referenced. The state information stored in the virtual space DB 42 is, for example, seat information of the stadium or hall that is the virtual live venue 12, and information such as the shape, material, temperature, and humidity of the space.

[0047] The object data generated for the viewer 11ME, the artist 11AR, the nearby audience 11AJ, or the non-neighboring audience 11NJ is composed of video data of moving images and audio data. In addition to generating object data at the present time, object data based on a prediction of the near future can also be generated.

[0048] As shown in Fig. 4, the client device 20 and the server device 30 are provided for each of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighborhood audience 11NJ. Note that, due to space limitations, only the object data server 31 and the server side renderer 34 are shown for the server device 30 in Fig. 4.

[0049] All devices constituting the data processing system 1 support a real-time control environment such as TSN (Time-Sensitive Networking) and are designed to guarantee the synchronicity of data exchanged between each device. In other words, in the data processing system 1, the time when sensor data was generated can be measured correctly, and the reservation of network / transport processing resources (classes such as transfer bandwidth, delay, and jitter) can be controlled in real time so that media synchronization during playback can be easily achieved.

[0050] An object data server 31 arranged in an edge cloud acquires sensor data generated by sensors 21 of one or more client devices 20 under its control, generates object data, and transmits it to other server devices 30. A server-side renderer 34 collects object data required for rendering from other server devices 30, and further adds shared state information such as data on the virtual live venue 12, expands it in memory, and renders it.

[0051] In a live entertainment service, the number of audience members is expected to be large, such as several million. If the number of client devices 20 corresponding to the viewers 11ME, the nearby audiences 11AJ, and the non-neighborhood audiences 11NJ becomes large, a huge amount of traffic may be generated, causing the system to collapse.

[0052] Traffic related to artists 11AR is essential and unavoidable, but traffic related to audiences should be reduced if possible. For example, it would be good to filter data according to the distance between audiences in the virtual space, such as "You can hear the movements, expressions, and voices of nearby audiences, but you only need to see loud voices and movements of distant audiences."

[0053] Therefore, in order to alleviate the excessive traffic, the data processing system 1 is provided with a live entertainment broker 41 that implements a traffic filtering function.

[0054] Specifically, the server-side renderer 34 of the client device 20 of the viewer 11ME is configured to obtain and render only the object data of the artists 11AR and nearby audience members 11AJ within the range visible to the viewer 11ME, and renders the object data of non-neighboring audience members 11NJ to a minimum or omits it, thereby reducing unnecessary traffic and reducing the load of the rendering process.

[0055] The operation of each of the client devices 20 and the server device 30 of the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighborhood audience 11NJ will be described with reference to FIG.

[0056] In Fig. 5, the components of the client device 20 and the server device 30 corresponding to the viewer 11ME, the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ are distinguished by the symbols "ME", "AR", "AJ", and "NJ", respectively. For example, the sensor 21ME and the display 22ME represent the sensor 21 and the display 22 of the client device 20 of the viewer 11ME. The object data server 31ME, the object data subscriber 33ME, and the server-side renderer 34ME represent the object data server 31, the object data subscriber 33, and the server-side renderer 34 of the server device 30 of the viewer 11ME. The sensor 21AR, the object data server 31AR, and the object data publisher 32AR represent the sensor 21 of the client device 20 of the artist 11AR, the object data server 31 of the server device 30, and the object data publisher 32 of the server device 30. The same is true for nearby audience 11AJ and non-neighborhood audience 11NJ.

[0057] The sensor 21AR of the artist 11AR generates sensor data obtained by sensing the artist 11AR, for example, an RGB image obtained by photographing the artist 11AR, and supplies the sensor data to the object data server 31AR.

[0058] The object data server 31AR generates object data from the sensor data transmitted from the sensor 21AR, stores the object data in an internal storage unit, and transfers the object data to the object data publisher 32AR.

[0059] The object data publisher 32AR extracts information required for rendering an object from the object data supplied from the object data server 31AR, and transmits it to the live entertainment broker 41.

[0060] The object data publisher 32AR checks the object data in the object data server 31AR when all updates of the object data based on the sensor data sequentially transferred from the sensor 21AR have been completed (snapshot at a predetermined period), extracts the object data that will affect the rendering, and transfers it to the live entertainment broker 41. When near-future object data is also predicted and generated, that object data is also extracted and transferred to the live entertainment broker 41.

[0061] The operations of the sensor 21, object data server 31, and object data publisher 32 for the nearby audience 11AJ and non-neighborhood audience 11NJ are basically the same as those of the artist 11AR described above, except that the objects being targeted are different, so a description thereof will be omitted. However, since the artist 11AR is the object of greatest interest in the live entertainment service, it is captured and transferred with the highest possible quality, whereas the nearby audience 11AJ and non-neighborhood audience 11NJ only need to be captured and transferred with the minimum necessary (certain) quality.

[0062] On the other hand, the sensor 21ME of the viewer 11ME generates sensor data obtained by sensing the viewer 11ME, for example, an RGB image obtained by photographing the viewer 11ME, and supplies the sensor data to the object data server 31ME.

[0063] The object data server 31ME generates object data from the sensor data transmitted from the sensor 21ME, stores the object data in an internal storage unit, and transfers the object data to the object data subscriber 33ME.

[0064] The object data subscriber 33ME detects a change in the viewpoint information of the viewer 11ME based on the object data provided by the object data server 31ME. Then, the object data subscriber 33ME negotiates with the subscription manager 51 (not shown in FIG. 5) of the live entertainment broker 41 about the display level of the objects included in the viewpoint / visibility scope based on the detected viewpoint information of the viewer 11ME.

[0065] The object data subscriber 33ME subscribes (requests) the live entertainment broker 41 to deliver only object data (including near-future predicted object data) related to target objects that are necessary for the artist 11AR and the surrounding audience 11AU to the viewer 33ME. This subscription is performed sequentially whenever the timing of a change in the object observed by the sensor 21ME, for example, whenever the state of the viewer's line of sight or the like changes, or whenever the viewer's interest or concern in the target object changes due to a change in consciousness. Of course, this change in consciousness is also detected by the sensor 21ME, or is transmitted from the sensor 21ME to the object data server 31ME via an input device (such as a remote controller) that generates sensor data.

[0066] The subscription manager 51 of the live entertainment broker 41 negotiates with the object data subscriber 33 regarding the display level of the object. In other words, it determines the filter conditions for how to filter each object, such as the artist 11AR, the nearby audience 11AJ, the non-neighborhood audience 11NJ, etc., that exists within the viewpoint / visible scope of the viewer 11ME.

[0067] The filter 52 of the live entertainment broker 41 filters the object data present within the viewpoint / visible scope of the viewer 11ME from among the object data supplied from the object data publisher 32AR of the artist 11AR, the object data publisher 32AJ of the nearby audience 11AJ, and the object data publisher 32NJ of the non-neighboring audience 11NJ, based on the determined filter conditions. The filter 52 transmits the filtered object data to the object data subscriber 33ME of the viewer 11ME.

[0068] The live entertainment broker 41 filters each object data transferred from a large number of object data publishers 32 based on the object data subscriptions for the object state information of the target object collected from the object data subscribers 33 of the viewer 11ME, the nearby audience 11AJ, and the non-neighboring audience 11NJ, and transfers it to the object data subscriber 33ME of the viewer 11ME. When filtering, the object data is weighted taking into account the difference in required quality between the nearby audience 11AJ and the non-neighboring audience 11NJ, and the object data is appropriately modified before being transferred.

[0069] For example, the direction of the face of the viewer 11ME at a given time point is captured based on the sensor data generated by the sensor 21ME. The object data subscriber 33ME can identify the viewpoint / visibility scope of the viewer 11ME from the object data generated by the object data server 31ME. The object data subscriber 33ME notifies the live entertainment broker 41 of its viewpoint / visibility scope and requests the live entertainment broker 41 to identify the objects included in the viewpoint / visibility scope. The subscription manager 51 of the live entertainment broker 41 replies with the identifier of the object included in the viewpoint / visibility scope, the attributes of the object such as whether it is a nearby audience 11AJ or a non-neighboring audience 11NJ, and negotiates the filter conditions for how to filter the object data of each object. As a result of the negotiation, the final object data subscription is determined. Various preferences of the viewer 11ME are also reflected in the object data subscription.

[0070] 6 shows information included as an object data subscription. An object data subscription includes an object data subscriber identifier ("Object Data Subscriber Identifier"), a target object reference list ("List of Target Object Reference"), and a condition ("Condition"). The object data subscriber identifier indicates the identifier of the object data subscriber 33 that registers this object data subscription. The target object reference list includes (a list of) identifiers that specify the objects to be acquired. The condition includes various filter conditions for the object data to be acquired.

[0071] Here, how much the subscription manager 51 takes into account the intentions of the viewer 11ME depends on the service operation method. In some cases, the intentions of each client device 20 are taken into account in detail, while in other cases, the intentions of the client device 20 are ignored and the contents of each object data subscription are determined from a global perspective, taking into account various factors such as the overall service operation cost and load status. The object data subscriptions are frequently updated to always maintain the latest individual viewpoint information of the viewer 11ME, the artist 11AR, and the surrounding audience 11AU.

[0072] Returning to FIG. 5, the object data subscriber 33ME receives object data of the viewpoint / visible scope transmitted from the filter 52 of the live entertainment broker 41, and supplies it to the server-side renderer .

[0073] The server-side renderer 34ME generates rendering data based on the object data of the viewpoint / visible scope supplied from the object data subscriber 33ME, and encodes the data using a predetermined compression encoding method. The server-side renderer 34ME then transmits the encoded stream data to the display 22ME of the viewer 11ME.

[0074] As will be described later, the rendering function (the function of performing GPU-dependent polygon / texture conversion and GPU transfer of objects in the viewpoint / visible scope, and outputting a 2D image to a frame buffer) may be provided on the client device 20 side, rather than on the server device 30 side. In that case, the renderer of the client device 20 generates rendering data and transfers it to the display 22ME.

[0075] The display 22ME receives the rendering data transmitted from the server-side renderer 34ME, decodes it, and then displays it. The display 22ME displays live video of the object of the artist 11AR performing at the virtual live venue 12. In addition, objects of the nearby audience 11AJ and the non-neighboring audience 11NJ are also displayed on the display 22ME according to the viewpoint / visible scope of the viewer 11ME.

[0076] In the above data processing system 1, the nearby audience 11AJ and the non-neighboring audience 11NJ each become viewers 11ME, and therefore the object data server 31 that generates object data and the server-side renderer 34 that acquires and renders the object data are configured in an n:m (n>0, m>0) relationship.

[0077] According to the data processing system 1, an object data publisher 32 and an object data subscriber 33 corresponding to the artist 11AR or the surrounding audience 11AU, and a live entertainment broker 41 are introduced. The object data publisher 32 and the object data subscriber 33 mitigate excessive traffic between the object data server 31 of the artist 11AR and the surrounding audience 11AU and the server side renderer 34 of the viewer 11ME. The live entertainment broker 41 executes a traffic filtering function. By filtering the object state information traffic between the server devices 30 that transmit and receive object data of objects in a virtual space in the live entertainment service by a combination of predetermined conditions, it is possible to suppress excessive traffic and reduce the load of the rendering process. In other words, it is possible to reduce the traffic of the object data that constitutes the virtual space.

[0078] Object state information traffic may involve the transfer of the entire original data of the object's state information at that time at regular intervals from the time a session is established until it is released, or the transfer of the entire original data first, and then, only when an update to the original data occurs, the transfer of only the update difference, or a combination of these, which is changed at specified time intervals.

[0079] In transmitting object data in a live entertainment service, the data required for the viewer 11ME are: high-quality (wide band) sensor data obtained by observing the artist 11AR, object data generated based on the sensor data, sensor data of a certain degree of quality obtained by observing the neighboring audience 11AJ in the vicinity of the viewer 11ME in the virtual space, object data generated based on the sensor data, and low-quality sensor data obtained by observing the non-neighboring audience 11NJ present within the viewpoint / visible scope in the virtual space, and object data generated based on the sensor data. However, while low quality may be acceptable for the viewer 11ME that is the focus here and the sensor data obtained by observing the non-neighboring audience 11NJ may be the neighboring audience 11AJ for other audiences, and a certain degree of quality may be required. Therefore, the sensor data itself is acquired uniformly for all audiences with a certain degree of quality, and object data is generated.

[0080] <3. Object Data Publication> 7, a process will be described up to the time when the object data of the artist 11AR and the audience 11AU is published to the live entertainment broker 41. This process is repeated and executed continuously.

[0081] First, in step S41, the sensor 21 of the client device 20 obtains sensor data obtained by observing a target subject, and transmits the sensor data to the object data server 31 of the corresponding server device 30.

[0082] In step S42, the object data server 31 receives the sensor data transmitted from the sensor 21, and generates object data based on the sensor data.

[0083] In step S43, the object data publisher 32 checks the object data at a predetermined cycle, and when all updating of the object data is completed, extracts the entire updated object data or only the updated portion of the object data, stores it in an object data publication structure together with an identifier for identifying the object, and transfers it to the live entertainment broker 41. If near-future object data is also predicted and generated, that object data is also extracted and transferred to the live entertainment broker 41.

[0084] FIG. 8 shows an example of an object data publication structure for transferring object data.

[0085] The object data publication structure has an object reference, object data, and object metadata. The object reference stores an identifier for identifying the object. The object data stores the entire updated object data or only the updated portion of the object data. The object metadata stores metadata for making it possible to refer to all attributes related to the object. It includes queries for referencing the virtual space DB 42 and queries for referencing various attributes of other objects stored in the object metadata DB 43. This object metadata is used for filtering in the filter 52 of the live entertainment broker 41.

[0086] The intervention of the object data publisher 32 can reduce excessive traffic between the object data server 31 of the artist 11AR or audience 11AU and the live entertainment broker 41.

[0087] <4. Object data transfer process before rendering> Next, a transfer process of object data up until just before rendering is performed by the client device 20 of the viewer 11ME will be described with reference to the flowchart of FIG.

[0088] First, in step S61, the sensor 21ME of the client device 20ME of the viewer 11ME obtains sensor data obtained by observing a target subject, and transmits the sensor data to the object data server 31ME of the corresponding server device 30ME.

[0089] In step S62, the object data server 31ME receives the sensor data transmitted from the sensor 21ME and generates object data based on the sensor data. The generated object data is stored in an internal storage unit. When generating the object data, the state information stored in the virtual space DB 42 of the center cloud is also referred to.

[0090] In step S63, the object data subscriber 33ME checks the object data generated by the object data server 31ME to detect a change in the viewpoint information of the viewer 11ME.

[0091] In step S64, the object data subscriber 33ME negotiates with the subscription manager 51 of the live entertainment broker 41 about the display level of the objects included in the viewpoint / visibility scope based on the viewpoint information of the detected viewer 11ME. As a result of the negotiation, the object data subscription shown in Figure 6 is determined.

[0092] Meanwhile, in step S65, the sensor 21AJ of the client device 20AJ of the nearby audience 11AJ around the viewer 11ME acquires sensor data obtained by observing the target subject, and transmits the sensor data to the object data server 31AJ of the corresponding server device 30AJ.

[0093] In step S66, the object data server 31AJ receives the sensor data transmitted from the sensor 21AJ, and generates object data based on the sensor data.

[0094] In step S67, the object data publisher 32AJ checks the object data at a predetermined cycle, and when all the object data updates have been completed, extracts the entire updated object data or only the updated part of the object data. The object data publisher 32AJ then stores the updated object data in an object data publication structure and transfers it to the live entertainment broker 41. If near-future object data is also predicted and generated, that object data is also extracted and transferred to the live entertainment broker 41.

[0095] In step S68, the filter 52 of the live entertainment broker 41 obtains from the subscription manager 51 the object data subscription determined by the negotiation.

[0096] In step S69, the filter 52 filters the object data based on the filter conditions determined by the subscription manager 51. Then, the filter 52 transmits the filtered object data to the object data subscriber 33ME of the viewer 11ME.

[0097] The object data subscriber 33ME receives viewpoint / visible scope object data sent from the filter 52 of the live entertainment broker 41 and supplies it to the server-side renderer 34.

[0098] <5. Rendering process explanation> Next, the rendering process will be described.

[0099] In the above-mentioned process, the server-side renderer 34ME was placed on the edge cloud, and the rendering process was performed on the edge cloud. That is, the live entertainment broker 41 collects and filters each object data transferred from a large number of object data publishers 32, and transfers it to the object data subscriber 33ME. The object data subscriber 33ME receives the object data of the viewpoint / visible scope, and supplies it to the server-side renderer 34 on the edge cloud. The server-side renderer 34ME generated rendering data based on the object data from the object data subscriber 33ME.

[0100] However, it is possible to provide a configuration in which a renderer is provided in the client device 20 so that rendering processing can also be performed on the client side, allowing for appropriate selection between rendering on the server side and rendering on the client side. When this configuration is adopted, as shown in Fig. 10, a client-side renderer 23 is provided in the client device 20, and an edge resource orchestrator 35 is provided in the server device 30 on the edge cloud.

[0101] The edge resource orchestrator 35 monitors the load status of the edge cloud and the traffic status between the client device 20 and the edge cloud to determine whether to perform rendering on the server side or on the client side. For example, the edge resource orchestrator 35 dynamically changes the rendering location according to the type of object data (e.g., difference between visible data, audible data, etc.), the traffic status between the client device 20 and the edge cloud, the load status of the computational resources and storage resources of the edge cloud, etc. The edge resource orchestrator 35 executes the renderer at the optimal execution location and notifies the object data subscriber 33 of the transfer destination of the object data. The object data subscriber 33 transmits the object data of the viewpoint / visible scope to the notified renderer. The client side renderer 23 receives the object data of the viewpoint / visible scope transmitted from the object data subscriber 33, generates rendering data, and supplies it to the display 22.

[0102] Rendering on the client side occurs when the traffic of object data transferred from the object data subscriber 33 on the edge cloud to the client-side renderer 23 of the client device 20 in the case of client-side rendering is extremely smaller than the traffic of the encoded stream (baseband stream if not encoded) transferred from the server-side renderer 34 on the edge cloud to the display 22 of the client device 20 in the case of server-side rendering. In other words, this is the case when the amount of transferred object data (polygon data, texture data, etc.) is extremely smaller than the amount of rendered data (baseband data).

[0103] A rendering process for dynamically changing the execution location of the renderer and rendering object data for the viewer 11ME will be described with reference to the flowchart of FIG.

[0104] In step S101, the edge resource orchestrator 35ME of the viewer 11ME determines the location of rendering depending on the load status of the computational and storage resources of the edge cloud, the traffic status between the client device 20ME and the edge cloud, etc. Then, the edge resource orchestrator 35ME executes the renderer at the determined execution location. If it is determined to render on the server side, the server-side renderer 34ME is executed, and if it is determined to render on the client side, the client-side renderer 23ME of the client device 20ME is executed.

[0105] In step S102, the edge resource orchestrator 35ME notifies the object data subscriber 33ME of the renderer (target renderer) to which the object data is to be transferred. For example, the address of the destination renderer is notified.

[0106] In step S103, the object data subscriber 33ME receives the notification of the target renderer from the edge resource orchestrator 35ME, and transmits the object data of the viewpoint / visible scope to the notified renderer.

[0107] In step S104, the started server-side or client-side renderer generates rendering data and transfers it to the display 22ME. That is, when the server-side renderer 34ME is executed, the server-side renderer 34ME receives the object data of the viewpoint / visible scope transmitted from the object data subscriber 33ME, generates rendering data, and supplies it to the display 22ME. When the client-side renderer 23ME is executed, the client-side renderer 23ME receives the object data of the viewpoint / visible scope transmitted from the object data subscriber 33ME, generates rendering data, and supplies it to the display 22ME.

[0108] The display 22ME receives and displays the rendering data in step S105.

[0109] By repeatedly executing the processes in steps S101 to S105 in FIG. 11, the location of rendering is dynamically changed depending on the traffic conditions and the like.

[0110] <6. Rendering data rounding process> Next, a rendering data rounding process for commonizing and transmitting rendering data to each of the client devices 20 of a plurality of nearby viewers 11ME will be described with reference to FIG.

[0111] For example, if two viewers 11ME are located close to each other in the virtual space and the server-side renderers 34 of the two viewers 11ME are running on the same edge server, the contents of the object data subscriptions of the two viewers 11ME will be almost identical. In other words, the viewpoint / visible scope of the two viewers 11ME will be almost identical. Therefore, the two client devices 20 share the viewpoint / visible scope, and rendering data generated by the server-side renderer 34 of one of the two viewers can be transmitted to both client devices 20.

[0112] In FIG. 12, rendering data generated by server-side renderer 34-1 corresponding to client device 20-1, which is one of client device 20-1 of viewer 11ME-1 and client device 20-2 of viewer 11ME-2, is transmitted to both display 22-1 of client device 20-1 and display 22-2 of client device 20-2.

[0113] If there is almost no difference in the viewpoint / visible scope when two viewers 11ME-1 and 11ME-2, who are close to each other, view a distant object, the conditions ("Condition" in FIG. 6) presented to the live entertainment broker 41 by object data subscription from the object data subscriber 33-1 of the viewer 11ME-1 and the object data subscriber 33-2 of the viewer 11ME-2 will be almost similar. Therefore, the live entertainment broker 41 considers the contents of those object data subscriptions to be almost identical and makes the conditions common by overwriting the contents of one condition over the other condition. Alternatively, the live entertainment broker 41 may synthesize and commonize the conditions of the two viewers 11ME-1 and 11ME-2, and generate a new condition corresponding to the viewpoint / visible scope between the two viewers.

[0114] The live entertainment broker 41 notifies the edge resource orchestrator 35 of the sharing of the rendering data. The edge resource orchestrator 35 notifies the object data subscribers 33-1 and 33-2 of the sharing of the rendering data, and causes one server-side renderer 34 to be executed on the edge cloud.

[0115] In the example of Fig. 12, the rendering data corresponding to the viewpoint / visible scope of the viewer 11ME-1 is unified and transmitted to the client devices 20 of the viewers 11ME-1 and 11ME-2. In this case, the live entertainment broker 41 transmits the object data of the viewpoint / visible scope to the object data subscriber 33-1. The object data subscriber 33-1 receives the object data and transfers it to the server-side renderer 34-1. The server-side renderer 34-1 generates rendering data based on the object data from the object data subscriber 33-1 and transmits it to the display 22-1 of the client device 20-1 and the display 22-2 of the client device 20-2.

[0116] Moreover, the edge resource orchestrator 35 notifies the object data subscriber 33-2 that no object data will arrive, and the object data subscriber 33-2 stops the process of waiting for object data.

[0117] The live entertainment broker 41 only needs to send object data to the object data subscriber 33-1, which reduces the traffic in the edge cloud. Also, only one server-side renderer 34 needs to be started, which reduces the load on the edge cloud and saves the resources of the edge cloud.

[0118] FIG. 13 is a diagram illustrating 2D images displayed on displays 22-1 and 22-2 of viewers 11ME-1 and 11ME-2 by the rendering data rounding process.

[0119] Images PIC-1 and PIC-2 are the original live images when viewers 11ME-1 and 11ME-2 each looked at a distant non-neighbor audience member 11NJ at a certain time T1 (Before). Images PIC-1 and PIC-2 are slightly different images due to the difference in viewpoint information between viewers 11ME-1 and 11ME-2.

[0120] However, when the rendering data is rounded off to be common to the conditions of viewer 11ME-1, the rendering data corresponding to the conditions of viewer 11ME-1 is transmitted to both display 22-1 of client device 20-1 and display 22-2 of client device 20-2. That is, as shown in the lower part, image PIC-1 is displayed on both displays 22-1 and 22-2.

[0121] At time T2 (After) after time T1, the object data is updated, and the original live video images when the viewer 11ME-1 and the viewer 11ME-2 look at the one distant non-neighbor audience member 11NJ are images PIC-3 and PIC-4. Images PIC-3 and PIC-4 are also slightly different images due to the difference in viewpoint information between the viewer 11ME-1 and the viewer 11ME-2.

[0122] However, due to the rendering data rounding process, the image PIC-3 that meets the conditions of the viewer 11ME-1 is displayed on both the displays 22-1 and 22-2.

[0123] In this way, the rendering data rounding process allows the same 2D image to be displayed on both displays 22-1 and 22-2.

[0124] In addition, when making conditions common by combining two conditions rather than overwriting the contents of one condition onto the other, and generating a new condition that corresponds to the viewpoint / visible scope between the two people, a 2D image is generated and displayed by combining the data of the two objects.

[0125] The rendering process when rounding the rendering data of the viewer 11ME-1 and the viewer 11ME-2 will be described with reference to the flowchart of FIG.

[0126] First, in step S141, the sensor 21-1 of the client device 20-1 of the viewer 11ME-1 obtains sensor data obtained by observing a target subject, and transmits the sensor data to the object data server 31-1 of the corresponding server device 30-1.

[0127] In step S142, the object data server 31-1 receives the sensor data transmitted from the sensor 21-1, generates object data based on the sensor data, and stores the object data in an internal storage unit.

[0128] In step S143, the object data subscriber 33-1 checks the object data generated by the object data server 31-1 and detects a change in the viewpoint information of the viewer 11ME-1.

[0129] On the other hand, in step S145, the sensor 21-2 of the client device 20-2 of the viewer 11ME-2 obtains sensor data obtained by observing the target subject, and transmits the sensor data to the object data server 31-2 of the corresponding server device 30-2.

[0130] In step S146, the object data server 31-2 receives the sensor data transmitted from the sensor 21-2, generates object data based on the sensor data, and stores the object data in an internal storage unit.

[0131] In step S147, the object data subscriber 33-2 checks the object data generated by the object data server 31-2 and detects a change in the viewpoint information of the viewer 11ME-2.

[0132] The processing of steps S141 through S143 corresponding to the viewer 11ME-1 and the processing of steps S145 through S147 corresponding to the viewer 11ME-2 can be executed in parallel.

[0133] In step S150, the object data subscriber 33-1 negotiates with the subscription manager 51 of the live entertainment broker 41 about the display level of the objects included in the viewpoint / visibility scope based on the detected viewpoint information of the viewer 11ME-1. As a result of the negotiation, an object data subscription is determined.

[0134] In step S151, the object data subscriber 33-2 negotiates with the subscription manager 51 of the live entertainment broker 41 about the display level of the objects included in the viewpoint / visibility scope based on the viewpoint information of the detected viewer 11ME-2. As a result of the negotiation, an object data subscription is determined.

[0135] The process of step S150 and the process of step S151 can be executed in parallel.

[0136] In step S152, the filter 52 of the live entertainment broker 41 obtains from the subscription manager 51 the object data subscription determined by the negotiation.

[0137] In step S153, the edge resource orchestrator 35 monitors the overall traffic situation in the edge cloud, the traffic situation between the subordinate client devices 20, and the load situation in the edge cloud ("Check Edge Cloud Resource Consumption"). If the edge resource orchestrator 35 determines that the load on the edge cloud is high as a result of the monitoring, it suggests to the filter 52 of the live entertainment broker 41 the possibility of sharing the rendering process on the edge cloud ("Recommend Renderer Aggregation").

[0138] If the filter 52 of the live entertainment broker 41 determines in step S154 that the conditions of the object data subscription based on the viewpoint information of the viewer 11ME-1 and the object data subscription based on the viewpoint information of the viewer 11ME-2 can be considered to be substantially identical in the filtering process, the filter 52 standardizes the contents of the two conditions into one. The contents of the two conditions are standardized into one by unifying the contents of one of the conditions or by combining the two conditions.

[0139] Then, in step S155, the Filter 52 notifies the edge resource orchestrator 35 of the commonization of the rendering data ("Instruct Renderer Aggregation").

[0140] In step S156, the edge resource orchestrator 35 notifies the object data subscribers 33-1 and 33-2 of the aggregation of rendering data ("Notify Aggregation"). Next, in step S157, the edge resource orchestrator 35 executes one server-side renderer 34-1 on the edge cloud that collectively processes the rendering data for the viewer 11ME-1 and the viewer 11ME-2 ("Aggregate Renderer").

[0141] When the filter 52 unifies the contents of the conditions for either the viewer 11ME-1 or the viewer 11ME-2, it filters according to the adopted condition in step S160 and generates object data of the viewpoint / visible scope. On the other hand, when the filter 52 unifies the contents of the conditions into one condition by combining two conditions, it combines the object data for each of the viewer 11ME-1 and the viewer 11ME-2 in step S160 and generates object data of the viewpoint / visible scope filtered according to the unified condition. The generated object data of the viewpoint / visible scope is transmitted to the server-side renderer 34-1 via the object data subscriber 33-1.

[0142] In step S161, the server-side renderer 34-1 acquires the transmitted object data, generates rendering data, and transmits it to each of the display 22-1 of the viewer 11ME-1 and the display 22-2 of the viewer 11ME-2.

[0143] In step S162, the display 22-1 of the viewer 11ME-1 acquires the rendering data and displays the 2D image. In step S163, the display 22-2 of the viewer 11ME-2 acquires the rendering data and displays the 2D image. The 2D images displayed on the display 22-1 and the display 22-2 are the same image.

[0144] As described above, the server-side renderer 34 is shared by a plurality of client devices 20, and rendering processing for rounding the rendering data is performed.

[0145] <7. Rounding of rendering data by data type> The above-mentioned rendering data rounding process can be performed for each type of data. For example, whether or not the rendering data rounding process is performed may be different for video data, which is visible data, and for audio data, which is audible data.

[0146] For example, as shown in Fig. 15, for audio data, which is audible data, rendering data rounding processing is performed in the same manner as the processing described with reference to Fig. 12 to Fig. 14. That is, the server-side renderer 34-3 generates common rendering data, for example, rendering data corresponding to the viewpoint / visible scope of the viewer 11ME-1, and transmits it to the display 22-1 of the viewer 11ME-1 and the display 22-2 of the viewer 11ME-2.

[0147] On the other hand, for video data, which is visible data, rendering data is generated separately by server-side renderer 34-1 corresponding to viewer 11ME-1 and server-side renderer 34-2 corresponding to viewer 11ME-2, without performing rendering data rounding processing. The rendering data generated by server-side renderer 34-1 is transmitted to display 22-1 of viewer 11ME-1, and the rendering data generated by server-side renderer 34-2 is transmitted to display 22-2 of viewer 11ME-2.

[0148] As described above, the criteria for determining whether an object data subscription is the same may differ depending on the type of data.

[0149] <8. Combination of rendering data rounding and client-side renderer> In addition, if there is sufficient resource (computational resources and storage resources) on the client side and the traffic of rendered visible data between the client device 20 and the edge cloud significantly affects congestion, the visible data, i.e., video data, may be rendered on the client side, as shown in FIG. 16.

[0150] That is, as shown in FIG. 16, the above-mentioned rendering data rounding process is performed on audio data, which is audible data, and the rendering data generated by the server-side renderer 34-3 is transmitted to the display 22-1 of viewer 11ME-1 and the display 22-2 of viewer 11ME-2.

[0151] On the other hand, for the video data, which is visible data, different object data for the viewer 11ME-1 and the viewer 11ME-2 is sent to the client-side renderers 23-1 and 23-2. Specifically, the filter 52 of the live entertainment broker 41 sends the object data for the viewer 11ME-1 to the client-side renderer 23-1 via the object data subscriber 33-1. The filter 52 of the live entertainment broker 41 sends the object data for the viewer 11ME-2 to the client-side renderer 23-2 via the object data subscriber 33-2.

[0152] In this way, it is possible to select whether to perform rendering on the client side or on the server side for each type of data, and further to perform rounding of the rendering data in the server side rendering.

[0153] As described above, in the data processing system 1, it is possible to flexibly (dynamically) change whether to execute one or both of the server-side renderer 34 and the client-side renderer 23, whether to execute consolidation processing, etc. For example, it is possible to change according to the traffic situation between the client device 20 and the edge cloud, the type of data (for example, differences between visible data, audible data, etc.), and the degree of sharing of rendered data and object data between the client devices 20.

[0154] <9. Local Live Entertainment Broker> Next, with reference to FIG. 17, a function of distributing part of the functions of the live entertainment broker 41 on the edge cloud will be described.

[0155] If all traffic is closed between object data publishers 32 and object data subscribers 33 on one edge cloud, the filter 52 functionality of the live entertainment broker 41 can be located on that edge cloud.

[0156] Specifically, as shown in Fig. 17, a local live entertainment broker 41L is placed on an edge cloud as an avatar of the live entertainment broker 41. The object data publisher 32-1 transmits the entire updated object data or only the updated portion to the local live entertainment broker 41L. The local live entertainment broker 41L transmits the object data filtered according to the object data subscription to the object data subscriber 33-2 on the same edge cloud. The object data subscriber 33-2 receives the object data of the viewpoint / visible scope of the viewer 11ME-2 and supplies it to the server-side renderer 34-2.

[0157] In this way, by arranging the local live entertainment broker 41L on the edge cloud, the object data can be folded back at the edge cloud. This makes it possible to cover delay requirements and reduce unnecessary traffic within the cloud. The local live entertainment broker 41L can be arranged as an alter ego of the live entertainment broker 41 in each edge cloud that performs the folding back of the object data, and the function of the filter 52 can be executed in a distributed manner.

[0158] The local live entertainment broker 41L can copy traffic that does not pass through the live entertainment broker 41 and transmit it to the live entertainment broker 41 on the central cloud. This allows the live entertainment broker 41 to obtain data for security monitoring and statistical processing purposes. Whether or not to transmit a copy of the traffic to the live entertainment broker 41 can be selected by setting "Partially Synchronized" or "Not Synchronized."

[0159] The object data transfer process when the local live entertainment broker 41L is installed will be described with reference to the flowchart in FIG.

[0160] First, in step S201, the sensor 21-2 of the client device 20-2 of the viewer 11ME-2 obtains sensor data obtained by observing a target subject, and transmits the sensor data to the object data server 31-2 of the corresponding server device 30-2.

[0161] In step S202, the object data server 31-2 receives the sensor data transmitted from the sensor 21-2, generates object data based on the sensor data, and stores the object data in an internal storage unit.

[0162] In step S203, the object data subscriber 33-2 checks the object data generated by the object data server 31-2 and detects a change in the viewpoint information of the viewer 11ME-2.

[0163] In step S204, the object data subscriber 33ME negotiates with the subscription manager 51 of the live entertainment broker 41 about the display level of the objects included in the viewpoint / visibility scope based on the viewpoint information of the detected viewer 11ME-2. As a result of the negotiation, an object data subscription is determined.

[0164] In step S205, the subscription manager 51 determines whether all traffic between the object data publisher 32-1 and the object data subscriber 33-2 on one edge cloud is closed. If it is determined that all traffic is closed on one edge cloud, the subscription manager 51 executes the local live entertainment broker 41L on that edge cloud in step S206 and delegates the function of the filter 52. Then, in step S207, the subscription manager 51 notifies the object data publisher 32-1 on the edge cloud of the address of the local live entertainment broker 41L.

[0165] On the other hand, in step S211, the sensor 21-1 of the client device 20-1 of the viewer 11ME-1 obtains sensor data obtained by observing a target subject, and transmits the sensor data to the object data server 31-1 of the corresponding server device 30-1.

[0166] In step S212, the object data server 31-1 receives the sensor data transmitted from the sensor 21-1, and generates object data based on the sensor data.

[0167] In step S213, the object data publisher 32-1 checks the object data at a predetermined cycle, and when all the object data updates have been completed, extracts the entire updated object data or only the updated portion and transfers it to the local live entertainment broker 41L. If near-future object data is also predicted and generated, that object data is also extracted and transferred to the local live entertainment broker 41L.

[0168] In step S214, the filter 52L of the local live entertainment broker 41L (hereinafter referred to as the local filter 52L) obtains from the subscription manager 51 the object data subscription determined by the negotiation.

[0169] In step S215, the local filter 52L performs filtering based on the filter conditions determined by the subscription manager 51. Then, the local filter 52L transmits the object data after the filtering process to the server-side renderer 34-2 via the object data subscriber 33-2 of the viewer 11ME-2.

[0170] In step S216, the server-side renderer 34-2 receives the object data of the viewpoint / visible scope transmitted from the object data subscriber 33-2, generates rendering data, and supplies it to the display 22-2.

[0171] The display 22-2 receives and displays the rendering data in step S217.

[0172] When rendering is performed on the client side, the object data is sent to the client-side renderer 23-2 instead of the server-side renderer 34-2. The display 22-2 receives the rendering data from the client-side renderer 23-2 and displays it.

[0173] According to the object data transfer process in which the local live entertainment broker 41L is deployed, the object data is returned at the edge cloud, thereby covering delay requirements and the like and reducing unnecessary traffic within the cloud.

[0174] <10. Variations in filter conditions> When the subscription manager 51 of the live entertainment broker 41 decides on an object data subscription through negotiation with the object data subscriber 33, the subscription manager 51 forms filter data 101 for each object data of the artist 11AR and the surrounding audience 11AU from the information described therein. The filter data 101 corresponds to the above-mentioned filter conditions.

[0175] In the example of FIG. 19, filter data 101AR corresponding to object data of the artist 11AR, filter data 101AJ corresponding to object data of the nearby audience 11AJ, and filter data 101NJ corresponding to object data of the non-neighboring audience 11NJ are generated.

[0176] The filter data 101 (101AR, 101AJ, 101NJ) describes filter conditions 111, and also stores weight parameters 112 generated from the filter conditions 111 and a weight parameter calculation formula 113. Depending on the contents of various conditions described in the filter conditions 111, shared state information (e.g., data on the virtual live venue 12) stored in the virtual space DB 42 or the contents of metadata for each object (object metadata) stored in the object metadata DB 43 may be referenced.

[0177] The filter condition 111 of the filter data 101 is described as a combination of the following elements (filter conditions). ·Filter criteria related to scope (visual / audible scope, etc.) The filter conditions for the visible scope may include filter conditions according to information on the line of sight or the visible range, etc. The information on the visible range may include filter conditions according to an in-scope level indicating whether the visible range is near the center or outside of the visible range. - Filter conditions related to the distance to the target object in virtual space The attention may be weighted depending on the difference between a nearby object and a distant object. The distance between objects stored in the virtual space DB 42 is also referenced. Filter conditions related to the movement (range, speed, or acceleration) of the target object There may be a filter condition such that only objects with large movements (objects whose motion vector length within the visible range or whose speed or acceleration per unit time is equal to or exceeds a predetermined threshold value, etc.) are focused on. Filter conditions related to the brightness or color of the target object, or the volume of the sound emitted by the target object There may be a filter condition such that only objects whose brightness, color, volume, etc. are above a threshold or within a predetermined range are noted. Filter conditions related to the target object's actions (gestures) or phrases uttered by the target object There may be filter conditions such as focusing only on audiences 11AU that make a specific gesture, move along a specific route, or that utter a specific keyword. · Filter conditions related to the degree of interest in the target object Possible filter conditions include interest or level of interest at the time (present and immediate future), interest or level of interest from an individual's past history, statistical interest or level of interest, level of interest (tendency) between the artist 11AR and the audience 11AU depending on the type of event, tendencies in the subjects of interest depending on the audience's 11AU's level of event proficiency (level of beginner, expert, etc.), level of interest in a part of the subject's body, belongings, or accessories, etc. Filter conditions that depend on the physical environment between the target object and the object For example, there may be filter conditions for physical environmental elements such as temperature, humidity, brightness, and dustiness that depend on the type of virtual space (stadium, hall, etc.).

[0178] The filter conditions 111 written in the filter data 101 are updated according to the object data subscription determined by the object data subscriber 33, but depending on the conditions, information stored in the virtual space DB 42 may also be referenced.

[0179] For example, a filter condition regarding the distance to a target in virtual space describes a weighting condition that depends on the distance (range) between objects in the virtual space, and the distance (range) in the virtual space is defined from the position information of the target object stored in virtual space DB42.

[0180] Furthermore, for example, filter conditions that depend on physical environmental factors such as temperature, humidity, brightness, and amount of dust between the target and the object are defined based on spatial state information that is updated from moment to moment and depends on the type of virtual space (shape, layout, material, etc.) stored in the virtual space DB42.

[0181] <11. Filtering examples> Next, filtering (weighting) will be explained using the example of filter data 101ME for the viewer 11ME.

[0182] FIG. 20 shows an example of filter data 101ME for a viewer 11ME.

[0183] Within the visible scope of the viewer 11ME, there are the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ as target objects. Here, the identifier of the object of the viewer 11ME is "Ojb-me", and the identifiers of the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ are "Obj-x", "Obj-y", and "Obj-z", respectively.

[0184] The method of weighting object data transmitted from the live entertainment broker 41 to the server side renderer 34 or client side renderer 23 of the viewer 11ME is described in filter data 101ME in Fig. 20. Filter data 101ME in Fig. 20 indicates a method of weighting object data, particularly image quality such as resolution of video object data (VisualObjectData).

[0185] In the filter condition 111, some of the filter conditions described above are described. "VisualScope" represents a filter condition related to the visible scope. "TargetDistance" represents a filter condition related to the distance from the target in the virtual space. "TargetInterest" represents a filter condition related to the degree of interest or curiosity in the target. "TargetDistance" is classified and defined into three categories: artist 11AR, nearby audience 11AJ, and non-neighborhood audience 11NJ. "TargetInterest" is defined as "Very High" for artist 11AR, "Little High" for nearby audience 11AJ, and "Very Low" for non-neighborhood audience 11NJ.

[0186] In the weighting parameters 112, weighting parameters W1, W2, and W3 corresponding to the video object data of the artist 11AR, the nearby audience 11AJ, and the non-neighboring audience 11NJ are defined. It is assumed that there is a relationship of "W1>W2>>W3" between the weighting parameters W1, W2, and W3. The weighting parameters W1, W2, and W3 are values ​​calculated from the filter conditions described in the filter conditions 111. For example, it is assumed that W1=100%, W2=50%, and W3=10%, and the magnitude relationship between W1 and W3 is reflected in the quality after the filtering process of the object data.

[0187] The weight parameter calculation formula 113 describes the formula "W1 x OBJ-x + W2 x OBJ-y + W3 x OBJ-z". This calculation formula indicates that for Obj-x with the largest weighting value W, the mesh density / texture resolution of the video object data is set to the highest level, for Obj-y with the second largest weighting value W, the mesh density / texture resolution of the video object data is set to the second highest level, and for Obj-z with the third largest weighting value W, the mesh density / texture resolution of the video object data is set to the third highest level. More specifically, the calculation formula indicates that for the video object data of artist 11AR with W1=100%, the mesh density / texture resolution is left as is, for the video object data of nearby audience 11AJ with W2=50%, the mesh density / texture resolution is reduced to half, and for the video object data of non-neighboring audience 11NJ with W3=10%, the mesh density / texture resolution is reduced to 1 / 10. The process of reducing the texture resolution includes processes of reducing the spatial resolution, the temporal resolution, and the bit depth.

[0188] The filtering process of video object data using the formula described in weight parameter calculation formula 113 in Figure 20 can be conceptually shown in relationship to each object data server 31 (31AR, 31AJ, 31NJ) of artist 11AR, nearby audience 11AJ, and non-neighboring audience 11NJ, and the server-side renderer 34ME (or client-side renderer 23ME) of viewer 11ME, as shown in Figure 21.

[0189] It is also possible that the load of the filtering process by the filter 52 of the live entertainment broker 41 becomes extremely high, that is, the load of the process of reducing the mesh density / texture resolution of the video object data becomes high, and the delay due to the filtering process is expected to become extremely large. In such a case, when each object data server 31 generates object data based on sensor data, it may generate object data with reduced resolution at multiple levels in advance. In that case, the weighting value W is also adjusted to match the level.

[0190] <12. Example of weighting parameter subdivision setting> It is also possible to set different values ​​of the weighting parameter W for different parts of the same type of object data. For example, a description will be given of an example in which the value of the weighting parameter W is set differently depending on the interest or degree of concern of the viewer 11ME, for the movement of the penlight of the target object, and the facial expression and body movement.

[0191] FIG. 22 shows another example of the filter conditions 111 of the filter data 101ME for the viewer 11ME.

[0192] The filter condition 111 in FIG. 22 describes “VisualScope” which represents a filter condition related to the visible scope, “TargetDistance” which represents a filter condition related to the distance in the virtual space to the target object, and “TargetInterest” which represents a filter condition related to the degree of interest or concern in the target object.

[0193] In “Target Distance”, audiences are classified and defined into two categories: nearby audience 11AJ and non-neighborhood audience 11NJ.

[0194] In "TargetInterest", filter conditions are set according to the difference in interest or degree of interest of the viewer 11ME, for the nearby audience 11AJ and the non-neighboring audience 11NJ, divided into the movement of the penlight and the facial expressions and body movements. Here, for the non-neighboring audience 11NJ, it is assumed that they are interested in the movement of the penlight ("Normal"), but not in the facial expressions and body movements ("Ignore"), while for the nearby audience 11AJ, they are very interested in both the movement of the penlight and the facial expressions and body movements ("Detailed").

[0195] FIG. 23 shows weighting parameters 112 and weighting parameter calculation formulas 113 of filter data 101ME for the viewer 11ME corresponding to the filter conditions 111 of FIG.

[0196] The video object data for the nearby audience 11AJ and the non-neighboring audience 11NJ is composed of video object data "VisualObjectDataForPenLight" corresponding to the penlight and video object data "VisualObjectDataForFace / Body" corresponding to the face and body. The identifier of the video object data "VisualObjectDataForPenLight" corresponding to the penlight is "VDfPL", and the identifier of the video object data "VisualObjectDataForFace / Body" corresponding to the face and body is "VDfFB".

[0197] Since the identifier of nearby audience member 11AJ is "Obj-y", the identifier of the video object data corresponding to the penlight of nearby audience member 11AJ is "Obj-y.VDfPL", and the identifier of the video object data "VisualObjectDataForFace / Body" corresponding to the face and body of nearby audience member 11AJ is "Obj-y.VDfFB".

[0198] Since the identifier of the non-neighborhood audience 11NJ is "Obj-z", the identifier of the video object data corresponding to the penlight of the non-neighborhood audience 11NJ is "Obj-z.VDfPL", and the identifier of the video object data "VisualObjectDataForFace / Body" corresponding to the face and body of the non-neighborhood audience 11NJ is "Obj-z.VDfFB".

[0199] In the weighting parameter 112, weighting parameters Wyp, Wyo, Wzp, and Wzo are defined for the image object data corresponding to the penlights and the image object data corresponding to the faces and bodies of the nearby audience 11AJ and the non-neighboring audience 11NJ.

[0200] The weighting parameter Wyp represents the weighting of the video object data corresponding to the penlights of the nearby audience 11AJ. The weighting parameter Wyo represents the weighting for the video object data corresponding to the faces and bodies of the nearby audience 11AJ. The weighting parameter Wzp represents the weighting for the video object data corresponding to the penlights of the non-neighboring audience 11NJ. The weighting parameter Wzo represents the weighting for the video object data corresponding to the faces and bodies of the non-neighboring audience 11NJ. The weighting parameters Wyp, Wyo, Wzp, and Wzo are assumed to have a relationship of "Wyp = Wyo > Wzp >> Wzo = 0", for example, Wyp = Wyo = 100%, Wzp = 50%, Wzo = 0%.

[0201] The weighting parameter calculation formula 113 describes the formula "Wyp x Obj-y.VDfPL + Wyo x Obj-y.VDfFB + Wzp x Obj-z.VDfPL + Wzo x Obj-z.VDfFB." This formula indicates that the image object data (Obj-y.VDfPL) corresponding to the penlight of the nearby audience member 11AJ, which has the largest weighting value W, and the image object data (Obj-y.VDfFB) corresponding to the face and body of the nearby audience member 11AJ, are processed with the highest mesh density / texture resolution (100%). The formula also indicates that the video object data (Obj-z.VDfPL) corresponding to the penlight of non-neighboring audience member 11NJ, which has the second weighting value W, is processed with the second mesh density / texture resolution (50%), and the video object data (Obj-z.VDfFB) corresponding to the face and body of non-neighboring audience member 11NJ, which has the third weighting value W, is processed with the third mesh density / texture resolution (0%).

[0202] The filtering process of video object data using the formula described in weight parameter calculation formula 113 in Figure 23 is conceptually shown in relation to each object data server 31 (31AJ, 31NJ) of the nearby audience 11AJ and non-neighboring audience 11NJ and the server-side renderer 34ME (or client-side renderer 23ME) of the viewer 11ME, as shown in Figure 24.

[0203] FIG. 25 is a diagram explaining the difference between the nearby audience 11AJ and the non-neighboring audience 11NJ displayed on the display 22ME of the viewer 11ME by filtering the video object data using the formula described in the weight parameter formula 113 of FIG.

[0204] In FIG. 25, the nearby audience 11AJ and the non-neighboring audience 11NJ in "Before" are the nearby audience 11AJ and the non-neighboring audience 11NJ before the updating of the object data to be filtered occurs.

[0205] Moreover, the nearby audience 11AJ and the non-neighboring audience 11NJ in the "After" state are the nearby audience 11AJ and the non-neighboring audience 11NJ after an update of the object data to be filtered has occurred.

[0206] For the nearby audience 11AJ and the non-neighboring audience 11NJ in the "Before" state, object data is generated based on the sensor data obtained by the sensor 21 and stored in the object data server 31. When the object data for the nearby audience 11AJ and the non-neighboring audience 11NJ in the "After" state is updated, the object data is extracted in the object data publisher 32 and transferred to the live entertainment broker 41.

[0207] The filter 52 of the live entertainment broker 41 executes filtering processing of the video object data according to the formula described in the weight parameter formula 113 of Fig. 23. The object data after the filtering processing is transmitted to the server-side renderer 34ME (or the client-side renderer 23ME) via the object data subscriber 33ME of the viewer 11ME, where it is rendered and displayed on the display 22ME of the viewer 11ME.

[0208] The face and body image 131FB and penlight image 131PL of the nearby audience member 11AJ shown in RenderedObjectsForMe in Fig. 25 are displayed on the display 22ME. Also, the face and body image 132FB and penlight image 132PL of the non-neighboring audience member 11NJ are displayed on the display 22ME of the viewer 11ME.

[0209] In the face and body image 131FB of nearby audience member 11AJ and the penlight image 131PL, the object data for both the movement of the penlight and the movement of the facial expression are updated and rendered in detail, i.e., they match the face and body of nearby audience member 11AJ and the penlight in the After image.

[0210] On the other hand, in the face and body image 132FB and penlight image 132PL of non-neighborhood audience member 11NJ, the movement of the penlight is rendered relatively accurately. However, compared to the movement of the penlight of nearby audience member 11AJ, the amount of information is half. In contrast, the movement of facial expressions is ignored. In other words, the face and body image 132FB of non-neighborhood audience member 11NJ remains the facial expression of non-neighborhood audience member 11NJ in the before image, not the facial expression in the after image.

[0211] <13. Local Live Entertainment Broker Filtering> The same can be done when the local live entertainment broker 41L located in the edge cloud performs filtering processing.

[0212] FIG. 26 shows the flow of filtering video object data of the nearby audience member 11AJ-a0 and transferring it to each of the viewers 11ME-a1 and 11ME-a2.

[0213] Consider a case where a neighboring audience 11AJ-a0 exists within the visibility scope of each of viewers 11ME-a1 and 11ME-a2, and the object data subscriber 33 of viewer 11ME-a1, the object data subscriber 33 of viewer 11ME-a2, and the object data publisher 32 of the neighboring audience 11AJ-a0 are located in different edge clouds.

[0214] A local live entertainment broker 41L is located on the same edge cloud as the object data server 31-a0 and object data publisher 32-a0 of the nearby audience 11AJ-a0.

[0215] The weighting parameter W-a0-a1 is the weighting of the video object data (Obj-a0) of the nearby audience 11AJ-a0 from the viewpoint of the viewer 11ME-a1. The weighting parameter W-a0-a2 is the weighting of the video object data (Obj-a0) of the nearby audience 11AJ-a0 from the viewpoint of the viewer 11ME-a2.

[0216] FIG. 27 shows an example of filter data 101ME-a1 for a viewer 11ME-a1 and filter data 101ME-a2 for a viewer 11ME-a2.

[0217] In weight parameter calculation formula 113 of filter data 101ME-a1, the calculation formula "+ W-a0-a1 x Obj-a0 + " is described.

[0218] In the weighting parameter calculation formula 113 of the filter data 101ME-a2, the calculation formula "+ W-a0-a2 x Obj-a0 + " is described.

[0219] Returning to Figure 26, the local live entertainment broker 41L performs filtering processing on the video object data (Obj-a0) of the nearby audience 11AJ-a0 corresponding to the weighting parameter W-a0-a1, and transmits it to the object data subscriber 33-a1 of the viewer 11ME-a1 on a different edge server.

[0220] The local live entertainment broker 41L performs filtering processing on the video object data (Obj-a0) of the nearby audience member 11AJ-a0 according to the weighting parameters W-a0-a2, and transmits the result to the object data subscriber 33-a2 of the viewer 11ME-a2 on a different edge server.

[0221] The object data subscriber 33-a1 of the viewer 11ME-a1 receives the object data and supplies it to the server-side renderer 34-a1. The server-side renderer 34-a1 receives the viewpoint / visible scope object data transmitted from the object data subscriber 33-a1, generates rendering data, and supplies it to the display 22-a1.

[0222] The object data subscriber 33-a2 of the viewer 11ME-a2 receives the object data and supplies it to the server-side renderer 34-a2. The server-side renderer 34-a2 receives the viewpoint / visible scope object data transmitted from the object data subscriber 33-a2, generates rendering data, and supplies it to the display 22-a2.

[0223] As described above, the local live entertainment broker 41L is distributed on the same edge cloud as the object data server 31-a0 and object data publisher 32-a0 of the nearby audience 11AJ-a0 that generate the object data to be filtered, and the filtering process is performed there.

[0224] By executing the filtering function in the vicinity of the object data publisher 32-a0 that publishes the object data to be filtered, it is possible to reduce traffic within the cloud (between the edge cloud and the center cloud). In other words, it is possible to reduce traffic within the cloud compared to the case where filtering is performed by the live entertainment broker 41 of the center cloud.

[0225] <14. Block diagram of functional components> Each functional component of the data processing system 1 will be described below.

[0226] FIG. 28 is a block diagram showing an example of a functional configuration of the client device 20. As shown in FIG.

[0227] The client device 20 may include, as functional components, a sensor 21, a display 22, and a client-side renderer 23.

[0228] The sensor 21 generates sensor data of the target subject, and transmits the generated sensor data from the data sensor 301 to the object data server 31.

[0229] When rendering processing is performed on the server side, the display 22 obtains the rendered rendering data from the server-side renderer 34 and displays it on the render data display 302. On the other hand, when rendering processing is performed on the client side, the display 22 obtains the rendering data from the client-side renderer 23 and displays it on the render data display 302.

[0230] The client-side renderer 23 receives object data of the viewpoint / visible scope transmitted from the object data subscriber 33 , generates rendering data in the object data renderer 303 , and supplies the data to the display 22 .

[0231] The rendering data transferred from the server-side renderer 34 is encoded by a predetermined compression encoding method suitable for the stream just before being transferred, and is transferred as an encoded stream. The object data sent from the object data subscriber 33 is also encoded by a predetermined compression encoding method suitable for the stream, and is transferred as an encoded stream. The same applies when sensor data or object data is exchanged between functional components.

[0232] FIG. 29 is a block diagram showing the functional configuration of the server-side renderer 34. As shown in FIG.

[0233] The server-side renderer 34 renders the object data of the viewpoint / visible scope supplied from the object data subscriber 33 in the object data renderer 311 to generate rendering data. The server-side renderer 34 encodes the generated rendering data using a predetermined compression encoding method, and transmits the encoded stream data to the display 22 of the client device 20.

[0234] FIG. 30 is a block diagram showing the functional configuration of the object data server 31. As shown in FIG.

[0235] The object data server 31 includes an object data generator 321 and an object data server 322 .

[0236] The object data generator 321 generates object data from the sensor data transmitted from the sensor 21 of the client device 20, and stores the object data in an internal storage unit, the object data server 322. When generating the object data, the object data generator 321 appropriately refers to state information (Virtual Space Data) shared by each object, which is stored in the virtual space DB 42.

[0237] The object data server 322 stores the object data from the object data generator 321. The object data server 322 also responds to a check of changes in the object data from the object data subscriber 33, which check is performed by a query and a response. For example, when the object data server 322 detects a change in the object data corresponding to a change in viewpoint information, the object data server 322 notifies the object data subscriber 33 of the change.

[0238] Furthermore, in response to a request from the object data publisher 32 to extract object data, the object data server 322 extracts the entire updated object data or only the updated portion of the object data, and supplies it to the object data publisher 32 .

[0239] FIG. 31 is a block diagram showing the functional configuration of the object data publisher 32. As shown in FIG.

[0240] The object data publisher 32 includes an object data extractor 331 and an object data publisher 332 .

[0241] The object data extractor 331 transmits an object data extraction request to the object data server 322, and acquires the entire updated object data or only the updated portion of the object data transmitted in response to the extraction request. The object data extractor 331 supplies the acquired object data to the object data publisher 332.

[0242] The object data publisher 332 stores the object data obtained from the object data extractor 331 in an object data publication structure and transfers it to the live entertainment broker 41. If a local live entertainment broker 41L is arranged, the object data in the object data publication structure is transferred to the local live entertainment broker 41L.

[0243] Furthermore, when the local live entertainment broker 41L is started, the object data publisher 332 acquires a notification of the local live entertainment broker 41L from the subscription manager 51. As the notification of the local live entertainment broker 41L, for example, the address of the local live entertainment broker 41L is notified. This allows the object data publisher 332 to recognize that the function of the filter 52 has been delegated to the local live entertainment broker 41L.

[0244] FIG. 32 is a block diagram showing the functional configuration of the object data subscriber 33. As shown in FIG.

[0245] The object data subscriber 33 includes an object data change detector 341 , an object data subscriber 342 , and an object data forwarder 343 .

[0246] The object data change detector 341 checks for changes in object data through queries and responses to the object data server 31. When the object data change detector 341 detects a change in object data, such as a change in viewpoint information of a target subject, it notifies the object data subscriber 342 of the change.

[0247] When a change in the object data is detected, the object data subscriber 342 negotiates with the subscription manager 51 of the live entertainment broker 41. As a result of the negotiation, an object data subscription is determined.

[0248] Furthermore, when rendering data sharing is performed, the object data subscriber 342 obtains a notification of rendering data sharing from the edge resource orchestrator 35. If no object data arrives at the object data subscriber 342 itself, the object data subscriber 342 stops the processing of the object data forwarder 343.

[0249] The object data forwarder 343 is notified of the renderer (target renderer) to which the object data is to be forwarded by the edge resource orchestrator 35. The target renderer is either the server-side renderer 34 or the client-side renderer 23. The target renderer is notified, for example, as the address of the destination renderer.

[0250] The object data forwarder 343 receives object data of the viewpoint / visible scope transmitted from the live entertainment broker 41 and forwards (transmits) it to the target renderer. The object data transmitted from the live entertainment broker 41 is data that has been filtered based on the determined object data subscription.

[0251] FIG. 33 is a block diagram showing the functional configuration of the edge resource orchestrator 35.

[0252] The edge resource orchestrator 35 includes a renderer life cycle manager 351 and a renderer aggregation manager 352 .

[0253] The renderer life cycle manager 351 determines whether to perform rendering on the server side or on the client side depending on the load status of the computational and storage resources of the edge cloud, the traffic status between the client device 20 and the edge cloud, etc. Then, the renderer life cycle manager 351 executes the renderer at the determined execution location. That is, when the renderer life cycle manager 351 determines to perform rendering on the server side, it executes the server side renderer 34, and when the renderer life cycle manager 351 determines to perform rendering on the client side, it executes the client side renderer 23 of the client device 20. The renderer life cycle manager 351 provides information about the executed renderer (target renderer) to the object data forwarder 343 of the object data subscriber 33.

[0254] The renderer aggregation manager 352 monitors the overall traffic status within the edge cloud, the traffic status between subordinate client devices 20, and the load status within the edge cloud. If the renderer aggregation manager 352 determines as a result of monitoring that the load of the edge cloud is high, it suggests to the filter 52 of the live entertainment broker 41 the possibility of sharing the rendering process on the edge cloud.

[0255] When the renderer aggregation manager 352 receives a notification of rendering data sharing from the live entertainment broker 41 in response to a suggestion of the possibility of rendering process sharing, the renderer aggregation manager 352 sends a notification of rendering data sharing to the object data subscriber 342 (Figure 32).

[0256] When rendering data is shared, a notification of the sharing of rendering data is also provided from the renderer aggregation manager 352 to the renderer life cycle manager 351. As a result, for example, the renderer life cycle manager 351 executes one server-side renderer 34 on the edge cloud, which collectively processes rendering data for multiple client devices 20.

[0257] FIG. 34 is a block diagram showing the functional configuration of the live entertainment broker 41. As shown in FIG.

[0258] The live entertainment broker 41 includes a subscription manager 51 and a filter 52 .

[0259] The subscription manager 51 negotiates with the object data subscriber 342 (FIG. 32) to determine an object data subscription. The subscription manager 51 supplies filter data based on the object data subscription to the filter 52. This filter data corresponds to, for example, the filter data 101ME shown in FIG. 20.

[0260] The subscription manager 51 judges whether all traffic between the object data publisher 32 and the object data subscriber 33 on one edge cloud is closed. Then, when it is judged that all traffic is closed on one edge cloud, the subscription manager 51 executes the local live entertainment broker 41L on that edge cloud and delegates the function of the filter 52. In addition, even when it is not judged that all traffic is closed on one edge cloud, the subscription manager 51 can be configured to execute the local live entertainment broker 41L on that edge cloud and delegate the function of the filter 52. The subscription manager 51 notifies the object data publisher 32 on the edge cloud of the address of the local live entertainment broker 41L.

[0261] The filter 52 obtains filter data based on the object data subscription from the subscription manager 51. The filter 52 also obtains object data stored in an object data publication structure from the object data publisher 32.

[0262] The filter 52 filters the object data based on the filter conditions 111 (FIG. 20) described in the filter data. Then, the filter 52 transmits the object data after the filtering process to the object data subscriber 33.

[0263] Furthermore, when the renderer aggregation manager 352 suggests the possibility of sharing the rendering process on the edge cloud, the filter 52 determines whether or not the contents of the object data subscriptions can be considered to be substantially identical based on the filter condition 111 of the filter data. When the contents of the object data subscriptions can be considered to be substantially identical, the filter 52 notifies (instructs) the renderer aggregation manager 352 to share the rendering data.

[0264] <15. Cloud Computing Configuration Examples> The methods and systems described herein, including the above-described data processing system 1 and its network control method, can be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or a combination or subset thereof.

[0265] FIG. 35 illustrates a block diagram of a computer capable of implementing various embodiments described herein.

[0266] The present disclosure may be embodied as a system, method, and / or computer program product that may include a computer-readable storage medium having computer-readable program instructions recorded thereon that cause one or more processors to perform aspects of the present embodiments.

[0267] A computer-readable storage medium may be a tangible device that can store instructions for use by an instruction execution device (processor). A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of such devices. More specific examples of computer-readable storage media include each of the following (and suitable combinations): a floppy disk, a hard disk, a solid state drive (SSD), a random access memory (RAM), a read only memory (ROM), an erasable and programmable read only memory (EPROM) or Flash (Flash memory), a static random access memory (SRAM), a compact disk (CD or CD-ROM), a digital versatile disc (DVD), a card-type or stick-type memory. A computer-readable storage medium as used in this disclosure is not to be construed as a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a wave guide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electric signal transmitted through a wire.

[0268] The computer readable program instructions of the present disclosure may be downloaded from a computer readable storage medium into a suitable computing or processing device or downloaded to an external computer or external storage device via a global network such as the Internet, a local area network, a wide area network, and / or a wireless network, including copper transmission lines, fiber optics, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within the computing or processing device may receive the computer readable program instructions from the network and transfer the computer readable program instructions for storage in a computer readable storage medium within the computing or processing device.

[0269] The computer readable program instructions for carrying out the processes of the present disclosure include machine language instructions and / or microcode, which may be compiled or interpreted from source code written in any combination of one or more program languages, including assembly language, Basic, Fortran, Java, Python, R, C, C++, C#, or similar programming languages. The computer readable program instructions may be executed entirely on a user's personal computer, notebook computer, tablet, or smartphone, or entirely on a remote computer or computer server, or any combination of these computing devices. The remote computer or computer server may be connected to the user's device or devices via a computer network, such as a local area network, a wide area network, or a global network (e.g., the Internet). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), may execute the computer readable program instructions using information from the computer readable program instructions to configure or customize the electronic circuitry, to implement aspects of the present disclosure.

[0270] Aspects of the present disclosure are described herein with reference to flow diagrams and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood by one skilled in the art that each block of the flow diagrams and block diagrams, and combinations of blocks in the flow diagrams and block diagrams, can be implemented by computer readable program instructions.

[0271] Computer-readable program instructions capable of implementing the systems and methods described in this disclosure are utilized in one or more processors (and / or one or more cores within a processor) of a general-purpose computer, special-purpose computer, or other programmable device to produce an apparatus. Execution of the program instructions via the processor of the computer or other programmable device creates a system for implementing the functions described in the flow diagrams and block diagrams of this disclosure. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable device, and / or other device to function in a particular manner. Thus, a computer-readable storage medium having instructions stored thereon is an article of manufacture that includes instructions implementing aspects of the functions identified in the flow diagrams and block diagrams of this disclosure.

[0272] The computer readable program instructions may also be loaded into a computer, other programmable apparatus, or other device to perform a series of operational steps on the computer, other programmable apparatus, or other device to generate a computerized process result. The program instructions, when executed on the computer, other programmable apparatus, or other device, provide the functions identified in the flow diagrams and block diagrams of this disclosure.

[0273] Fig. 35 is a functional block diagram of a network system 800 in which one or more computers, servers, etc. are connected via a network. Note that the hardware and software environment shown in the embodiment of Fig. 35 is shown as an example providing a platform for realizing the software and / or method according to the present disclosure.

[0274] As shown in Figure 35, network system 800 may include, but is not limited to, a computer 805, a network 810, a remote computer 815, a web server 820, a cloud storage server 825, and a computer server 830. In one embodiment, multiple instances of one or more of the functional blocks shown in Figure 35 are used.

[0275] 35 illustrates a more detailed configuration of the computer 805. Note that the functional blocks illustrated in the computer 805 are illustrated to establish exemplary functions, and are not all shown. Also, detailed configurations of the remote computer 815, the web server 820, the cloud storage server 825, and the computer server 830 are not illustrated, but they may include configurations similar to the functional blocks illustrated for the computer 805.

[0276] The computer 805 may be a personal computer (PC), a desktop computer, a laptop computer, a tablet computer, a netbook computer, a personal digital assistant (PDA), a smartphone, or any other programmable electronic device capable of communicating with other devices on the network 810.

[0277] The computer 805 is then configured with a processor 835, a bus 837, memory 840, non-volatile storage 845, a network interface 850, a peripherals interface 855, and a display interface 865. Each of these functions may be implemented in some embodiments as individual electronic subsystems (integrated circuit chips or combinations of chips and related devices), while in other embodiments several functions may be combined and implemented as a single chip (System on Chip or SoC).

[0278] Processor 835 may be one or more single- or multi-chip microprocessors, such as those designed and / or manufactured by Intel Corporation, Advanced Micro Devices, Inc. (AMD), Arm Holdings (Arm), Apple Computer, etc. Examples of microprocessors include Celeron, Pentium, Core i3, Core i5, and Core i7 from Intel Corporation, Opteron, Phenom, Athlon, Turion, and Ryzen from AMD, and Cortex-A, Cortex-R, and Cortex-M from Arm.

[0279] Bus 837 may employ any proprietary or industry standard high speed parallel or serial peripheral interconnect bus, such as, for example, ISA, PCI, PCI Express (PCI-e), AGP, etc.

[0280] The memory 840 and the non-volatile storage 845 are computer readable storage media. The memory 840 may employ any suitable volatile storage device, such as a dynamic random access memory (DRAM) or a static RAM (SRAM). The non-volatile storage 845 may employ one or more of a flexible disk, a hard disk, a solid state drive (SSD), a read only memory (ROM), an erasable and programmable read only memory (EPROM), a flash memory, a compact disk (CD or CD-ROM), a digital versatile disk (DVD), a card type memory, or a stick type memory.

[0281] Additionally, program 848 is a collection of machine-readable instructions and / or data that is stored in non-volatile storage 845 and is used to create, manage, and control certain software functions as detailed herein and illustrated in the drawings. Note that in configurations where memory 840 is significantly faster than non-volatile storage 845, program 848 may be transferred from non-volatile storage 845 to memory 840 before being executed by processor 835.

[0282] Through network interface 850, computer 805 can communicate and interact with other computers over network 810. Network 810 can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of LANs and WANs, including wired, wireless, or fiber optic connections. In general, network 810 consists of any combination of connections and protocols that support communication between two or more computers and associated devices.

[0283] The peripheral interface 855 can input and output data to and from other devices that may be locally connected to the computer 805. For example, the peripheral interface 855 provides a connection to an external device 860, which may be a keyboard, a mouse, a keypad, a touch screen, and / or other suitable input devices. The external device 860 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data implementing embodiments of the present disclosure, such as the program 848, may be stored on such portable computer-readable storage media. In such an embodiment, the software may be loaded onto the non-volatile storage 845, or alternatively, may be loaded directly onto the memory 840 via the peripheral interface 855. The peripheral interface 855 may use industry standards, such as RS-232 or Universal Serial Bus (USB), to connect to the external device 860.

[0284] Display interface 865 may connect computer 805 to a display 870, which may be used to present a command line or a graphical user interface to a user of computer 805. Display interface 865 may connect to display 870 using one or more of an industry standard or proprietary connection, such as a Video Graphics Array (VGA), Digital Visual Interface (DVI), DisplayPort, or High-Definition Multimedia Interface (HDMI).

[0285] As discussed above, the network interface 850 provides for communication with other computers, storage systems, or devices external to the computer 805. The software programs and data described herein may be downloaded to the non-volatile storage 845 from, for example, the remote computer 815, the web server 820, the cloud storage server 825, and the computer server 830 via the network interface 850 and the network 810. Additionally, the systems and methods of the present disclosure may be performed by one or more computers connected to the computer 805 via the network interface 850 and the network 810. For example, in one embodiment, the systems and methods of the present disclosure may be performed by the remote computer 815, the computer server 830, or a combination of multiple interconnected computers on the network 810.

[0286] Data, datasets, and / or databases employed in embodiments of the systems and methods of the present disclosure may be downloaded from and stored on remote computers 815, web servers 820, cloud storage servers 825, and computer servers 830.

[0287] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0288] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0289] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0290] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0291] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0292] The present technology can have the following configurations. (1) A broker is provided for filtering first object data, which is object data of a first object displayed in a virtual space, and second object data, which is object data of a second object, based on a determined filter condition, and transmitting the filtered object data to a rendering server. Server device. (2) the first object data is object data generated based on sensor data acquired by a client device at a first location where a first subject is present; The second object data is object data generated based on sensor data acquired by a client device at a second location different from the first location where a second subject is present. The server device according to (1) above. (3) The broker: a determination unit that negotiates with a server corresponding to a client device of a viewer who views the first object and the second object and determines the filter condition; a filter for filtering the object data based on the determined filter condition; Equipped The server device according to (1) or (2). (4) The determining unit and the filter are located on a cloud different from the rendering server. The server device according to (3) above. (5) the determination unit is disposed on a cloud different from the rendering server, The filter is located on the same cloud as the rendering server. The server device according to (3) above. (6) The filter condition includes a filter condition regarding a scope of viewers who view the first object and the second object. The server device according to any one of (1) to (5). (7) The filter condition includes a filter condition regarding a distance to a target object in the virtual space. The server device according to any one of (1) to (6). (8) The filter condition includes a filter condition regarding a movement of the target object. The server device according to any one of (1) to (7). (9) The filter conditions include a filter condition related to the brightness or color of the target object, or a filter condition related to the volume of the sound emitted by the target object. The server device according to any one of (1) to (8). (10) The filter conditions include filter conditions related to the action of the target object or phrases uttered by the target object. The server device according to any one of (1) to (9). (11) The filter condition includes a filter condition regarding a degree of interest in the target object. The server device according to any one of (1) to (10). (12) The filter conditions include a filter condition that depends on a physical environment element between the target object and the target object. The server device according to any one of (1) to (11). (13) The object data server that generates the object data and the rendering server are configured in an n:m relationship. The server device according to any one of (1) to (12). (14) Based on the load status of the cloud or the traffic status between the cloud and a client device that displays an image based on the object data, a selection is made as to whether to place the rendering server in the cloud or in the client device. The server device according to any one of (1) to (13). (15) The location of the rendering server is selected between the cloud and the client device for each type of object data. The server device according to (14) above. (16) The rendering server is shared by a plurality of client devices that display images based on the object data. The server device according to any one of (1) to (15). (17) Whether or not the rendering server is shared among the plurality of client devices is selected for each type of object data. The server device according to (16) above. (18) A server device, Filtering each of the first object data, which is object data of the first object displayed in the virtual space, and the second object data, which is object data of the second object, based on the determined filter conditions, and transmitting the filtered object data to a rendering server. Network control methods. [Explanation of symbols]

[0293] 1: Data processing system, 11AJ: Nearby audience, 11AR: Artist, 11AU: Audience, 11ME: Viewer, 11NJ: Non-neighboring audience, 12: Live venue, 20: Client device, 21: Sensor, 22: Display, 23: Client side renderer, 30: Server device, 31: Object data server, 32: Object data publisher, 33: Object data subscriber, 34: Server side renderer, 35: Edge resource orchestrator, 40: Server device, 41: Live entertainment broker, 41L: Local live entertainment broker, 42: Virtual space DB, 43: Object metadata DB, 51: Subscription manager, 52: Filter, 101: Filter data, 111: Filter condition, 112: Weight parameter, 113: Weight parameter calculation formula, 301: Data sensor, 3 02: Render data display, 303: Object data renderer, 311: Object data renderer, 321: Object data generator, 322: Object data server, 331: Object data extractor, 332: Object data publisher, 341: Object data change detector, 342: Object data subscriber, 343: Object data forwarder, 351: Renderer lifecycle manager, 352: Renderer aggregation manager, 800: Network system, 805: Computer, 810: Network, 815: Remote computer, 820: Web server, 825: Cloud storage server, 830: Computer server, 835: Processor, 840: Memory, 845: Non-volatile storage, 848: Program, 850: Network interface, 870: Display

Claims

1. a broker that filters first object data, which is object data of a first object, and second object data, which is object data of a second object, to be displayed in a virtual space, based on a determined filter condition, and transmits the object data after the filtering process to a rendering server; Based on the load status of the cloud or the traffic status between the cloud and a client device that displays an image based on the object data, a selection is made as to whether the rendering server should be located in the cloud or in the client device for each type of object data. Server device.

2. A broker that filters first object data, which is object data of a first object, and second object data, which is object data of a second object, displayed in a virtual space, based on determined filter conditions, and transmits the object data after the filtering process to a rendering server; When the rendering server is shared among a plurality of client devices that display images based on the object data, whether or not to share the object data among the plurality of client devices is selected for each type of the object data. Server device.

3. the first object data is object data generated based on sensor data acquired by a client device at a first location where a first subject is present; The second object data is object data generated based on sensor data acquired by a client device at a second location different from the first location where a second subject is present.

3. The server device according to claim 1 or 2.

4. The broker: a determination unit that negotiates with a server corresponding to a client device of a viewer who views the first object and the second object and determines the filter condition; a filter for filtering the object data based on the determined filter condition; Equipped 3. The server device according to claim 1 or 2.

5. The determining unit and the filter are located on a cloud different from the rendering server. The server device according to claim 4.

6. the determination unit is disposed on a cloud different from the rendering server, The filter is located on the same cloud as the rendering server. The server device according to claim 4.

7. The filter condition includes a filter condition regarding a scope of viewers who view the first object and the second object.

3. The server device according to claim 1 or 2.

8. The filter condition includes a filter condition regarding a distance to a target object in the virtual space.

3. The server device according to claim 1 or 2.

9. The filter condition includes a filter condition regarding a movement of the target object.

3. The server device according to claim 1 or 2.

10. The filter conditions include a filter condition related to the brightness or color of the target object, or a filter condition related to the volume of the sound emitted by the target object.

3. The server device according to claim 1 or 2.

11. The filter conditions include filter conditions related to the action of the target object or phrases uttered by the target object.

3. The server device according to claim 1 or 2.

12. The filter condition includes a filter condition regarding a degree of interest in the target object.

3. The server device according to claim 1 or 2.

13. The filter conditions include a filter condition that depends on a physical environment element between the target object and the target object.

3. The server device according to claim 1 or 2.

14. The object data server that generates the object data and the rendering server are configured in an n:m relationship.

3. The server device according to claim 1 or 2.

15. The server device filtering, based on the determined filter conditions, each of the first object data, which is object data of the first object, and the second object data, which is object data of the second object, displayed in the virtual space; and transmitting the filtered object data to a rendering server; Based on the load status of the cloud or the traffic status between the cloud and a client device that displays an image based on the object data, a selection is made as to whether the rendering server should be located in the cloud or in the client device for each type of object data. Network control methods.

16. A server device, filtering, based on the determined filter conditions, each of the first object data, which is object data of the first object, and the second object data, which is object data of the second object, displayed in the virtual space; and transmitting the filtered object data to a rendering server; When the rendering server is shared among a plurality of client devices that display images based on the object data, whether or not to share the object data among the plurality of client devices is selected for each type of the object data. Network control methods.

Images