Rendering control device

Abstract

In the present invention, an acquisition unit acquires rendering performance information which represents rendering performance of virtual object data, in relation to a terminal device that displays one or more virtual objects disposed in one or more layers on the basis of the virtual object data. On the basis of the rendering performance information, a determination unit determines each of the one or more layers to be a first layer for which rendering is performed with a terminal device or a second layer for which rendering is performed with a rendering server that is connected to the terminal device via a network.

Description

Technical Field

[0001] The present invention relates to a rendering control device.

Background Art

[0002] In recent years, XR (Extended Reality) technology has been spreading. XR is a general term for AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality), etc. For example, Patent Document 1 below discloses an extended reality (XR) learning system that gives a user hands-on visual guidance from an instructor or an expert using an XR device. First, while an expert's hand is performing a manual task, a series of images of the expert's hand are recorded using a camera. Based on the series of images of the expert's hand, a representation of the expert's hand is generated using a deep learning network implemented by a processor operably coupled to the camera. Based on the representation of the expert's hand, a model of the expert's hand is generated. While the user is performing a manual task, a model of the expert's hand placed on the user's hand is rendered so as to guide the user when performing the manual task using the XR device. The rendering of the model of the expert's hand may be distributed between a first processor operably disposed in a server and a second processor operably disposed in the XR device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] High-quality rendering of XR content requires significant processing resources. For example, equipping XR devices with high-performance processing units makes them expensive, placing a heavy burden on users. While high-quality rendering is possible by constantly using rendering servers located on the network, using these servers incurs usage fees based on the amount of data, also increasing the burden on users.

[0005] The objective of this invention is to enable the provision of high-quality rendered content while keeping costs down. [Means for solving the problem]

[0006] A rendering control device according to one aspect of the present invention relates to a terminal device that displays one or more virtual objects arranged on one or more layers based on virtual object data, and comprises: an acquisition unit that acquires rendering performance information indicating the rendering performance of the virtual object data; and a determination unit that determines, based on the rendering performance information, each of the one or more layers to be either a first layer rendered by the terminal device or a second layer rendered by a rendering server connected to the terminal device via a network. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to provide high-quality rendered content while keeping costs down. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of System 1 according to the embodiment. [Figure 2] This is a block diagram showing the configuration of terminal device 10. [Figure 3] This is a block diagram showing the configuration of data server 20. [Figure 4]This is a block diagram showing the configuration of the rendering server 30. [Figure 5] This figure shows an example of a virtual object V displayed based on virtual object data VD. [Figure 6A] This figure shows an example of a layer L displayed in the user's field of view S1. [Figure 6B] This figure shows an example of a layer L displayed in the user's field of view S1. [Figure 6C] This figure shows an example of a layer L displayed in the user's field of view S1. [Figure 7] This is a flowchart illustrating the operation of System 1. [Figure 8] This is a block diagram showing the configuration of the data server 20 according to the first modified example. [Modes for carrying out the invention]

[0009] A. Embodiment A-1. System Configuration Figure 1 is a block diagram showing the configuration of System 1 according to an embodiment. System 1 includes a terminal device 10, a data server 20, and a rendering server 30. The data server 20 is an example of a rendering control device. The terminal device 10, the data server 20, and the rendering server 30 are connected via a network N. Although only one terminal device 10 is shown in Figure 1, System 1 can be equipped with any number of terminal devices 10.

[0010] In this embodiment, System 1 is a system that presents various information to a user holding a terminal device 10 using VR technology. Here, VR technology is a technology that allows users to feel as if they are actually in a space by projecting 360° images that cover the user's field of vision using special goggles or the like. In this embodiment, the terminal device 10 displays to the user one or more virtual objects V (see Figure 5, etc.) arranged on one or more layers L based on virtual object data VD (see Figure 3). In VR technology, virtual objects V are generally three-dimensional images (images captured from real space or three-dimensional CG (Computer Graphics) models), but may also be two-dimensional still images or videos. Furthermore, virtual objects V may also include text. In addition, other types of information, such as audio, may be output along with the virtual objects V.

[0011] The terminal device 10 is, for example, a head-mounted display. Alternatively, the terminal device 10 may be a system in which an information processing terminal such as a smartphone, tablet, personal computer, or game console is connected to goggles. In this embodiment, the terminal device 10 is assumed to be a head-mounted display.

[0012] The data server 20 stores virtual object data VD (see Figure 3) for outputting virtual object V at the terminal device 10. In this embodiment, the data server 20 stores multiple virtual object data VDs. Each virtual object data VD includes one or more layer data LDs. One layer data LD corresponds to one layer L. One or more virtual objects V are arranged in one layer L.

[0013] For example, the virtual object data VD includes layer data LD, such as layer data LD corresponding to a background layer and layer data LD corresponding to a foreground layer, as the layer data LD. By superimposing and displaying the foreground layer on the background layer, the virtual object V arranged on the foreground layer and the virtual object V arranged on the background layer are superimposed and displayed.

[0014] Each layer L can be edited individually. Also, the virtual object data VD can be edited by replacing, adding, or deleting the layer L in the virtual object data VD. The layer data LD may include not only images but also audio data. By separating the layer L where the audio data is arranged from the layer L where the image data is arranged, the replacement or editing of the audio can be facilitated.

[0015] The rendering server 30 renders the layer data LD. Hereinafter, "rendering the layer data LD" means rendering all the layer data LD included in the virtual object data VD and rendering a part of the layer data LD included in the virtual object data VD.

[0016] In the present embodiment, the rendering server 30 renders the layer data LD received from the data server 20. Also, the rendering server 30 transmits the rendering data, which is the layer data LD after rendering, to the terminal device 10. The rendering server 30 is installed, for example, by a provider of a cloud rendering service. For example, the provider of the rendering service collects a fee from the user of the terminal device 10 according to the amount of rendered data.

[0017] A-2. Terminal Device 10 FIG. 2 is a block diagram showing the configuration of the terminal device 10. The terminal device 10 includes a projection device 101, a speaker 102, a communication device 103, a sensor 104, an input device 105, a storage device 107, a processing device 108, and a bus 109. Each configuration shown in FIG. 2 is stored in a frame, for example. The projection device 101, the speaker 102, the communication device 103, the sensor 104, the input device 105, the storage device 107, and the processing device 108 are interconnected by a bus 109 for communicating information. The bus  109 may be configured using a single bus or may be configured using different buses for each element such as a device.

[0018] The projection device 101 includes a display panel and an optical member. Two display panels are provided corresponding to the left and right eyes of the user. Alternatively, the display panel has two divided display areas corresponding to the left and right eyes of the user. The images projected onto the two display panels are shifted so that a parallax occurs between the left and right eyes. This parallax enables stereoscopic vision. The projection device 101 displays an image based on rendering data on the display panel based on the control of the processing device 108. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical member guides the light emitted from the display panel to the left and right lenses.

[0019] The speaker 102 outputs sound. The sound output by the speaker 102 is, for example, the sound included in VR content. The speaker 102 is controlled by the processing device 108.

[0020] The communication device 103 communicates with the data server 20 and the rendering server 30 using wireless communication or wired communication. In the present embodiment, the communication device 103 includes a communication interface connectable to the network N and communicates with the data server 20 and the rendering server 30 via the network N.

[0021] Sensor 104 detects the movement of the user wearing the terminal device 10. Sensor 104 may be, for example, an accelerometer, an angular acceleration sensor, an inertial measurement unit (IMU), or a geomagnetic sensor. Alternatively, sensor 104 may be a device that detects the position of the terminal device 10 using GPS (Global Positioning System) or VPS (Visual Positioning Service / System).

[0022] The input device 105 receives input from the user. In this embodiment, the input device 105 is used for selecting VR content to be used by the user, etc. The input device 105 may be, for example, a switch or remote controller provided on the head-mounted display.

[0023] The storage device 107 is a recording medium that can be read by the processing unit 108. The storage device 107 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory). The storage device 107 stores program PG1. Program PG1 is a program for operating the terminal device 10.

[0024] The processing unit 108 includes one or more CPUs (Central Processing Units). One or more CPUs are examples of one or more processors. Each processor and CPU is an example of a computer.

[0025] The processing unit 108 reads program PG1 from the storage device 107. By executing program PG1, the processing unit 108 functions as a communication control unit 111, a rendering unit 112, and a display control unit 113. At least some of the functions of the communication control unit 111, the rendering unit 112, and the display control unit 113 may be configured by circuits such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array).

[0026] The communication control unit 111 controls communication between the data server 20 and the rendering server 30 and the terminal device 10. For example, the communication control unit 111 transmits the selection result of VR content input to the input device 105 to the data server 20. The communication control unit 111 also transmits the detection value of the sensor 104 to the data server 20. The communication control unit 111 also receives virtual object data VD from the data server 20. The communication control unit 111 also receives rendering data from the rendering server 30.

[0027] The rendering unit 112 generates rendering data by rendering the virtual object data VD received from the data server 20. As will be described later, the rendering unit 112 may render only some of the layer data LD from the virtual object data VD received from the data server 20, render all of the layer data LD, or not render any of the layer data LD.

[0028] The rendering unit 112, for example, if the virtual object V is a 3D computer graphics (CG) object, calculates and visualizes information about the object and its shape given as numerical data. More specifically, the rendering unit 112 combines various data representing the viewpoint position, the type and position of the light source, the shape of the object and the coordinates of its vertices, the drawing color, or the texture of the surface, and creates an image by performing processes such as removing hidden surfaces and adding shading.

[0029] The display control unit 113 controls the projection device 101 based on the rendering data rendered by the rendering unit 112 or the rendering data received by the communication device 103 from the rendering server 30, causing the virtual object V to be displayed on the display panel.

[0030] A-3. Data Server 20 Figure 3 is a block diagram showing the configuration of the data server 20. The data server 20 includes a communication device 203, a storage device 205, a processing unit 206, and a bus 207. The communication device 203, the storage device 205, and the processing unit 206 are interconnected by a bus 207 for communicating information. The bus 207 may be configured using a single bus, or different buses may be configured for each device.

[0031] The communication device 203 communicates with the terminal device 10 and the rendering server 30 using wireless or wired communication. In this embodiment, the communication device 203 has a communication interface that can connect to the network N and communicates with the terminal device 10 and the rendering server 30 via the network N.

[0032] The storage device 205 is a recording medium that can be read by the processing unit 206. The storage device 205 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM, EPROM, and EEPROM. Volatile memory is, for example, RAM. The storage device 205 stores program PG2 and virtual object data VD (e.g., VD1, VD2).

[0033] Program PG2 is a program for operating the data server 20. Virtual object data VD is data for outputting virtual object V on the terminal device 10. For example, in Figure 3, multiple virtual object data VDs, including virtual object data VD1 and VD2, are stored in the storage device 205. As described above, virtual object data VD includes one or more layer data LDs. For example, virtual object data VD1 includes one layer data LD. Also, virtual object data VD2 includes two or more layer data LDs. In addition to the layer data LDs, virtual object data VD also includes information such as the order of layer data LDs (stacking order).

[0034] Furthermore, at least some of the layer data LD of the virtual object data VD may be stored on a device different from the data server 20. For example, if some of the layers L of the virtual object data VD are advertisements, the data server 20 may obtain the layer data LD of the layer L assigned to the advertisements from an advertisement server (not shown). The layer data LD of the layer L assigned to the advertisements may be sent directly from the advertisement server to the terminal device 10.

[0035] The processing unit 206 includes one or more CPUs. One or more CPUs are examples of one or more processors. Each processor and CPU is an example of a computer.

[0036] The processing unit 206 reads program PG2 from the storage device 205. By executing program PG2, the processing unit 206 functions as a selection unit 211, an acquisition unit 212, a determination unit 213, and a transmission control unit 214. At least some of the functions of the selection unit 211, the acquisition unit 212, the determination unit 213, and the transmission control unit 214 may be configured by circuits such as a DSP, ASIC, PLD, and FPGA. Details of the selection unit 211, the acquisition unit 212, the determination unit 213, and the transmission control unit 214 will be described later.

[0037] A-4. Rendering Server 30 Figure 4 is a block diagram showing the configuration of the rendering server 30. The rendering server 30 includes a communication device 303, a storage device 305, a processing unit 306, and a bus 307. The communication device 303, the storage device 305, and the processing unit 306 are interconnected by a bus 307 for communicating information. The bus 307 may be configured using a single bus, or different buses may be configured for each device.

[0038] The communication device 303 communicates with the terminal device 10 and the data server 20 using wireless or wired communication. In this embodiment, the communication device 303 has a communication interface that can be connected to the network N and communicates with the terminal device 10 and the data server 20 via the network N.

[0039] The storage device 305 is a recording medium that can be read by the processing unit 306. The storage device 305 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM, EPROM, and EEPROM. Volatile memory is, for example, RAM. The storage device 305 stores program PG3. Program PG3 is a program for operating the rendering server 30.

[0040] The processing unit 306 includes one or more CPUs. One or more CPUs are examples of one or more processors. Each processor and CPU is an example of a computer.

[0041] The processing unit 306 reads the program PG3 from the storage device 305. By executing the program PG3, the processing unit 306 functions as the communication control unit 311 and the rendering unit 312. At least part of the functions of the communication control unit 311 and the rendering unit 312 may be configured by circuits such as a DSP, ASIC, PLD, and FPGA.

[0042] The communication control unit 311 controls communication between the data server 20 and the terminal device 10 and the rendering server 30. For example, the communication control unit 311 receives virtual object data VD or layer data LD from the data server 20. The communication control unit 311 also transmits the rendering data rendered by the rendering unit 312 to the terminal device 10.

[0043] The rendering unit 312 generates rendering data by rendering the virtual object data VD received from the data server 20. As will be described later, the rendering unit 312 may either render only some of the layer data LD from the virtual object data VD received from the data server 20, or it may render all of the layer data LD.

[0044] A-5. Details of Processing Unit 206 Next, we will describe the selection unit 211, acquisition unit 212, determination unit 213, and transmission control unit 214, which are realized when the processing unit 206 executes the program PG2. Before that, we will describe the details of the virtual object data VD stored in the storage device 205.

[0045] Figure 5 shows an example of a virtual object V displayed based on virtual object data VD. In this embodiment, the user experiences a virtual tour of a tourist destination using the terminal device 10. In Figure 5, the user's field of view S1 displays a cityscape V1, a guide V2, and a map V3. The virtual object V in the user's field of view S1 is displayed when the virtual object data VD is rendered by the rendering unit 112 or 312, and the projection device 101 of the terminal device 10 is driven based on the rendering data.

[0046] The cityscape V1 is, for example, a 3D computer graphics model. The cityscape V1 may also be an image captured from real space. The display of the cityscape V1 changes in conjunction with the user's movements. For example, when the user turns their head to the right, the area to the right of the cityscape V1 shown in Figure 5 is displayed.

[0047] Guide V2 is, for example, a 3D CG model of a person. Guide V2 stands beside the user and explains the history, culture, and points of interest of the tourist destination. Guide V2's mouth movements and facial expressions change in conjunction with the user's speech. Guide V2 also moves through the virtual space as the explanation progresses. Furthermore, Guide V2's position within the user's field of view S1 changes in conjunction with the user's movements. For example, if the user turns their head to the right, the position of Guide V2 within the user's field of view S1 shifts to the left. AI (Artificial Intelligence) technology may be used to enable interaction between Guide V2 and the user.

[0048] Map V3 is, for example, a two-dimensional image. Map V3 includes a planar map of the area shown in Streetscape V1 and a marker (a star in Figure 5) indicating the user's current location. The marker indicating the user's current location in Map V3 changes its position on the planar map in conjunction with the user's movement.

[0049] Here, the streetscape V1, guide V2, and map V3 are each placed on different layers L (L1 to L3). Figures 6A to 6C show examples of layers L displayed in the user's field of view S1. Layer L1, shown in Figure 6A, contains a 3D CG model of the streetscape V1. Layer L2, shown in Figure 6B, contains a 3D CG model of the guide V2. Layer L3, shown in Figure 6C, contains a 2D image of the map V3. In this embodiment, the virtual objects V placed on layers L are assumed to be non-transparent. The user's field of view S1 shown in Figure 5 is an image in which layers L are stacked in the order of layer L1, layer L2, and layer L3 from bottom to top. As mentioned above, audio data may also be placed on layers L. For example, the virtual object data VD may include, in addition to layers L1 to L3, a layer L containing the spoken audio of guide V2 and a layer L containing the sounds of daily life in the streetscape V1.

[0050] Let the layer data corresponding to layer L1 be layer data LD1, the layer data corresponding to layer L2 be layer data LD2, and the layer data corresponding to layer L3 be layer data LD3. Comparing layer data LD1 to LD3, the rendering processing load is assumed to be in the order of layer data LD1, layer data LD2, and layer data LD3. Furthermore, the relative data sizes of layer data LD1 to LD3 are assumed to be 3000 for layer data LD1, 1000 for layer data LD2, and 200 for layer data LD3.

[0051] The selection unit 211 shown in Figure 3 selects the virtual object data VD to be transmitted to the terminal device 10 from among a plurality of virtual object data VD stored in the storage device 205. The selection unit 211 receives, for example, a city name for a virtual tour from the user and identifies the virtual object data VD corresponding to that city name. The selection unit 211 also identifies the portion of the virtual object data VD to be rendered. The selection unit 211 identifies the portion of the virtual object data VD to be rendered based, for example, the user's movement detected by the sensor 104. The selection unit 211 detects the user's movement by acquiring the detection value from the sensor 104 of the terminal device 10 via, for example, the network N and analyzing the detection value. For example, if the user moves forward while looking at the field of view S1 shown in Figure 5, rendering of the virtual object data VD corresponding to the area in front of the cityscape V1 in Figure 5 becomes necessary.

[0052] The acquisition unit 212 acquires rendering performance information indicating the rendering performance of virtual object data VD with respect to the terminal device 10. In this embodiment, the acquisition unit 212 acquires at least one of the following as rendering performance information: specification information indicating the specifications of the terminal device 10, or processing load information indicating the processing load of the terminal device 10.

[0053] Specification information refers to information that indicates the specifications of the terminal device 10. For example, the clock speed, number of cores, or memory capacity of the storage device 107 of the terminal device 10 can be used as specification information. When using clock speed, number of cores, or memory capacity as specification information, the larger these values ​​are, the higher the rendering performance of the terminal device 10. In general, if the model number of an information processing device such as the terminal device 10 can be identified, the specifications can also be identified, so the acquisition unit 212 may acquire the model number of the terminal device 10 as specification information. When using specification information as rendering performance information, the acquisition unit 212 requests the terminal device 10 to send the specification information, for example, when communication between the terminal device 10 and the data server 20 begins.

[0054] Furthermore, the processing load information reflects the state of the terminal device 10 at any given time and indicates the level of processing load on the terminal device 10. For example, frame rate or resource usage can be used as processing load information. When using frame rate as processing load information, a lower frame rate indicates a greater processing load on the terminal device 10 and a decrease in rendering performance. When using resource usage as processing load information, a higher resource usage indicates a greater processing load on the terminal device 10 and a decrease in rendering performance. When processing load information is used as rendering performance information, the acquisition unit 212 continuously requests the terminal device 10 to transmit processing load information, for example, while communication is taking place between the terminal device 10 and the data server 20.

[0055] Based on rendering performance information, the decision unit 213 determines whether each of the one or more layers L will be a first layer rendered by the rendering unit 112 of the terminal device 10, or a second layer rendered by the rendering unit 312 of the rendering server 30 connected to the terminal device 10 via the network N.

[0056] As mentioned above, using the rendering server 30 is generally costly. Therefore, it is generally preferable to render all layers L on the terminal device 10. In other words, it is generally preferable that all layers L be designated as the first layer. On the other hand, for example, when rendering virtual object data VD that includes high-resolution layers L that are difficult to render with the specifications of the terminal device 10, or when the processing load on the terminal device 10 is large and rendering all layers L may cause delays, etc. In this case, the decision unit 213 decides to render at least some of the layers L on the rendering server 30. In other words, when it is difficult to render all layers L on the terminal device 10, at least some of the layers L are designated as the second layer.

[0057] When the acquisition unit 212 acquires specification information as rendering performance information, the determination unit 213 determines, based on the specification information, that each of one or more layers L be either the first layer or the second layer. For example, if the specifications of the terminal device 10 indicate that rendering of layers L exceeding 2000 in the relative value of the above-mentioned data volume is not possible, the determination unit 213 determines that layers L2 and L3 are the first layer and layer L1 is the second layer.

[0058] When the acquisition unit 212 acquires processing load information as rendering performance information, the determination unit 213 determines, based on the processing load information, that each of one or more layers L be either the first layer or the second layer. For example, if the current processing load of the terminal device 10 allows rendering only of layers L with a relative data volume of 1000 or less, the determination unit 213 determines that layer L3 is the first layer, and layers L1 and L2 are the second layers.

[0059] Generally, the lower the rendering performance of the terminal device 10, the more layers L will be rendered by the rendering server 30. For example, comparing the case where the terminal device 10 cannot render more than 2000 layers L based on the relative data volume mentioned above, with the case where it cannot render more than 1000 layers L based on the relative data volume mentioned above, the rendering performance of the terminal device 10 is lower in the latter case. In this case, the determination unit 213 will determine layer L1 as the second layer in the former case, and layers L1 and L2 as the second layer in the latter case. In other words, the lower the rendering performance of the terminal device 10 indicated by the rendering performance information, the larger the proportion of the second layer among one or more layers L the determination unit 213 will make.

[0060] Furthermore, high-resolution layers L or layers L with motion require a large amount of resources for rendering, so it is preferable to render them on the rendering server 30. Generally, high-resolution layers L or layers L with motion have a large amount of data. Therefore, the determination unit 213 may determine one or more layers as the second layer, starting with the layers L with the largest data volume. High-resolution layers L include layers L containing high-polygon CG, layers L containing high-resolution textures, and layers L with ray tracing. For example, if the determination unit 213 determines that it cannot render all layers L on the terminal device 10 alone, it will determine layer L1, which has the largest data volume among layers L1 to L3, as the second layer. If it determines that rendering is still difficult on the terminal device 10 even after making layer L1 the second layer, the determination unit 213 will further determine layer L2 as the second layer. In this way, by determining the second layer in order of the amount of data volume, the layers L to be rendered on the rendering server 30 can be efficiently determined.

[0061] The transmission control unit 214 controls the transmission of virtual object data VD or layer data LD to the terminal device 10 and the rendering server 30. For example, the transmission control unit 214 transmits virtual object data VD to the terminal device 10. The virtual object data VD includes information indicating whether each layer L has been determined to be the first layer or the second layer. Of the virtual object data VD, the layer data LD of the layer L determined to be the second layer does not need to be transmitted to the terminal device 10. The transmission control unit 214 also transmits the layer data LD of the layer L determined to be the second layer to the rendering server 30. This layer data LD includes destination information for the rendering data obtained by rendering the layer data LD (identification information for identifying the terminal device 10 on the network N).

[0062] The rendering unit 312 of the rendering server 30 renders the layer data LD transmitted from the data server 20 to generate rendering data. The communication control unit 311 of the rendering server 30 transmits the rendering data to the terminal device 10.

[0063] The rendering unit 112 of the terminal device 10 generates rendering data by rendering the layer data LD of the layer L determined to be the first layer from the virtual object data VD transmitted from the data server 20. If all layers L are the first layer, the display control unit 113 superimposes the rendering data of each layer L rendered by the terminal device 10 and displays it to the user. If there is a mix of first and second layers, the display control unit 113 superimposes the rendering data of the first layer layer L rendered by the terminal device 10 with the rendering data of the second layer received from the rendering server 30 and displays it to the user. Furthermore, if all layers L are the second layer, the display control unit 113 of the terminal device 10 superimposes the rendering data of the second layer received from the rendering server 30 and displays it to the user.

[0064] A-6. Operation of System 1 Figure 7 is a flowchart showing the operation of System 1. When the virtual tour starts, the terminal device 10 sends the name of the virtual tour city specified by the user to the data server 20, and during the execution of the virtual tour, it sends the detected value of the sensor 104 (indicated as "sensor value" in the figure) (step S101). Based on the specified city name and the detected value of the sensor 104, the data server 20 selects the virtual object data VD and the portion of the virtual object data VD to be rendered (step S102, indicated as "select virtual object data" in the figure).

[0065] Furthermore, the terminal device 10 transmits rendering performance information indicating its own rendering performance to the data server 20 (step S103). Based on the rendering performance information, the data server 20 determines whether all layers L of the virtual object data VD can be rendered by the terminal device 10. In the example in Figure 6, it is determined that not all layers L can be rendered by the terminal device 10, and that some of the layers L will be rendered by the rendering server 30. The data server 20 determines whether each layer L included in the virtual object data VD will be a first layer to be rendered by the terminal device 10, or a second layer to be rendered by the rendering server 30 (step S104, indicated as "determine first or second layer" in the figure).

[0066] The data server 20 sends the layer data LD corresponding to the second layer (indicated as "data for the second layer" in the figure) to the rendering server 30 (step S105). The rendering server 30 renders the layer data LD corresponding to the second layer (step S106) and sends the generated rendering data to the terminal device 10 (step S109).

[0067] Furthermore, the data server 20 transmits virtual object data VD, which includes layer data LD (indicated as "data for the first layer" in the figure) corresponding to the first layer, to the terminal device 10 (step S107). The terminal device 10 renders the layer data LD corresponding to the first layer (step S108). The terminal device 10 uses the rendering data generated in step S108 and the rendering data transmitted in step S109 to superimpose and display all layers L of the virtual object data VD (step S110).

[0068] A-7. Summary of Embodiments As described above, the data server 20 in this embodiment determines, based on the rendering performance of the terminal device 10, whether to render each layer L included in the virtual object data VD on the terminal device 10 or on the rendering server 30. Therefore, even if the rendering performance of the terminal device 10 is low, high-quality content can be displayed. In addition, since the rendering server 30 is used on a layer-by-layer basis, costs are reduced compared to the case where the entire virtual object data VD is rendered on the rendering server 30.

[0069] Furthermore, the data server 20 uses the specification information of the terminal device 10 as rendering performance information. Since the specification information is specific to the terminal device 10, the rendering performance of the terminal device 10 can be easily determined.

[0070] Furthermore, the data server 20 uses the processing load information of the terminal device 10 as rendering performance information. Since the processing load information reflects the state of the terminal device 10 at any given time, the rendering performance of the terminal device 10 can be grasped with greater accuracy.

[0071] Furthermore, the data server 20 increases the proportion of layer L rendered by the rendering server 30 as the rendering performance of the terminal device 10 decreases. Therefore, the number of layers L rendered by the terminal device 10 is determined according to the rendering performance of the terminal device 10.

[0072] Furthermore, the data server 20 determines the second layer from among the layers L included in the virtual object data VD, starting with the layer L with the largest data volume. Therefore, the layer L that requires high rendering performance for rendering is determined to be the second layer. Thus, the second layer is determined efficiently.

[0073] B: Modification The following are examples of modifications in the above-described embodiment. Two or more modifications selected from the following examples may be combined as appropriate, provided they do not contradict each other.

[0074] B1: First variation Figure 8 is a block diagram showing the configuration of the data server 20 according to the first modified example. The data server 20 according to the first modified example includes a reception unit 215 in addition to the configuration of the data server 20 according to the embodiment. The reception unit 215 receives a specification from the user of the terminal device 10, indicating whether the rendering of virtual object data VD should be performed on the terminal device 10 or on the rendering server 30. For example, if the user specifies that the rendering of virtual object data VD should be performed on the terminal device 10, the determination unit 213 determines all layers L to be first layers, regardless of the rendering performance of the terminal device 10. Also, if the user specifies that the rendering of virtual object data VD should be performed on the rendering server 30, the determination unit 213 determines all layers L to be second layers. In other words, when the reception unit 215 receives a specification, the determination unit 213 determines each of one or more layers L to be either a first layer or a second layer according to that specification.

[0075] Furthermore, if the virtual object data VD to be rendered is determined, the reception unit 215 may also accept a specification on a layer L basis, indicating whether rendering should be performed on the terminal device 10 or on the rendering server 30. For example, it may be possible to specify that layers L1 and L2 be rendered on the rendering server 30 and layer L3 be rendered on the terminal device 10.

[0076] According to the first modification, the user can specify the device that performs the rendering, thus improving user convenience. For example, some users may want to render on the terminal device 10 without using the rendering server 30, even if it causes delays, because they do not want to incur costs. On the other hand, even if rendering is possible on the terminal device 10, some users may want to render on the rendering server 30, even if it costs more, so that they can view VR content more comfortably. According to the first modification, various user needs can be easily reflected in the rendering process.

[0077] B2: Second variation In the embodiment described above, the data server 20 had the function of a rendering control device. However, it is not limited to this, and for example, the terminal device 10 may also have the function of a rendering control device. In this case, the terminal device 10 may, for example, detect that the processing load of its own device has increased and rendering performance has deteriorated, or detect that the virtual object data VD contains layers L that cannot be rendered with the specifications of its own device. If either of the above is detected, the terminal device 10 sends at least some of the layers L to the rendering server 30 and causes the rendering server 30 to render the layers L.

[0078] According to the second modification, since the terminal device 10 has the function of a rendering control device, the rendering performance of the terminal device 10 can be grasped immediately, which is particularly advantageous when processing load information is used as rendering performance information.

[0079] B3: Third variation In the embodiment described above, the terminal device 10 was a device that displays VR content, but it is not limited to this, and may be a device that displays other XR content such as AR content or MR content.

[0080] C: Other (1-1) The block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may be realized by combining the above one device or the above multiple devices with software. Functions include, but are not limited to, judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. In all cases, as mentioned above, the method of implementation is not particularly limited.

[0081] (1-2) Notification of information is not limited to the embodiments / models described herein and may be carried out by other means. For example, notification of information may be carried out by physical layer signaling (e.g., DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (e.g., RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, broadcast information (MIB (Master Information Block), SIB (System Information Block))), other signals, or combinations thereof. RRC signaling may also be called RRC messages, for example, RRC Connection Setup messages, RRC Connection Reconfiguration messages, etc.

[0082] (1-3) Each aspect / embodiment described in this disclosure is LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (where x is, for example, an integer or decimal)), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide This may apply to systems utilizing Bluetooth (Typing Band), Bluetooth®, or other appropriate systems, and to at least one of next-generation systems that are extended, modified, created, or defined based thereon. It may also apply to a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G).

[0083] (1-4) The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described herein may be in any order, provided that they do not contradict each other. For example, the methods described herein present various step elements in an exemplary order and are not limited to the specific order presented.

[0084] (1-5) Certain operations described in this disclosure as being performed by a base station may, in some cases, be performed by its upper node. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal can be performed by the base station and at least one other network node (for example, an MME or S-GW, but not limited to these). The above example illustrates the case where there is one other network node besides the base station, but it may also be a combination of multiple other network nodes (for example, an MME and an S-GW).

[0085] (1-6) Information, etc. (see the section on "Information, Signals") may be output from a higher layer (or lower layer) to a lower layer (or higher layer). Input and output may also occur via multiple network nodes.

[0086] (1-7) Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.

[0087] (1-8) The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0088] (1-9) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed in practice. Furthermore, notification of certain information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0089] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in the present disclosure are illustrative and not intended to be restrictive in any way.

[0090] (2-1) Software should be interpreted broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages ​​or by any other name. Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0091] (2-2) The information, signals, etc. described in this disclosure may be represented using any of the various different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. The terms described in this disclosure and the terms necessary for understanding this disclosure may be replaced with terms that have the same or similar meanings. For example, at least one of the channels and symbols may be a signal (signaling). Also, a signal may be a message. Also, a component carrier (CC) may be called a carrier frequency, cell, frequency carrier, etc.

[0092] (2-3) The terms “system” and “network” as used in this disclosure are interchangeable.

[0093] (2-4) Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information. For example, radio resources may be indicated by an index. The names used for the parameters described above are not limiting in any way. Moreover, the formulas, etc., that use these parameters may differ from those expressly disclosed in this disclosure. Various channels (e.g., PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, so the various names assigned to these various channels and information elements are not limiting in any way.

[0094] (2-5) In this disclosure, terms such as “Base Station (BS)”, “wireless base station”, “fixed station”, “NodeB”, “eNodeB (eNB)”, “gNodeB (gNB)”, “access point”, “transmission point”, “reception point”, “transmission / reception point”, “cell”, “sector”, “cell group”, “carrier”, and “component carrier” may be used interchangeably. A base station may also be referred to by terms such as macrocell, small cell, femtocell, picocell. A base station may house one or more (e.g., three) cells. When a base station accommodates multiple cells, the entire coverage area of ​​the base station can be divided into multiple smaller areas, each of which may be provided with communication services by a base station subsystem (for example, a Remote Radio Head (RRH)). The terms “cell” or “sector” refer to part or all of the coverage area of ​​at least one of the base station and / or base station subsystems that provide communication services in that coverage. In this disclosure, the transmission of information by a base station to a terminal may be interpreted as the base station instructing the terminal to perform information-based control or operation.

[0095] (2-6) In this disclosure, terms such as “Mobile Station (MS),” “user terminal,” “User Equipment (UE),” and “terminal” may be used interchangeably. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms.

[0096] (2-7) At least one of the base station and the mobile station may be called a transmitting device, a receiving device, a communication device, etc. At least one of the base station and the mobile station may be a device mounted on a mobile body, the mobile body itself, etc. The mobile body means a movable object, and the speed of movement is arbitrary. This also includes the case when the mobile body is stationary. The mobile body includes, but is not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and things mounted on them. The mobile body may also be a mobile body that moves autonomously based on operation commands. It may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). Furthermore, at least one of the base station and the mobile station may include devices that do not necessarily move during communication operations. For example, at least one of the base station and the mobile station may be an IoT (Internet of Things) device such as a sensor. Also, the term "base station" in this disclosure may be interpreted as "user terminal." For example, each aspect / embodiment of this disclosure may be applied to a configuration in which communication between a base station and a user terminal is replaced with communication between multiple user terminals (which may be called D2D (Device-to-Device), V2X (Vehicle-to-Everything), etc.). In this case, the user terminal may have the functions that the base station has. Also, terms such as "uplink" and "downlink" may be interpreted as terms corresponding to terminal-to-terminal communication (for example, "side"). For example, uplink channel, downlink channel, etc. may be interpreted as side channel. Similarly, the term "user terminal" in this disclosure may be interpreted as "base station." In this case, the base station may have the functions that the user terminal has.

[0097] (3-1) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or other data structure), and ascertaining. “Determining” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0098] (3-2) The terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0099] (3-3) The reference signal may also be abbreviated as RS (Reference Signal) and may be called Pilot depending on the applicable standard.

[0100] (3-4) The phrase “based on” as used in this disclosure does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”

[0101] (3-5) Any reference to elements using the designations “first,” “second,” etc., as used in this disclosure does not limit the quantity or order of those elements in general. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to first and second elements do not imply that only two elements may be adopted, or that the first element must precede the second element in any way.

[0102] (3-6) The term "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.

[0103] (3-7) Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

[0104] (3-8) In the present disclosure, if articles are added by translation, such as a, an, and the in English, the present disclosure may include the fact that the noun following these articles is plural.

[0105] (3-9) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

[0106] (4) It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Accordingly, the description herein is for illustrative purposes only and is not intended to be restrictive in any way to the present invention. Furthermore, multiple embodiments selected from those illustrated herein may be combined. [Explanation of symbols]

[0107] 1...System, 10...Terminal device, 20...Data server, 30...Rendering server, 101...Projection device, 102...Speaker, 103...Communication device, 104...Sensor, 105...Input device, 107...Storage device, 108...Processing device, 111...Communication control unit, 112...Rendering unit, 113...Display control unit, 203...Communication device, 205...Storage device, 206...Processing device, 211...Selection unit, 212...Acquisition unit, 213...Decision unit, 214...Transmission control unit, 215...Reception unit, 303...Communication device, 305...Storage device, 306...Processing device, 311...Communication control unit, 312...Rendering unit, L(L1~L3)...Layer, LD(LD1~LD3)...Layer data, N...Network, V...Virtual object, VD...Virtual object data.

Claims

1. A terminal device that displays one or more virtual objects arranged on one or more layers based on virtual object data, comprising: an acquisition unit that acquires rendering performance information indicating the rendering performance of the virtual object data; The system includes a determination unit that determines, based on the rendering performance information, whether each of the one or more layers is a first layer rendered by the terminal device or a second layer rendered by a rendering server connected to the terminal device via a network, The aforementioned determination unit, If all of the one or more layers can be rendered by the terminal device, all of the one or more layers are designated as the first layer. If it is difficult to render all of the one or more layers on the terminal device, at least a portion of the one or more layers is designated as the second layer. Rendering control device.

2. The acquisition unit acquires specification information indicating the specifications of the terminal device as rendering performance information. The determination unit, if one or more layers include a first unprocessable layer that cannot be rendered according to the specifications of the terminal device, determines the first unprocessable layer to be the second layer and determines the layers other than the first unprocessable layer to be the first layer. The rendering control device according to claim 1.

3. The acquisition unit acquires processing load information indicating the processing load of the terminal device as rendering performance information, The determination unit, if one or more layers include a second unprocessable layer that cannot be rendered with the current processing load of the terminal device, determines the second unprocessable layer as the second layer and determines the layers other than the second unprocessable layer as the first layer. The rendering control device according to claim 1.

4. The determination unit increases the proportion of the second layer to the one or more layers as the rendering performance of the terminal device, as indicated by the rendering performance information, decreases. The rendering control device according to claim 1.

5. The determination unit determines one or more layers from the one or more layers, starting with the layer with the largest amount of data, to be the second layer. The rendering control device according to claim 1.

6. The terminal device further includes a receiving unit that receives a specification from the user of the terminal device regarding whether the rendering of the virtual object data should be performed on the terminal device or on the rendering server. When the receiving unit receives the designation, the determination unit determines each of the one or more layers to be the first layer or the second layer according to the designation. The rendering control device according to claim 1.

Images