Virtual object data selection device

Abstract

A selection unit according to the present invention selects one or more virtual object data from among a plurality of virtual object data on the basis of first location information, first shape information, a plurality of units of second shape information and a plurality of units of second location information. The first location information expresses the location of a user in actual space. The first shape information expresses the shape of an actual object positioned in actual space. Each of the plurality of units of second shape information expresses the shape of each virtual object. Each of the plurality of units of second location information expresses the location in the actual space at which each virtual object is positioned. A transmission control unit transmits virtual object data to the display device. The selection unit estimates one or more visible objects which would be visible from the location of the user, and selects virtual object data which correspond to the one or more visible objects as the one or more virtual object data.

Description

[Technical Field]

[0001] The present invention relates to a virtual object data selection device. [Background technology]

[0002] In recent years, XR (Extended Reality) technology has become widespread. XR is a general term for AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality), etc. For example, in a VR content management system described in Patent Document 1 below, a server converts object data in a specific format into object data in a common format that is different from the specific format. The server manages VR content including object data in the common format and a setting file. A content creation device creates VR content including object data in the common format and a setting file downloaded from the server. The content creation device uploads the setting file to the server. A viewing device allows a viewer to virtually experience immersion in the VR content to be viewed that has been downloaded from the server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-168934 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, virtual object data is transmitted to an XR device in units of areas set in real or virtual space. That is, when a user holding an XR device enters an area, virtual object data corresponding to all virtual objects placed within the area is transmitted. On the other hand, if visibility within the area is poor, the user may not be able to see all of the virtual objects within the area, and some of the virtual object data transmitted to the XP device may not be used. Furthermore, even if a large virtual object is visible from outside the area, transmission may not begin until the user enters the area.

[0005] An object of the present invention is to efficiently transmit and receive virtual object data. [Means for solving the problem]

[0006] A virtual object data selection device according to one embodiment of the present invention includes: a selection unit that selects at least one virtual object data from a plurality of virtual object data that correspond one-to-one to a plurality of virtual objects based on first position information, first shape information, a plurality of second shape information, and a plurality of second position information, wherein the first position information indicates a user's position in real space, the first shape information indicates a shape of a real object located in the real space, each of the plurality of second shape information indicates a shape of a respective virtual object, and each of the plurality of second position information indicates a position in real space at which each virtual object is placed; and a transmission control unit that transmits the at least one virtual object data selected by the selection unit to a display device, wherein the display device displays a virtual object based on the at least one virtual object data by superimposing it on a field of view of the user in the real space, and the selection unit estimates at least one visual object that is visible from the user's position from the plurality of virtual objects, and selects virtual object data corresponding to the at least one visual object as the at least one virtual object data. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to efficiently transmit and receive virtual object data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a system 1 according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of a terminal device 10. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server 20. [Figure 4] FIG. 2 is a plan view showing the positional relationship of objects in a virtual object map MP. [Figure 5] FIG. 2 is a schematic diagram showing a virtual object map MP. [Figure 6] 10 is a flowchart showing the operation of the processing device 206. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. Embodiment A-1. System Configuration FIG. 1 is a block diagram showing the configuration of a system 1 according to an embodiment. The system 1 includes a terminal device 10 and a server 20. The terminal device 10 is an example of a display device, and the server 20 is an example of a virtual object data selection device. The terminal device 10 and the server 20 are connected via a communication network N. Although FIG. 1 shows only one terminal device 10, the system 1 can include any number of terminal devices 10.

[0010] In this embodiment, the system 1 is a system that uses AR technology to present various types of information to a user holding a terminal device 10. Here, AR technology is a technology that displays a virtual object superimposed on real space, allowing the user to visually recognize the virtual object as if it were present in real space. That is, the terminal device 10 displays a virtual object V (see FIG. 5, etc.) based on virtual object data VD (see FIG. 3) superimposed on real space within the user's field of view. The virtual object is, for example, a still image, a video, a 3DCG model, text, etc. Note that the terminal device 10 may output other types of information, such as audio information, when displaying the virtual object V.

[0011] The terminal device 10 is, for example, a see-through head-mounted display or a mobile information processing terminal such as a smartphone or tablet. In this embodiment, the terminal device 10 is a see-through head-mounted display. The server 20 stores a plurality of virtual object data VD for displaying virtual objects on the terminal device 10. Each virtual object data VD corresponds one-to-one to a different virtual object V. The terminal device 10 receives the virtual object data VD from the server 20 and displays the virtual object V corresponding to the virtual object data VD.

[0012] A-2. Terminal device 10 2 is a block diagram showing the configuration of the terminal device 10. The shape of the terminal device 10 is similar to that of ordinary eyeglasses, for example. The terminal device 10 has a left lens placed in front of the user's left eye, a right lens placed in front of the user's right eye, and a frame that supports the left and right lenses. The frame has a bridge provided between the left and right lenses and a pair of temples that rest on the left and right ears.

[0013] The terminal device 10 includes a projection device 101, a speaker 102, a communication device 103, a GPS device 104, a storage device 107, a processing device 108, and a bus 109. Each component shown in FIG. 2 is stored in a frame, for example. The projection device 101, the speaker 102, the communication device 103, the GPS device 104, the storage device 107, and the processing device 108 are interconnected by a bus 109 for communicating information. The bus 109 may be configured using a single bus, or may be configured using different buses for each element of the device, etc.

[0014] The projection device 101 includes left and right lenses, a display panel, and optical members. The display panel and optical members are housed in, for example, a frame. A pair of display panels and optical members may be provided, one on each side, corresponding to the left and right lenses. The projection device 101 displays a projection image corresponding to a virtual object on the display panel based on control from the processing device 108. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The optical members guide light emitted from the display panel to the left and right lenses.

[0015] Each of the left and right lenses has a half mirror. The half mirrors in the left and right lenses transmit light representing real space, thereby guiding the light representing real space to the user's eyes. The half mirrors in the left and right lenses also reflect light representing virtual objects guided by optical elements toward the user's eyes. The light of real space that has passed through the half mirror and the light representing virtual objects that has been reflected by the half mirror are superimposed and enter the user's eyes, causing the user to perceive the virtual objects as being located in real space. In other words, the left and right lenses function as a transmissive display placed in front of the eyeballs.

[0016] The speaker 102 outputs sound. The sound output by the speaker 102 may be, for example, sound output in association with a virtual object. The speaker 102 is controlled by the processing device 108. The speaker 102 is arranged, for example, in a frame. Alternatively, the speaker 102 may not be included in the terminal device 10 and may be separate from the terminal device 10.

[0017] The communication device 103 communicates with the server 20 using wireless communication or wired communication. In this embodiment, the communication device 103 has an interface connectable to a communication network N, and communicates with a communication device 203 (see FIG. 3) of the server 20 via the communication network N.

[0018] The GPS device 104 receives radio waves from multiple satellites and generates location information from the received radio waves. The location information indicates the location of the terminal device 10. The location information may be in any format as long as it can identify the location. In this embodiment, latitude and longitude are used as the location information. Note that the location information may be obtained using means other than the GPS device 104. For example, the terminal device 10 may receive information identifying the name of the facility where it is located and its location within the facility from a beacon or the like installed in the facility.

[0019] In addition, instead of or in addition to the GPS device 104, the position and attitude of the terminal device 10 may be detected by sensors such as a geomagnetic sensor, an acceleration sensor, an angular acceleration sensor, or an inertial measurement unit (IMU).

[0020] Alternatively, instead of or in addition to the GPS device 104, the position of the terminal device 10 may be detected by a VPS (Visual Positioning Service / System). In this case, the terminal device 10 has an imaging device that captures images of the user's surroundings, and transmits the captured images captured by the imaging device to the server 20 (or a location specifying server that specifies the position of the terminal device 10). The server 20 (or the location specifying server) has a map in which images of real objects placed in real space are associated with position information of the real objects. The server 20 (or the location specifying server) collates the captured image with the map to specify the position where the captured image was captured, i.e., the position of the terminal device 10.

[0021] Alternatively, for example, markers indicating position information may be placed in real space in advance, and the terminal device 10 may acquire the position information by detecting the markers from an image captured by an imaging device.

[0022] The storage device 107 is a recording medium readable by the processing device 108. The storage device 107 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The storage device 107 stores a program PG1. The program PG1 is a program for operating the terminal device 10.

[0023] The processing device 108 includes one or more central processing units (CPUs). The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0024] The processing device 108 reads the program PG1 from the storage device 107. The processing device 108 executes the program PG1 to function as an operation control unit 110. The operation control unit 110 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0025] The operation control unit 110 controls the operation of the terminal device 10. For example, the operation control unit 110 controls the projection device 101 based on the virtual object data VD from the server 20 via the communication device 103, and displays a virtual object superimposed on real space.

[0026] Note that when the terminal device 10 is a mobile information processing terminal such as a smartphone or a tablet, the terminal device 10 includes, for example, an imaging device and a display instead of the projection device 101. The terminal device 10 displays an image captured by the imaging device on the display, and displays a virtual object V corresponding to virtual object data VD acquired from the server 20 superimposed on the captured image displayed on the display.

[0027] A-3. Server 20 3 is a block diagram showing the configuration of server 20. Server 20 includes a communication device 203, a storage device 205, a processing device 206, and a bus 207. Communication device 203, storage device 205, and processing device 206 are interconnected by bus 207 for communicating information. Bus 207 may be configured using a single bus, or may be configured using different buses between each device.

[0028] The communication device 203 communicates with the terminal device 10 using wireless communication or wired communication. In this embodiment, the communication device 203 has an interface connectable to a communication network N, and communicates with the terminal device 10 via the communication network N.

[0029] The storage device 205 is a recording medium readable by the processing device 206. The storage device 205 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 205 stores a program PG2, a plurality of virtual object data VD (VD to VDn), and a virtual object map MP.

[0030] The program PG2 is a program for operating the server 20. The virtual object data VD is data for outputting virtual objects on the terminal device 10. The multiple virtual object data VD correspond to different virtual objects, respectively. That is, the multiple virtual object data VD correspond one-to-one to the multiple virtual objects V. Hereinafter, as necessary, the virtual object data VD corresponding to the virtual object Vn will be referred to as virtual object data VDn. For example, the virtual object data VD corresponding to the virtual object V1 is virtual object data VD1. The virtual object map MP is data indicating the arrangement of the virtual object V in real space. The virtual object map MP will be described in detail later.

[0031] The processing unit 206 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processors and CPUs is an example of a computer.

[0032] The processing device 206 reads the program PG2 from the storage device 205. The processing device 206 executes the program PG3 to function as an acquisition unit 210, a selection unit 212, a decision unit 214, and a transmission control unit 216. At least some of the acquisition unit 210, the selection unit 212, the decision unit 214, and the transmission control unit 216 may be configured by circuits such as a DSP, an ASIC, a PLD, and an FPGA.

[0033] A-4. Details of the processing device 206 Next, details of the acquisition unit 210, selection unit 212, decision unit 214, and transmission control unit 216, which are realized by the processing unit 206 executing the program PG2, will be described. Prior to the explanation, details of the virtual object map MP stored in the storage device 205 will be described.

[0034] Fig. 5 is a schematic diagram showing a virtual object map MP, and Fig. 4 is a plan view showing the positional relationship of objects in the virtual object map MP. Fig. 5 is a view of the three-dimensional virtual object map MP viewed in the direction indicated by arrow D from point P shown in Fig. 4. In Figs. 4 and 5, virtual object V is shown shaded.

[0035] The virtual object map MP includes first shape information, multiple pieces of second shape information, and multiple pieces of second position information. The first shape information indicates the shape of a real object located in real space. Each of the multiple pieces of second shape information indicates the shape of each of the multiple virtual objects V1 to Vn. Each of the multiple pieces of second position information indicates the position in real space where each of the multiple virtual objects V1 to Vn is located. For example, FIG. 5 shows buildings B1 to B4, trees T1 to T3, road S, and the like, which are examples of real objects located in real space, in three dimensions. FIG. 5 also shows virtual objects V1, V3, and V6, which are virtually located in real space, superimposed on the real objects, in three dimensions.

[0036] The first shape information is obtained, for example, by three-dimensionally scanning the real space in advance. The second shape information and second position information are obtained from the virtual object data VD. The virtual object map MP does not necessarily include all of the first shape information, second shape information, and second position information. For example, these pieces of information may be stored separately and read each time to form a three-dimensional map such as that shown in FIG. 5.

[0037] By using the virtual object map MP, it is possible to grasp how real-space objects and virtual objects appear when viewed from any position in real space. For example, referring to FIG. 4, virtual objects V1 to V7 are located around point P, but when viewed from point P in the direction indicated by arrow D, only virtual objects V1, V3, and V6 are visible. For example, virtual object V2 is hidden by tree T2 and is not visible. Virtual object V4 is located inside building B1 and is therefore not visible. Virtual object V5 is located in an area opposite the direction indicated by arrow D from point P and is therefore not visible. Virtual object V7 is located in an area opposite the direction indicated by arrow D from point P and is hidden by tree T5 and is therefore not visible.

[0038] Next, each functional unit (see FIG. 3) of the processing device 206 will be described. The acquisition unit 210 acquires first position information indicating the user's position in real space. In this embodiment, the acquisition unit 210 acquires position information generated by the GPS device 104 (see FIG. 2) as the first position information.

[0039] The selection unit 212 selects at least one virtual object data VD from the plurality of virtual object data VD1 to VDn based on the first position information, the first shape information, the plurality of pieces of second shape information, and the plurality of pieces of second position information. The virtual object data VD selected by the selection unit 212 is the virtual object data VD to be transmitted to the terminal device 10 via the transmission control unit 216, which will be described later. Hereinafter, the virtual object data VD selected by the selection unit 212 may be referred to as "selected object data."

[0040] The selection unit 212 estimates at least one viewed object from among the multiple virtual objects V1 to Vn that is viewable from the user's position, and selects virtual object data VD corresponding to the at least one viewed object as at least one selected object data. As described above, the virtual object map MP includes first shape information, multiple pieces of second shape information, and multiple pieces of second position information. The selection unit 212 estimates how the real space and virtual object V appear at the user's position based on the virtual object map MP and the first position information acquired by the acquisition unit 210. The virtual object visible from the user's position is the viewed object. For example, when the user is located at point P shown in FIG. 4 and is facing the direction indicated by arrow D, the selection unit 212 estimates that the viewed objects are virtual objects V1, V3, and V6. In this case, the selection unit 212 selects the virtual object data VD1, VD3, and VD6 as the selected object data.

[0041] The range that the user can view varies depending on the direction of the user's face. For example, the selection unit 212 may estimate the user's moving direction based on a change over time in the first position information, and estimate that the user's face is facing in the moving direction. Furthermore, if the direction of the face is unknown, the selection unit 212 may estimate the viewed objects over the entire periphery from the user's position. For example, if the user's position is point P shown in FIG. 4 and the direction the user is facing is unknown, the selection unit 212 estimates that the viewed objects are virtual objects V1, V3, V5, and V6. In this case, the selection unit 212 selects virtual object data VD1, VD3, VD5, and VD6 as selected object data.

[0042] Here, the viewed object is a virtual object V that is currently viewable by the user, but the selection unit 212 may also predict a virtual object V that is predicted to be viewable by the user in the near future (e.g., several seconds, several tens of seconds, or several minutes). That is, the selection unit 212 estimates the user's position after a predetermined time based on at least one of the user's moving direction and the user's moving speed. The selection unit 212 estimates at least one predicted viewed object from the user's estimated position after the predetermined time from among the multiple virtual objects V1 to Vn, and selects virtual object data VD corresponding to the at least one predicted viewed object as selected object data. In this case, the transmission control unit 216, which will be described later, transmits the virtual object data VD (selected object data) corresponding to the predicted viewed object to the terminal device 10.

[0043] For example, if the user's current location is point P shown in FIG. 4 and the user is walking in the direction indicated by arrow D, it is predicted that the user will reach point Q after a predetermined time. It is predicted that virtual object V2, which was hidden by tree T2 at point P and was not visible, will become visible from point Q. In this case, the selection unit 212 estimates that the predicted visible object is virtual object V2, and selects virtual object data VD2 as the selected object data.

[0044] In this way, the virtual object data VD of a predicted visible object that is likely to be visible to the user in the near future is transmitted in advance to the terminal device 10 as selected object data, allowing the terminal device 10 to smoothly display the virtual object V. Furthermore, even if the amount of selected object data is large, for example, the data transmission time can be secured, making it less likely that delays or display defects will occur in the display of the virtual object V. This makes it possible to improve the quality of the visual experience of the user using the terminal device 10.

[0045] As described above, the user's movement direction can be estimated based on the direction of change over time in the user's position indicated by the first position information. The user's movement speed can be estimated based on the displacement per unit time of the user's position indicated by the first position information. The user's movement direction and movement speed can be estimated more accurately by using the first shape information. For example, if the user is located in an aisle, it can be predicted that the user's movement direction is likely to follow the aisle. Furthermore, in areas with elevation differences, such as stairs or slopes, it is predicted that the movement speed will be slower than in flat areas. Therefore, the selection unit 212 may predict the user's movement direction and movement speed using the first shape information in addition to the first position information.

[0046] When multiple virtual object data VD are selected by the selection unit 212 (when there is multiple selected object data), the determination unit 214 determines the transmission order of the multiple virtual object data VD. The determination unit 214 determines the transmission order of the multiple virtual object data VD by, for example, one of the following methods [1] to [3].

[0047] Note that when transmitting multiple virtual object data VD, there are two possible cases: one in which the virtual object data VD is transmitted serially one by one, and the other in which the multiple virtual object data VD is transmitted in parallel. In the latter case, the transmission order of the multiple virtual object data VD may be, for example, the order in which the transmission of each virtual object data VD is started. In this case, depending on the amount of virtual object data VD, the transmission of virtual object data VD that started later than the transmission of virtual object data VD that started earlier may be completed earlier.

[0048] [1] Transmits virtual object V in order from the closest to the user's position. Among visual objects, a visual object located closer to the user may be the one that the user is more likely to pay attention to. Furthermore, with regard to predicted visual objects, a predicted visual object closer to the user's current location is likely to be visible first. Therefore, in [1], the determination unit 214 determines to transmit the plurality of virtual object data VD in order, starting with the virtual object data VD of the virtual object V whose position in real space is closest to the user's position.

[0049] For example, when the user is located at point P shown in Fig. 4 and is facing the direction indicated by arrow D, the visual objects are virtual object data VD1, VD3, and VD6 as shown in Fig. 5. In this case, determination unit 214 determines to transmit virtual object data VD1, VD3, and VD6 to terminal device 10 in this order. According to method [1], it is possible to more reliably transmit virtual object data VD of virtual object V, which the user is likely to pay attention to.

[0050] [2] Sends virtual objects V in order of priority. When a priority order is set for each virtual object V, by transmitting the virtual object data VD in descending order of priority, the virtual object V with the highest priority order can be reliably displayed on the terminal device 10. That is, in [2], when a priority order is set for each of multiple virtual objects V, the determination unit 214 determines to transmit the multiple virtual object data VD in descending order of priority order, starting with the virtual object data VD of the virtual object V with the highest priority order.

[0051] For example, virtual objects V1 to V7 are arranged in descending order of priority as follows: virtual objects V7, V6, V5, V4, V3, V2, and V1. When the user is located at point P shown in FIG. 4 and facing the direction indicated by arrow D, the visual objects are virtual objects V1, V3, and V6 as shown in FIG. 5. In this case, determination unit 214 determines to transmit virtual object data VD6, VD3, and VD1 to terminal device 10 in this order. According to method [2], virtual object data VD of virtual object V with high priority can be transmitted more reliably.

[0052] The priority order is set, for example, by an administrator who manages the virtual object V. For example, if the virtual object V is an advertisement that is placed by paying an advertising fee from a company or the like, the priority order may be set according to the advertising fee. Specifically, the higher the advertising fee for a virtual object V, the higher the priority order may be.

[0053] [3] Transmit virtual objects in order of their occupancy in the user's field of view. Among the visual objects, the user may be more likely to pay attention to a visual object that occupies a large proportion of the user's field of view, i.e., a visual object that appears large to the user. Therefore, in [3], the determination unit 214 estimates the proportion of the user's field of view that each of the multiple visual objects displayed by the terminal device 10 occupies, and determines to transmit the virtual object data VD in descending order of the proportion of the user's field of view that each of the multiple visual objects occupies in the field of view.

[0054] For example, when the user is located at point P shown in Fig. 4 and is facing the direction indicated by arrow D, the proportion of the user's field of view that occupies the virtual objects V5, V1, and V3 in descending order, as shown in Fig. 5. Therefore, determination unit 214 determines to transmit virtual object data VD5, VD1, and VD3 to terminal device 10 in this order. According to method [3], it is possible to more reliably transmit virtual object data VD of virtual object V, which occupies a large proportion of the user's field of view and to which the user is likely to pay attention.

[0055] [4] Transmit virtual objects in order of data volume It may take a long time to complete transmission of virtual object data VD with a large amount of data. In particular, when transmitting virtual object data VD of a predicted visible object, it may be effective to transmit virtual object data VD with a large amount of data at an early timing. Therefore, in [4], the determination unit 214 determines to transmit multiple virtual object data VD in descending order of the amount of data of the virtual object data VD.

[0056] For example, when the virtual object data VD1 to VD7 are arranged in descending order of data volume, the virtual objects are V6, V5, V4, V2, V1, V3, and V7. When the user is located at point P shown in FIG. 4 and facing the direction indicated by arrow D, the visual objects are virtual objects V1, V3, and V6. In this case, determination unit 214 determines to transmit virtual object data VD6, VD1, and VD3 to terminal device 10 in this order. According to method [4], virtual object data VD with a large data volume can be transmitted more reliably.

[0057] In the above [1] to [4], the transmission order of virtual object data VD was explained using a specific example of a visual object when the user is located at point P in Figure 4, but the transmission order can also be determined in the same way as in [1] to [4] when transmitting a predicted visual object.

[0058] The transmission control unit 216 transmits the virtual object data VD to the terminal device 10. More specifically, the transmission control unit 216 controls the communication device 203 to transmit at least one virtual object data VD selected by the selection unit 213 from the plurality of virtual object data VD stored in the storage device 205 to the terminal device 10. Furthermore, when the selection unit 213 selects a plurality of virtual object data VD, the transmission control unit 216 transmits the plurality of virtual object data VD in the order determined by the determination unit 214.

[0059] The virtual object data VD transmitted by the transmission control unit 216 is received by the terminal device 10. The terminal device 10 uses the projection device 101 to display a virtual object V corresponding to at least one piece of virtual object data VD, superimposed on the user's field of view in real space. The user perceives the virtual object V as if it actually exists in the surrounding real space.

[0060] A-5. Operation of the processing unit 206 6 is a flowchart showing the operation of the processing device 206. The processing device 206 repeats the processing of FIG. 6 at predetermined processing intervals. The processing device 206 functions as the acquisition unit 210 and receives first position information from the terminal device 10 (step S101). The processing device 206 functions as the determination unit 214 and maps the position indicated by the first position information onto the virtual object map MP and estimates the viewed object (step S102). At this time, the processing device 206 may estimate the user's moving direction and moving speed based on the first position information and estimate the predicted viewed object.

[0061] The viewed object changes as the user moves. If there is a viewed object that was not present when the previous viewed object was estimated, the viewed object is designated as a "new viewed object." If there is no new viewed object (step S103: NO), the processing device 206 returns the process to step S101. On the other hand, if there is a new viewed object (step S103: YES), the processing device 206 determines whether there are multiple new viewed objects (step S104). If there are multiple new viewed objects (step S104: YES), the processing device 206 functions as the determination unit 214 and determines the transmission order of the virtual object data VD using, for example, any of the methods [1] to [4] above (step S105). If there are not multiple new viewed objects (step S104: NO), the processing device 206 advances the process to step S106.

[0062] The processing device 206 functions as the transmission control unit 216 and transmits the virtual object data VD of the new viewed object to the terminal device 10 (step S106). If there are multiple new viewed objects, the processing device 206 transmits the multiple virtual object data VD in the transmission order determined in step S105. Thereafter, the processing device 206 returns the process to step S101.

[0063] A-5. Summary of embodiments As described above, according to the embodiment, the server 20 estimates a viewed object, which is a virtual object V visible to the user, based on the user's position in real space, the shape of a real object located in real space, the shape of each of the multiple virtual objects V, and the placement of each of the multiple virtual objects V in real space, and transmits virtual object data VD of the viewed object to the terminal device 10. Therefore, compared to the prior art in which virtual object data VD within an area is transmitted all at once, unnecessary data transmission and reception is reduced. More specifically, the server 20 avoids transmitting virtual object data VD corresponding to a virtual object V that the user cannot view. This reduces the processing load on the server 20 and the terminal device 10 and enables effective use of communication resources. Furthermore, even for a large virtual object V located far from the user, such as virtual object V6, virtual object data VD is transmitted at an appropriate time.

[0064] Furthermore, the server 20 estimates a predicted viewed object, which is at least one virtual object that will be visible from the user's position after a predetermined time, and transmits virtual object data VD corresponding to the predicted viewed object to the terminal device 10. By transmitting the virtual object data VD of a predicted viewed object that is likely to be visible to the user in the near future to the terminal device 10 in advance, the terminal device 10 can smoothly display the virtual object V. Furthermore, even if the amount of virtual object data VD is large, for example, the data transmission time can be secured, making it less likely that delays or display defects will occur in the display of the virtual object V. This makes it possible to improve the quality of the visual experience of the user using the terminal device 10.

[0065] Furthermore, when transmitting the plurality of virtual object data VD, the server 20 transmits the plurality of virtual object data VD in order, for example, starting with the virtual object data VD of the virtual object V whose position in real space is closest to the user's position. This makes it possible to more reliably transmit the virtual object data VD of the virtual object V that the user is likely to pay attention to.

[0066] Furthermore, when transmitting multiple pieces of virtual object data VD, the server 20 transmits the multiple pieces of virtual object data VD in order of priority, starting with the virtual object data VD of the virtual object V with the highest priority. This makes it possible to more reliably transmit the virtual object data VD of the virtual object V with the highest priority.

[0067] Furthermore, when transmitting the plurality of virtual object data VD, the server 20 transmits the plurality of virtual object data VD in descending order of the virtual object V whose visual object displayed on the terminal device 10 occupies a large proportion of the user's visual field. This makes it possible to more reliably transmit the virtual object data VD of the virtual object V that occupies a large proportion of the user's visual field and to which the user is likely to pay attention.

[0068] Furthermore, when transmitting multiple pieces of virtual object data VD, the server 20 determines to transmit the multiple pieces of virtual object data VD in descending order of data volume, for example. This allows the virtual object data VD with the largest data volume to be transmitted more reliably.

[0069] B: Modified example The following are variations of the above-described embodiment. Two or more variations arbitrarily selected from the following variations may be combined as appropriate within the scope of not mutually contradicting each other.

[0070] B1: First modified example In the above-described embodiment, the terminal device 10 is configured only as an eyeglass-type terminal device. However, the terminal device 10 may include an eyeglass-type terminal device and a portable terminal device such as a smartphone, a tablet device, or a laptop computer. In this case, the eyeglass-type terminal device and the portable terminal device are connected to each other. When the terminal device 10 includes a portable terminal device, some or all of the functions of the operation control unit 110 may be performed by the portable terminal device.

[0071] C:Other (1-1) The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wires, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or multiple devices with software. Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, election, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0072] (1-2) Notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination thereof. Furthermore, RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0073] (1-3) Each aspect / embodiment described in the present disclosure may be implemented using a standard technology such as LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (xG (x is, for example, an integer or a decimal number)), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-Wide The present invention may be applied to at least one of systems that use the LTE Band, Bluetooth (registered trademark), or other appropriate systems, and next-generation systems that are expanded, modified, created, or defined based on these. It may also be applied to a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G).

[0074] (1-4) The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order and are not limited to the particular order presented.

[0075] (1-5) In the present disclosure, a specific operation that is described as being performed by a base station may be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and another network node other than the base station (for example, an MME or an S-GW, etc., but is not limited to these). Although the above example illustrates a case where there is one other network node other than the base station, a combination of multiple other network nodes (for example, an MME and an S-GW) may also be used.

[0076] (1-6) Information, etc. (See the "Information, Signals" section) can be output from a higher layer (or a lower layer) to a lower layer (or a higher layer). Information, etc. may be input / output via multiple network nodes.

[0077] (1-7) Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0078] (1-8) The determination may be made based on a value represented by one bit (0 or 1), a Boolean (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0079] (1-9) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0080] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0081] (2-1) Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0082] (2-2) Information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). A signal may also be a message. A component carrier (CC) may also be called a carrier frequency, a cell, a frequency carrier, etc.

[0083] (2-3) As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0084] (2-4) Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or other corresponding information. For example, radio resources may be indicated by an index. The names used for the above-described parameters are not limiting in any way. Furthermore, mathematical formulas, etc. using these parameters may differ from those explicitly disclosed in the present disclosure. Various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not limiting in any way.

[0085] (2-5) In the present disclosure, terms such as "base station (BS)," "radio base station," "fixed station," "NodeB," "eNodeB (eNB)," "gNodeB (gNB)," "access point," "transmission point," "reception point," "transmission / reception point," "cell," "sector," "cell group," "carrier," and "component carrier" may be used interchangeably. A base station may also be referred to by terms such as a macrocell, a small cell, a femtocell, and a picocell. A base station can accommodate one or more (e.g., three) cells. When a base station accommodates multiple cells, the entire coverage area of ​​the base station can be divided into multiple smaller areas, and each smaller area can be provided with communication services by a base station subsystem (e.g., a small indoor base station (RRH: Remote Radio Head)). The term "cell" or "sector" refers to a part or the entire coverage area of ​​at least one of the base station and base station subsystem that provides communication services within this coverage. In the present disclosure, the base station transmitting information to a terminal may be interpreted as the base station instructing the terminal to control and operate based on the information.

[0086] (2-6) In this disclosure, terms such as "Mobile Station (MS)," "user terminal," "User Equipment (UE)," "terminal," etc. may be used interchangeably. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

[0087] (2-7) At least one of the base station and the mobile station may be referred to as a transmitting device, a receiving device, a communication device, etc. At least one of the base station and the mobile station may be a device mounted on a mobile object, the mobile object itself, etc. The mobile object refers to a movable object, and may move at any speed. Naturally, this also includes cases where the mobile object is stationary. Examples of the mobile object include, but are not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcars, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and objects mounted thereon. The mobile object may also be a mobile object that moves autonomously based on an operational command. It may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile object (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). Note that at least one of the base station and the mobile station may include devices that do not necessarily move during communication operation. For example, at least one of the base station and the mobile station may be an IoT (Internet of Things) device such as a sensor. Furthermore, the base station in the present disclosure may be interpreted as a user terminal. For example, the aspects / embodiments of the present disclosure may be applied to a configuration in which communication between a base station and a user terminal is replaced with communication between multiple user terminals (which may be referred to as, for example, Device-to-Device (D2D) or Vehicle-to-Everything (V2X)). In this case, the user terminal may be configured to have the functions possessed by the above-mentioned base station. Furthermore, terms such as "uplink" and "downlink" may be interpreted as terms corresponding to communication between terminals (for example, "side"). For example, terms such as an uplink channel and a downlink channel may be interpreted as a side channel. Similarly, the user terminal in the present disclosure may be interpreted as a base station. In this case, the base station may be configured to have the functions possessed by the above-mentioned user terminal.

[0088] (3-1) The terms "determining" and "determining" used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0089] (3-2) The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0090] (3-3) The reference signal may be abbreviated as RS (Reference Signal) or may be called a pilot depending on the applicable standard.

[0091] (3-4) As used in this disclosure, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0092] (3-5) Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0093] (3-6) The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.

[0094] (3-7) When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0095] (3-8) In this disclosure, when articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are plural.

[0096] (3-9) In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0097] (4) It is clear to those skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the present invention as defined by the claims. Therefore, the description in this specification is intended as an illustrative explanation and does not have any limiting meaning on the present invention. Furthermore, multiple embodiments selected from the embodiments exemplified in this specification may be combined. [Explanation of symbols]

[0098] 1...system, 10...terminal device, 20...server, 101...projection device, 103...communication device, 104...GPS device, 203...communication device, 205...storage device, 206...processing device, 210...acquisition unit, 212...selection unit, 214...determination unit, 216...transmission control unit, MP...virtual object map, N...communication network, V(V1 to Vn)...virtual object, VD(VD1 to Vn)...virtual object data.

Claims

1. a selection unit that selects at least one virtual object data from a plurality of virtual object data that correspond one-to-one to a plurality of virtual objects, based on first position information, first shape information, a plurality of second shape information, and a plurality of second position information, the first position information indicates a position of the user in real space; the first shape information indicates a shape of a real object located in the real space; each of the plurality of second shape information indicates a shape of a respective virtual object; each of the plurality of second position information indicates a position in the real space at which each virtual object is placed; a selection section; a determination unit that determines a transmission order of the plurality of virtual object data when the selection unit selects the plurality of virtual object data; a transmission control unit that transmits the at least one virtual object data selected by the selection unit to a display device, the display device displays a virtual object based on the at least one piece of virtual object data in a field of view of the user in the real space by superimposing the virtual object, the selection unit estimates at least one viewable object that is viewable from the user's position without being hidden by the real object when the virtual objects are arranged in the real space based on the plurality of second position information, from among the plurality of virtual objects, and selects virtual object data corresponding to the at least one viewable object as the at least one virtual object data; the determination unit determines to transmit the plurality of virtual object data in order starting from virtual object data of a virtual object whose position in the real space is closest to a position of the user; Virtual object data selection device.

2. a selection unit that selects at least one virtual object data from a plurality of virtual object data that correspond one-to-one to a plurality of virtual objects, based on first position information, first shape information, a plurality of second shape information, and a plurality of second position information, the first position information indicates a position of the user in real space; the first shape information indicates a shape of a real object located in the real space; each of the plurality of second shape information indicates a shape of a respective virtual object; each of the plurality of second position information indicates a position in the real space at which each virtual object is placed; a selection section; a determination unit that determines a transmission order of the plurality of virtual object data when the selection unit selects the plurality of virtual object data; a transmission control unit that transmits the at least one virtual object data selected by the selection unit to a display device, the display device displays a virtual object based on the at least one piece of virtual object data in a field of view of the user in the real space by superimposing the virtual object, the selection unit estimates at least one viewable object that is viewable from the user's position without being hidden by the real object when the virtual objects are arranged in the real space based on the plurality of second position information, from among the plurality of virtual objects, and selects virtual object data corresponding to the at least one viewable object as the at least one virtual object data; a priority level is set for each of the plurality of virtual objects; the determination unit determines to transmit the plurality of virtual object data in descending order of priority of the virtual object data of the virtual object having the highest priority. Virtual object data selection device.

3. a selection unit that selects at least one virtual object data from a plurality of virtual object data that correspond one-to-one to a plurality of virtual objects, based on first position information, first shape information, a plurality of second shape information, and a plurality of second position information, the first position information indicates a position of the user in real space; the first shape information indicates a shape of a real object located in the real space; each of the plurality of second shape information indicates a shape of a respective virtual object; each of the plurality of second position information indicates a position in the real space at which each virtual object is placed; a selection section; a determination unit that determines a transmission order of the virtual object data when the selection unit selects a plurality of virtual object data; a transmission control unit that transmits the at least one virtual object data selected by the selection unit to a display device, the display device displays a virtual object based on the at least one piece of virtual object data in a field of view of the user in the real space by superimposing the virtual object, the selection unit estimates at least one viewable object that is viewable from the user's position without being hidden by the real object when the virtual objects are arranged in the real space based on the plurality of second position information, from among the plurality of virtual objects, and selects virtual object data corresponding to the at least one viewable object as the at least one virtual object data; the plurality of virtual object data selected by the selection unit correspond to a plurality of visual objects estimated to be visually recognizable from the position of the user; the determination unit estimates a proportion of the field of view that each of the plurality of visual objects displayed by the display device occupies in the field of view, and determines to transmit the plurality of virtual object data in descending order of the proportion of the field of view that each of the plurality of visual objects occupies in the field of view; Virtual object data selection device.

4. a selection unit that selects at least one virtual object data from a plurality of virtual object data that correspond one-to-one to a plurality of virtual objects, based on first position information, first shape information, a plurality of second shape information, and a plurality of second position information, the first position information indicates a position of the user in real space; the first shape information indicates a shape of a real object located in the real space; each of the plurality of second shape information indicates a shape of a respective virtual object; each of the plurality of second position information indicates a position in the real space at which each virtual object is placed; a selection section; a determination unit that determines a transmission order of the plurality of virtual object data when the selection unit selects the plurality of virtual object data; a transmission control unit that transmits the at least one virtual object data selected by the selection unit to a display device, the display device displays a virtual object based on the at least one piece of virtual object data in a field of view of the user in the real space by superimposing the virtual object, the selection unit estimates at least one viewable object that is viewable from the user's position without being hidden by the real object when the virtual objects are arranged in the real space based on the plurality of second position information, from among the plurality of virtual objects, and selects virtual object data corresponding to the at least one viewable object as the at least one virtual object data; the determination unit determines to transmit the plurality of virtual object data in descending order of data amount. Virtual object data selection device.

5. the selection unit estimates a position of the user after a predetermined time based on at least one of a moving direction of the user and a moving speed of the user, estimates at least one predicted viewed object from the plurality of virtual objects that is viewable from the estimated position of the user after the predetermined time, and selects virtual object data corresponding to the at least one predicted viewed object as the at least one virtual object data; 5. The virtual object data selection device according to claim 1.

Images