Systems, methods, terminals, methods and programs

Abstract

To solve the problem in which: when a plurality of virtual objects are displayed in disorder in the same place, users cannot show the virtual objects they want to show or cannot see the virtual objects they want to see.SOLUTION: A system is to manage virtual objects, and has management means that manages, in association with identification information, a feature quantity in a real world for displaying the virtual objects in linkage with the real world. When the virtual object is further provided, the management means manages a parameter for performing control of display relative to the other virtual objects in association with the identification information.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a technique for displaying virtual objects handled in AR (Augmented Reality) or MR (Mixed Reality) in the real world.

Background Art

[0002] XR has attracted attention as a general term for technologies that create a space for providing a pseudo-experience by fusing the real and virtual worlds such as AR (Augmented Reality) and MR (Mixed Reality), and various standardization efforts are being carried out. In recent years, a mechanism for displaying a single virtual object on multiple terminals at the same location in the real world has been realized on the platforms provided by each company. For example, there is a cloud system that manages by associating a virtual object to be placed in the real world with feature amounts of the real world captured by a camera or the like. Thereafter, by capturing the real world that matches the feature amounts managed by the system with the camera of an arbitrary terminal, it becomes possible to view the virtual object managed in association with the feature amounts on that arbitrary terminal.

[0003] Here, the association information that associates a virtual object with feature amounts of the real world captured by a camera or the like may be referred to as an anchor.

[0004] Patent Document 1 switches the display of a specific virtual object using user behavior information and physical environment information. For example, initially, a virtual object of a simple blue spherical globe is displayed, but when approaching or staring at it, it switches to one that represents detailed terrain.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] However, Patent Document 1 does not consider the display control of multiple virtual objects in accordance with user behavior information or physical environment information.

[0007] For example, if multiple users each place a virtual object in the same real-world location (such as a popular spot in a city), the virtual objects might appear haphazardly in multiple locations. This could result in the system failing to provide each user with the virtual object they intended to see, or viewers being unable to see the virtual objects they wanted to see. [Means for solving the problem]

[0008] Therefore, the present invention is a system for managing virtual objects, comprising: a management means for managing, in association with identification information, real-world feature quantities for displaying virtual objects linked to the real world, and parameters for controlling the relative display of the virtual object with other virtual objects when the virtual object is provided; and an update means for dynamically updating the numerical evaluation information based on feedback including at least one of the number of times the virtual object has been displayed linked to the real world and user input to the virtual object, wherein the parameters include numerical information of evaluations performed on the virtual object. The aforementioned parameters, in addition to the numerical information of the evaluation, include priority information for determining whether a virtual object should be prioritized over other virtual objects when displaying a virtual object in the real world. It is characterized by the following: [Effects of the Invention]

[0009] According to the present invention, a mechanism can be provided that enables the provision of appropriate information even in cases where multiple virtual objects are placed in a specific location in the real world. [Brief explanation of the drawing]

[0010] [Figure 1]The overall configuration of the virtual object management system in Example 1 is shown. [Figure 2] This is a hardware configuration diagram of the server computer and client terminals that constitute the virtual object management system in Example 1. [Figure 3] This is a software configuration diagram of the virtual object management system in Example 1. [Figure 4] This illustrates the process flow for sharing virtual objects across multiple terminals in the virtual object management system in Example 1. [Figure 5] This is a flowchart illustrating the process for generating anchors in Example 1. [Figure 6] This flowchart illustrates the process of setting priorities for anchors stored in the virtual object management system in Example 1. [Figure 7] This flowchart illustrates the process by which the virtual object management system in Example 1 searches for an anchor. [Figure 8] This flowchart illustrates the process flow in which the client terminal renders a virtual object in Example 1. [Figure 9] This flowchart illustrates the process flow in which the client terminal renders a virtual object in Example 2. [Figure 10] This flowchart illustrates the process by which the virtual object management system in Example 2 searches for an anchor. [Figure 11] This is an example of the client terminal screen in Example 1. [Figure 12] This is an example of the client terminal screen in Example 2. [Modes for carrying out the invention]

[0011] The best mode for carrying out the present invention will be described below with reference to the drawings. [Examples]

[0012] FIG. 1 is a diagram showing the overall configuration of a virtual object management system according to an embodiment of the present invention.

[0013] In FIG. 1, a virtual object management system 121, client terminals 131 to 133 are connected via networks 100 to 102. The networks 100 to 102 are so-called communication networks realized by, for example, LANs such as the Internet, WANs, telephone lines, dedicated digital lines, ATM or frame relay lines, cable TV lines, wireless lines for data broadcasting, etc. The networks 100 to 102 only need to be capable of transmitting and receiving data. In the present invention, network 100 is the Internet, and networks 101 and 102 are the Internet, networks within general households or companies, wireless LANs set up in the city, etc.

[0014] The client terminals 131 to 133 are, for example, dedicated hardware corresponding to the drawing of virtual objects handled by XR such as HMD (Head Mounted Display) or smart glasses, or mobile phones with a built-in program execution environment such as a smartphone. Further, the client terminals 131 to 133 include a camera for photographing the surroundings and a display for displaying virtual objects. The client terminals 131 to 133 photograph the surroundings through a camera or the like, draw and project virtual objects superimposed on the real world on the display, thereby providing a pseudo-experience in which the real and virtual worlds are fused for the user. When the client terminals 131 to 133 are not dedicated hardware such as a smartphone, etc., the drawing of virtual objects is performed using APIs provided by a web browser, OS, etc.

[0015] The virtual object management system 121 is a system for providing services such as associating and managing virtual objects to be placed in the real world with feature amounts of the real world captured by a camera or the like, and providing the virtual objects to an external terminal. The virtual object management system 121 is constructed using a server computer. It can be constructed by adopting cloud computing technology or the like.

[0016] In this embodiment, the association information that associates the virtual objects managed by the virtual object management system 121 with the feature amounts of the real world captured by a camera or the like is hereinafter referred to as an anchor. As the association information, in addition to those pieces of information, as will be described later, property information including an identifier for identifying the anchor itself, a session ID, and various parameters is included.

[0017] The virtual object management system 121 receives an anchor registration request from the client terminals 131 to 133 and stores the registered anchors. Further, the virtual object management system 121 receives an anchor acquisition request from the client terminals 131 to 133 and responds with an anchor that matches the conditions from the stored anchors.

[0018] Hereinafter, the functions of the server described in the present invention may be realized by a single server or a single virtual server, or may be realized by a plurality of servers or a plurality of virtual servers. Alternatively, a plurality of virtual servers may be executed on a single server.

[0019] FIG. 2 is a hardware configuration diagram of the virtual object management system 121 and the client terminals 131 to 133 according to an embodiment of the present invention.

[0020] In Figure 2, the Central Processing Unit (CPU) 202 controls the entire system. The CPU 202 executes application programs, the OS, etc., stored in the Hard Disk Drive (HDD) 205, and controls the temporary storage of information and files necessary for program execution in the Random Access Memory (RAM) 203. The Graphics Processing Unit (GPU) 210 performs the calculations necessary to render virtual objects in real time. The Read Only Memory (ROM) 204 is a storage means and stores various data such as basic I / O programs. The RAM 203 is a temporary storage means and functions as the main memory and work area for the CPU 202 and GPU 210. The HDD 205 is one of the external storage means and functions as a large-capacity memory, storing application programs such as web browsers, service server programs, the OS, and related programs. The display 206 is a display means that displays virtual objects and information necessary for operation. The interface 208 is an external device I / F and connects peripheral devices such as various external sensors. Camera 207 captures images of the surroundings at client terminals 131-133. By analyzing the images captured by camera 207 with an application program stored in HDD 205, it becomes possible to place virtual objects overlaid on the real world and calculate features of the real world. Furthermore, if client terminals 131-133 are dedicated XR terminals such as HMDs, the virtual objects displayed on display 206 can be manipulated by the user's fingers recognized by camera 207. If client terminals 131-133 are not dedicated XR terminals such as smartphones, the virtual objects displayed on display 206 can be manipulated by operating the display's touch panel or similar. The virtual object management system 121 does not necessarily require camera 207.

[0021] The system bus 201 controls the flow of data within the device. The Network Interface Card (NIC) 209 exchanges data with external devices via the interface 208 and networks 100-102. Note that the above computer configuration is just one example and is not limited to the configuration example in Figure 2. For example, the storage location of data and programs can be changed to ROM 204, RAM 203, HDD 205, etc., depending on their characteristics. In addition, the CPU 202 and GPU 210 execute processing based on the programs stored in HDD 205, thereby realizing the processing in the software configuration shown in Figure 3.

[0022] Next, the software configuration of the virtual object management system according to this embodiment will be explained using Figures 3 and 11.

[0023] Figure 3 illustrates a software configuration in which the functions related to the present invention are extracted from the virtual object management system according to this embodiment.

[0024] The virtual object management system 121 consists of an anchor receiving unit 311, an anchor providing unit 312, an anchor storage unit 313, and an anchor management unit 314.

[0025] When the anchor receiving unit 311 receives an anchor registration request from client terminals 131 to 133, it stores the received anchor information in the anchor storage unit 313. Similarly, when the anchor providing unit 312 receives an anchor acquisition request from client terminals 131 to 133, it searches the anchor storage unit 313 for an anchor that matches the specified criteria and returns it to the client terminals 131 to 133.

[0026] Table 1 shows an example of data stored in the anchor storage unit 313.

[0027] [Table 1]

[0028] When the anchor receiving unit 311 receives an anchor registration request from client terminals 131 to 133, it stores the corresponding anchor record in the anchor storage unit 313. The anchor ID column is a unique identification information (ID) for identifying the anchor.

[0029] The Session ID column assigns the same ID to all items within the same session. By linking multiple anchors to a single session, it's possible to present multiple anchors to the user simultaneously. The Virtual Object Data column contains 3D model data in various formats.

[0030] The features used to link and display virtual objects indicated by anchors in the present invention to the real world are managed by three pieces of information in Table 2: (1) the feature column, (2) the location information column, and (3) the sensor information column.

[0031] The feature vector column consists of three-dimensional features of the real world surrounding the anchor. The location information column represents the three-dimensional position of the virtual object in the real world. The sensor information column includes location information of the anchor (GPS coordinates), identifiers (IDs) included in signals using wireless communication functions such as beacons and Wi-Fi to which the anchor is associated.

[0032] In this invention, the features indicated by anchors are managed using three pieces of information: (1) the feature column, (2) the location information column, and (3) the sensor information column, as shown in Table 2. However, it is also possible to manage the features of each anchor using at least one or more of these, or a combination of them, and other pieces of information.

[0033] The anchor provider unit 312 can return one anchor associated with a specific anchor ID in response to an anchor acquisition request from client terminals 131 to 133, and can also return multiple anchors associated with the same session ID or the same sensor ID.

[0034] In this embodiment, priority information is further associated with the anchor ID as a parameter for controlling the relative display of virtual objects when a virtual object is provided. The priority column is a numerical value indicating the priority of each anchor, and this value can be additionally set for each anchor after anchor registration.

[0035] The anchor management unit 314 can perform operations on anchors stored in the anchor storage unit, and the anchor management unit 314 can add priority settings to anchors stored in the anchor storage unit 313. Priority can be set as a numerical value, for example, as shown in Table 1, and a smaller numerical value may be associated with a higher priority, or a larger numerical value may be associated with a higher priority.

[0036] Client terminals 131-133 consist of a virtual object data management unit 301, an anchor generation unit 302, an anchor acquisition unit 303, an anchor drawing unit 304, and a local anchor management unit 305.

[0037] The virtual object data management unit 301 stores 3D model data in various formats. The 3D data stored in the virtual object data management unit 301 are virtual objects that users can freely place, project, and display overlaid on the real world.

[0038] The anchor generation unit 302 is responsible for creating anchors based on user operations. The user can select a 3D model stored in the virtual object data management unit 301 via the anchor generation unit 302 and place the virtual object in the real world using their fingers captured by the camera 207 or by operating the touch panel on the display 206. Figures 11(a) to (h) show the images displayed on the displays 206 of the HMD-type client terminals 131 and 133, respectively. As shown in Figure 11(a), the user can manipulate the cylindrical virtual object 1102 stored in the virtual object data management unit 301 of the client terminal 131 using the method described above and place it on the desk 1101 in the real world. Subsequently, the anchor generation unit 302 analyzes the image of the surrounding area where the virtual object is placed, captured via the camera 207, extracts features, links them to the virtual object, and stores them in the local anchor management unit 305. The anchor generation unit 302 also uses the GPS sensor connected via the interface 208 to identify the anchor's position information and links it to the anchor. Furthermore, the user links the anchor and the sensor through the anchor generation unit 302. The anchor generation unit 302 transmits the anchor, which has been created as described above and stored in the local anchor management unit 305, to the anchor receiving unit 311.

[0039] The anchor acquisition unit 303 requests anchor acquisition from the anchor provision unit 312 based on sensor information connected to the interface 208, and stores the acquired anchors in the local anchor management unit 305. For example, if the anchor acquisition unit 303 detects an anchor near the current location based on a GPS signal, or a Wi-Fi or Beacon signal, it requests the anchor provision unit 312 to acquire the anchor associated with that Wi-Fi or Beacon.

[0040] The anchor drawing unit 304 compares the feature quantities contained in each anchor stored in the local anchor management unit 305 with the real-world image captured by the camera 207, and places the virtual object contained in the anchor in the area where the features match. Figure 11(b) shows how the cylindrical virtual object 1102, which the user placed on the desk 1101 on the client terminal 131 as shown in Figure 11(a), is drawn as a cylindrical virtual object 1112 on the desk 1111 with the same feature quantities on the client terminal 133.

[0041] Figure 11(c) shows how multiple anchors acquired by the anchor acquisition unit 303 are drawn as multiple virtual objects 1122-1124 on the desk 1121 with the same feature quantities. In this case, it is difficult to see virtual object A1122 because virtual object B1123 is in the foreground.

[0042] However, there are times when you want to render virtual object A1122 more visible than other virtual objects, such as when the user wants to see virtual object A1122 the most, or when the owner of desk 1121, such as a sign or advertisement, wants to show virtual object A1122 the most.

[0043] Therefore, the priority of each anchor is determined by the priority value stored in the local anchor management unit 305, and the anchor drawing unit 304 controls the drawing of virtual objects according to the priority of each anchor, drawing anchors with higher priority more clearly. Figures 11(d) to (h) show examples of anchor drawing control according to priority.

[0044] Figures 11(d) to (h) show an example where virtual object A1122 in Figure 11(c) has the highest priority, and virtual objects A1122, B1123, and C1124 are drawn according to their priority.

[0045] Figure 11(d) shows an example of rendering a high-priority virtual object in front of other virtual objects. By rendering the high-priority virtual object A1132 in front of virtual object B1133, virtual object A1132 is made more visible.

[0046] Figure 11(e) shows an example where, if there are other virtual objects whose rendering overlaps with a high-priority virtual object, only the highest-priority virtual object is rendered. By rendering only the high-priority virtual object A1142, virtual object A1142 is made more visible.

[0047] Figure 11(f) shows an example where, if there are other virtual objects whose rendering overlaps with a high-priority virtual object, the other virtual objects are rendered transparently. By rendering virtual objects B1153 and C1154, which have lower priority than virtual object A1152, transparently, virtual object A1152 is made more visible.

[0048] Figure 11(g) shows an example of switching the rendering of virtual objects according to time, rendering higher-priority virtual objects for longer periods. The timetable dialog 1163 is a table that shows the display time for each virtual object, and the rendering of virtual objects is switched according to this table. Timetable 1164 shows the display time for virtual object A, timetable 1165 shows the display time for virtual object B, and timetable 1166 shows the display time for virtual object C. The local anchor management unit 305 or the anchor rendering unit 304 sets the timetable so that the time increases according to priority. The anchor rendering unit 304 renders virtual object A for 20 seconds, virtual object B for 10 seconds, and virtual object C for 5 seconds according to the timetable. By displaying the higher-priority virtual object A for a longer period, virtual object A is made more visible. The timetable dialog 1163 may or may not be displayed on the display 206.

[0049] Figure 11(h) shows an example where virtual objects are not drawn if there are low-priority virtual objects in a specific area. The boundary surface 1174 indicates the boundary of the drawable area for anchors, and the anchor drawing unit 304 can only draw low-priority anchors in the space defined by the boundary surface 1174. Let anchor ID=a in the anchor list in Table 1 be virtual object A, anchor ID=b be virtual object B, and anchor ID=c be virtual object C. The boundary surface 1174 is the plane of position information y=20 in Table 1, and the anchor drawing unit 304 does not draw virtual objects in the space where the position information is y≦20. There are virtual object A and virtual object B for anchors with position information y≦20, and since the low-priority virtual object B is not drawn, only virtual object A is drawn. Also, although virtual object C has low priority, it is drawn because y≦20 is not present. By not drawing virtual objects in the space where the drawing overlaps with the high-priority virtual object A, virtual object A is made more visible. The interface 1174 may or may not be drawn on the display 206.

[0050] Note that Figure 11 is just one example of controlling the rendering of virtual objects according to priority, and is not limited to the example in Figure 11, as long as rendering can be controlled according to priority.

[0051] Next, we will explain the sequence of events from generating an anchor on client terminal 131 to displaying it on another client terminal 133, using Figures 4, 5, 6, 7, and 8.

[0052] Figure 4 shows the sequence from when an anchor generated by client terminal 131 is registered with the virtual object management system 121, to when client terminal 133 retrieves and displays the anchor stored in the virtual object management system 121.

[0053] First, the sequence for registering an anchor by operating the client terminal 131 will be explained in S401 to S403 in Figure 4. The user operates the client terminal 131 and generates an anchor by placing a virtual object stored in the virtual object data management unit 301 via the anchor generation unit 302, and saves it in the local anchor management unit 305 (S401). The anchor generation unit 302 sends an anchor registration request for the generated anchor to the anchor receiving unit 311 (S402). The anchor receiving unit 311, upon receiving the anchor registration request, saves the received anchor in the anchor storage unit 313, and then returns the saving result to the anchor generation unit 302 (S403). If multiple anchors are to be registered with the same session ID, steps S401 to S403 are repeated (S404).

[0054] Using Figure 5, the detailed flow of the anchor generation process S401 performed by the anchor generation unit 302 in Figure 4 will be explained. The anchor generation unit 302 places a virtual object in the real world space based on user operation and determines its position and orientation (S502). Next, in S503, the anchor generation unit 302 takes a picture of the surroundings through the camera 207 and acquires three-dimensional feature points of the space. If the anchor generation unit 302 determines in S504 that it has not collected enough feature points, it performs the feature point acquisition again in S503. If the anchor generation unit 302 determines in S504 that it has collected enough feature points, it asks the user to set an expiration date and other necessary properties for the anchor in S505, and terminates the process in S506 if sensor information is not set for the anchor. If sensor information is set for the anchor in S506, the anchor generation unit 302 asks the user to set the appropriate settings according to the type of sensor (S507), and then terminates the process. For example, as shown in the data with anchor ID=a in Table 1, the anchor generation unit 302 associates the Beacon with id=123 with the anchor information in S507 according to the user's operation.

[0055] The above is the sequence for operating client terminal 131 to register the anchor in the virtual object management system.

[0056] Next, we will explain the sequence in S405 to S406 in Figure 4 for setting a priority for the anchors stored in the anchor storage unit 313.

[0057] The user operates on the virtual object management system 121 and adds priority settings to the anchors stored in the anchor storage unit 313 via the anchor management unit 314 (S405). If priorities are to be set for multiple anchors, S405 is repeated (S406). Figure 6 illustrates the detailed flow of anchor priority setting S405, which sets priorities for the anchors stored in the anchor storage unit 313 in Figure 4.

[0058] The user selects an anchor stored in the anchor storage unit 313 through an operation on the virtual object management system 121 (S602). In S603, the anchor management unit 314 searches for anchors related to the anchor selected by the user. Related anchors here refer to anchors that match or are similar in any of the session ID, feature quantity, or sensor information in Table 1. In S604, if there are anchors related to the anchor selected by the user among the anchors stored in the anchor storage unit 313, the list of related anchors is created in S605. In S605, the user sets a priority for each anchor in the list of anchors, and sets a priority for the anchors stored in the anchor storage unit 313 via the anchor management unit (S606). Using Table 1 as an example, if the user selects anchor ID=a in S602, the anchor with anchor ID=b and anchor with anchor ID=c, which have matching sensor information, are searched for in S603. In S605, the user lists anchors with anchor IDs a, b, and c, and in S606, sets the priority of anchor with anchor ID = a to 1, the priority of anchor with anchor ID = b to 2, and the priority of anchor with anchor ID = c to 3.

[0059] Next, we will explain the sequence in S407 to S416 in Figure 4 in which the user operates the client terminal 133 to acquire and draw an anchor.

[0060] The anchor acquisition unit 303 acquires sensor information in S407. If sensor information cannot be acquired, the anchor acquisition unit 303 repeats S407 (S408). Sensor information is acquired by the anchor acquisition unit 303 via a sensor that detects Bluetooth signals connected to the client terminal 133 via interface 208. For example, if a signal from Beacon terminal with id=123 is detected, the anchor acquisition unit 303 sends a search request to the anchor provision unit 312 for the anchor associated with Beacon terminal with id=123 (S409). The anchor provision unit 312 searches the anchor storage unit 313 for the anchor associated with Beacon terminal with id=123. Since there are two anchors associated with Beacon with id=123, anchor ID=a and b, the anchors with anchor ID=a and b are returned to the anchor acquisition unit 303 (S411). Next, the anchor acquisition unit 303 stores the anchors returned by the anchor provision unit 312 in the local anchor management unit 305 (S412), and then sends a search request to the anchor provision unit 312 for each anchor to find anchors in the same session (S413). That is, the anchor acquisition unit 303 sends a search request to the anchor provision unit 312 for the same anchor as session ID 111, which is anchor ID a (S413). The anchor provision unit 312 searches the anchor storage unit 313 for the anchor with session ID 111 (S414) and returns it to the anchor acquisition unit 303 (S415). That is, the anchor with session ID 111 has anchor ID d, so the anchor provision unit 312 returns this to the anchor acquisition unit 303 (S415). Next, the anchor acquisition unit 303 stores the anchors returned by the anchor provision unit 312 in the local anchor management unit 305 (S416). Finally, the anchor drawing unit 304 performs anchor drawing processing on the anchors stored in the local anchor management unit 305 (S417).

[0061] Here, the detailed flow of the anchor search process S410 linked to the sensor and the anchor search process S414 linked to the same session, performed by the anchor provisioning unit 312, will be explained using the flowchart in Figure 6.

[0062] Figure 7(a) is a flowchart of the anchor search process S410 performed by the anchor provision unit 312 to search for an anchor associated with a sensor. The anchor provision unit 312 checks whether an anchor matching the sensor included in the anchor acquisition request from the anchor acquisition unit 303 exists in Table 1, which is still stored in the anchor storage unit 313 (S702). If no anchor exists in S702, the anchor provision unit 312 terminates the process. If an anchor exists in S702, the anchor provision unit 312 acquires the anchor from the anchor storage unit 313 (S703). The anchor provision unit 312 stores the acquired anchor in the anchor list in S703 (S704) and returns to S702. Figure 7(b) is a flowchart of the anchor search process S414 performed by the anchor provision unit 312 to search for an anchor associated with the same session. The anchor provisioning unit 312 checks whether an anchor matching the session ID included in the anchor acquisition request from the anchor acquisition unit 303 exists in Table 1, which is still stored in the anchor storage unit 313 (S712). If no anchor exists in S712, the anchor provisioning unit 312 terminates processing. If an anchor exists in S712, the anchor provisioning unit 312 acquires the anchor from the anchor storage unit 313 (S713). The anchor provisioning unit 312 holds the anchor acquired in S713 in the anchor list (S714) and returns to S712. In both the anchor search process S410 associated with a sensor and the anchor search process S414 associated with the same session, the only anchors that the anchor provisioning unit 312 returns to the anchor acquisition unit 303 are those held in the anchor list in S704 and S714.

[0063] Figure 8 is a flowchart of the anchor drawing process S417 performed by the client terminal 133. The anchor drawing unit 304 acquires feature quantities of the real-world region from the video captured through the camera 207 (S802) and checks whether they match the feature quantities for each anchor stored in the local anchor management unit 305 (S803). If there is no match in S803, the anchor drawing unit 304 terminates the anchor drawing process S417. If there is a match in S803, the anchor drawing unit 304 acquires the priority for each matched anchor (S804) and draws virtual objects according to the priority of each anchor (S805). The drawing of virtual objects according to the priority of each anchor in S805 is a drawing method as shown in Figures 11(d) to (h) as an example.

[0064] By using the methods described above, it is possible to prioritize virtual objects and control their rendering according to their priority, thereby making high-priority virtual objects more visible. This allows users to easily see the virtual objects they want to show, or to easily see the virtual objects they want to view. [Examples]

[0065] In Example 1, instead of rendering virtual objects according to predetermined priorities, there may be cases where the priority is to be determined by user feedback. For example, users can assign review scores and high ratings to anchors, and anchors with a high average review score or a large number of high ratings may be given higher priority. In this embodiment, in addition to the virtual object, an evaluation dialog is displayed on display 206 (described later). When the user enters an evaluation in the evaluation dialog, client terminals 131 to 133 send the evaluation to the virtual object management system.

[0066] Since this embodiment overlaps considerably with Embodiment 1, only the differences from Embodiment 1 will be explained.

[0067] Table 2 shows an example of data stored in the anchor storage unit 313.

[0068] [Table 2]

[0069] In this embodiment, when a virtual object is provided to an anchor ID, an evaluation score, which represents numerical information of the evaluation given to the virtual object, is further associated with and managed in an evaluation score column as a parameter for controlling the relative display of the virtual object in relation to other virtual objects. In Table 2, the definitions of the columns are the same as those of the corresponding columns in Table 1, except for the evaluation score column. The evaluation score column in Table 2 stores the evaluation score entered by the user.

[0070] Although not shown in Table 2, it is also possible to manage both the priority column and the evaluation score column. In that case, it becomes possible to control the display, for example, by prioritizing the display of virtual objects with higher evaluation scores if they have the same priority. Furthermore, the values ​​for these columns can be set arbitrarily, and if a value is missing for one column, the information from the other column can be used for display control.

[0071] Figure 12 shows the images displayed on the displays 206 of client terminals 131 to 133 in this embodiment.

[0072] Figure 12(a) shows an example of adding evaluation points to each anchor. The evaluation dialog 1205 is a dialog for entering evaluations for virtual objects 1202 to 1204. Input window 1206 is for virtual object A, input window 1207 is for virtual object B, and input window 1208 is for virtual object C. By selecting "Good" in input windows 1206 to 1208, evaluation points are added to each virtual object.

[0073] Figure 12(b) shows an example of inputting evaluation points for each anchor. The evaluation dialog 1215 is a dialog for inputting evaluations for virtual objects 1212 to 1214. Input window 1216 is for virtual object A, input window 1217 is for virtual object B, and input window 1218 is for virtual object C. By selecting the evaluation points in input windows 1216 to 1218, the selected evaluation points are entered for each virtual object.

[0074] Note that Figure 12 is just one example of how to assign evaluation points to anchors, and is not limited to the example in Figure 12 as long as evaluation can be performed on the anchors.

[0075] Figure 9(a) is a flowchart of the anchor drawing process performed by client terminals 131-133 in Embodiment 2. The anchor drawing unit 304 acquires feature quantities of the real-world region from the video captured by the camera 207 (S902) and checks whether they match the feature quantities for each anchor stored in the local anchor management unit 305 (S903). If they do not match in S903, the anchor drawing unit 304 terminates the anchor drawing process. If they do match in S903, the anchor drawing unit 304 acquires evaluation points for each matched anchor (S904) and draws virtual objects according to the priority of each anchor (S905). The drawing of virtual objects according to the priority of each anchor in S905 is a drawing method as shown in Figures 11(d)-(h) as an example. After that, the user inputs evaluation points for each anchor via the interface 208 (S906), and client terminal 133 sends them to the virtual object management system (S908).

[0076] Figure 9(b) is a flowchart of the evaluation score update process performed by the virtual object management system 121 in Example 2. The virtual object management system 121 receives evaluation scores for each anchor from client terminals 131 to 133 (S912) and updates the evaluation scores for each anchor stored in the anchor storage unit 313. This update includes adding evaluation scores and calculating the average score.

[0077] In this embodiment, the evaluation score was entered by the user as shown in the example in Figure 12. However, the evaluation score could also be automatically obtained based on the number of times the virtual object was drawn or the number of times the user directed their gaze towards the virtual object.

[0078] This embodiment allows user feedback to be sent to the virtual object management system 121, enabling the system to determine priorities based on user feedback and control the rendering of virtual objects. [Examples]

[0079] In Example 1, there may be cases where the user wants to draw anchors of their preference, regardless of priority. By preparing anchor properties such as author, creation date, and tags, and using them selectively, the user can retrieve only the anchors they prefer. For example, by adding the anchor's author to the anchor properties, the user can retrieve only anchors created by their preferred authors.

[0080] Since this embodiment overlaps considerably with Embodiment 1, only the differences from Embodiment 1 will be explained.

[0081] Table 3 shows an example of property data stored in the anchor storage unit 313.

[0082] [Table 3]

[0083] In this embodiment, property data is further associated with the anchor ID and managed as a parameter for controlling the relative display of virtual objects when a virtual object is provided. Although omitted in Table 3, the session ID column, virtual object data column, feature column, location information column, and sensor information column included in Tables 1 and 2 are managed in each record. An evaluation score column may also be included.

[0084] Although not shown in Table 2, it is also possible to manage both the priority column and the evaluation score column. In that case, it becomes possible to control the display, for example, by prioritizing the display of virtual objects with higher evaluation scores if they have the same priority. Furthermore, the values ​​for these columns can be set arbitrarily, and if a value is missing for one column, the information from the other column can be used for display control.

[0085] Each anchor possesses the data from Table 1 in addition to the property data from Table 3.

[0086] In S409 and S413, the user specifies property conditions for the search request, and in S411 and S415, the virtual object management system sends only anchors containing the user's specified properties to the client terminal 133. This allows the user to view only their preferred anchors.

[0087] Figure 10(a) is a flowchart of the anchor search process S410 associated with a sensor, performed by the anchor providing unit 312 in Embodiment 3. The anchor providing unit 312 checks whether an anchor matching the sensor included in the anchor acquisition request from the anchor acquisition unit 303 exists in Table 1, which is still stored in the anchor storage unit 313 (S1002). If no anchor exists in S1002, the anchor providing unit 312 terminates the process. If an anchor exists in S1002, the anchor providing unit 312 acquires the anchor from the anchor storage unit 313 (S1003). If the acquired anchor matches the property conditions specified by the user (S1004), the anchor providing unit 312 stores the acquired anchor in the anchor list (S1005) and returns to S1002. The property conditions specified by the user include exact match, partial match, and comparison of time values.

[0088] Figure 10(b) is a flowchart of the anchor search process S414 performed by the anchor provision unit 312 in Embodiment 3, which is linked to the same session. The anchor provision unit 312 checks whether an anchor matching the session ID included in the anchor acquisition request from the anchor acquisition unit 303 exists in Table 1, which is still stored in the anchor storage unit 313 (S1012). In S1012, if no anchor exists, the anchor provision unit 312 terminates the process. If an anchor exists in S1012, the anchor provision unit 312 acquires the anchor from the anchor storage unit 313 (S1013). If the acquired anchor matches the property conditions specified by the user (S1014), the anchor provision unit 312 stores the acquired anchor in S1013 in the anchor list (S1015) and returns to S1012. In both cases of the anchor search process S410 associated with a sensor and the anchor search process S414 associated with the same session, the anchors that the anchor providing unit 312 returns to the anchor acquisition unit 303 are only those anchors that are held in the anchor list in S1005 and S1015.

[0089] Note that the properties that can be set for an anchor are not limited to those listed in Table 3.

[0090] This embodiment provides properties for anchors, allowing users to selectively use them to display only their preferred anchors.

[0091] (Other examples) The present invention also includes apparatuses, systems, and methods configured by appropriately combining the embodiments described above.

[0092] Here, the present invention is a device or system that is the main body for executing one or more software programs that realize the functions of the embodiments described above. Furthermore, a method for realizing the embodiments described above that are executed on that device or system is also part of the present invention. The program is supplied to the system or device via a network or various storage media, and the program is read into one or more memories by one or more computers (CPU, MPU, etc.) of the system or device and executed. In other words, as part of the present invention, the program itself or various storage media that can be read by the computer storing the program are also included. Furthermore, the present invention can also be realized by a circuit (e.g., ASIC) that realizes the functions of the embodiments described above. [Explanation of Symbols]

[0093] 121 Virtual Object Management System 131 Client terminal (head-mounted display) 132 Client terminals (smartphones)

Claims

1. A system for managing virtual objects, A management means for managing, in association with identification information, real-world feature quantities for displaying virtual objects linked to the real world, and parameters for controlling the relative display of the virtual object in relation to other virtual objects when the virtual object is provided. The parameters include numerical information representing an evaluation performed on a virtual object, and the system includes an update means for dynamically updating the numerical evaluation information based on feedback that includes at least one of the number of times the virtual object has been displayed in relation to the real world and user input to the virtual object. The system is characterized in that, in addition to the numerical information of the evaluation, the parameters include priority information for determining whether a virtual object should be given priority over other virtual objects when displaying a virtual object in the real world.

2. The system according to claim 1, characterized in that when multiple virtual objects having the same priority information become objects that are linked to and displayed in the real world, relative display control is performed among the multiple virtual objects using numerical information of the evaluation.

3. The system includes a terminal capable of projecting virtual objects into the real world. In response to a request using identification information from a terminal and at least one of the features in the real world, the management means responds with information about the virtual object managed by the identification information and at least one of the features. The system according to claim 1 or 2, characterized in that the virtual object is projected into the real world by the terminal based on the parameters and the information received.

4. The system according to claim 3, characterized in that the terminal includes a head-mounted display.

5. A method in a system for managing virtual objects, The system includes a management process that manages real-world feature quantities for displaying virtual objects in the real world, and parameters for controlling the relative display of the virtual object in relation to other virtual objects when the virtual object is provided, in association with identification information. The parameters include numerical information representing an evaluation performed on a virtual object, and the update process dynamically updates the numerical evaluation information based on feedback including at least one of the number of times the virtual object is displayed in the real world and user input to the virtual object. The method is characterized in that the parameters include, in addition to the numerical information of the evaluation, priority information for determining whether a virtual object should be given priority over other virtual objects when displaying a virtual object in the real world.

6. A terminal that can project virtual objects into the real world, A first transmission means that transmits a request using identification information and at least one of real-world features to a system that manages virtual objects, The system includes a receiving means for receiving information about virtual objects in response to the request. Based on the received information, projection means for projecting the virtual object into the real world, The system includes a second transmission means for transmitting user input that can serve as feedback to the virtual object to the system, The projection of the received virtual object is controlled based on parameters for controlling the relative display of the virtual object in relation to other virtual objects managed by the system in association with that virtual object. The aforementioned parameters include numerical information representing an evaluation performed on a virtual object, and the system dynamically updates this numerical evaluation information based on user input feedback to the virtual object transmitted from the terminal. The terminal is characterized in that the parameters further include priority information for determining whether a virtual object should be prioritized over other virtual objects when displaying a virtual object in the real world, in addition to the numerical information of the evaluation.

7. The above requirement includes the above feature quantity, The terminal according to claim 6, characterized in that the feature quantity includes an identifier included in a signal using a wireless communication function available to the terminal.

8. A method for a terminal that can project virtual objects into the real world, A first transmission step involves sending a request using identification information and at least one of real-world features to a system that manages virtual objects, A receiving step of receiving information about a virtual object from the aforementioned system in response to the request, Based on the received information, a projection step is performed to project the virtual object into the real world, The system includes a second transmission step of sending user input that can serve as feedback to the virtual object to the system, The projection of the received virtual object is controlled based on parameters for controlling the relative display of the virtual object in relation to other virtual objects managed by the system in association with that virtual object. The aforementioned parameters include numerical information representing an evaluation performed on a virtual object, and the system dynamically updates this numerical evaluation information based on user input feedback to the virtual object transmitted from the terminal. The method is characterized in that the parameters further include priority information for determining whether a virtual object should be prioritized over other virtual objects when displaying a virtual object in the real world, in addition to the numerical information of the evaluation.

9. Computers A first transmission means that transmits a request using identification information and at least one of real-world features to a system that manages virtual objects, A receiving means that receives information about virtual objects from the aforementioned system in response to the request, Based on the received information, projection means for projecting the virtual object into the real world, A program for functioning as a second transmission means that transmits user input that can serve as feedback to the virtual object to the system, The projection of the received virtual object is controlled based on parameters for controlling the relative display of the virtual object in relation to other virtual objects managed by the system in association with that virtual object. The parameters include numerical information representing an evaluation performed on a virtual object, and the system dynamically updates this numerical evaluation information based on user input feedback to the virtual object transmitted by the second transmission means. The program is characterized in that the parameters further include priority information for determining whether a virtual object should be prioritized over other virtual objects when displaying a virtual object in the real world, in addition to the numerical information of the evaluation.

Images