Coarse relocation using signal fingerprints and session-specific identifiers

Abstract

A first display device and method are provided for accelerating a coarse relocation process of a first display device by generating a session-specific identifier and sending it to a second display device, the second display device transmits the identifier to a cloud service, and the cloud service returns at least a portion of a map data set corresponding to the identifier. The returned map data set is then stitched into the local map data of the first display device to create an integrated map, which is used to render one or more holograms. The first display device may include a processor, a memory operably coupled to the processor, and a fingerprint program stored in the memory and executed by the processor.

Description

Background Art

[0001] 6-DoF tracking, also known as six degrees of freedom tracking, is a method used by a device (e.g., a mixed reality head mounted device (HMD), a robot, a smartphone, etc.) to locate itself or determine its position relative to the surrounding physical environment using sensors (e.g., cameras, inertial measurement units, etc.). When a device is turned on in a new environment, the device initially performs a coarse relocalization step where a rough vicinity of the device's current position is determined before the device performs a more fine-grained localization step to determine a more precise position of itself relative to its surrounding physical environment. For example, a mixed reality HMD or smartphone can use this localization understanding to place holograms or digital content so that it appears to be world-locked to a certain position in the physical world, and a robot can use this localization understanding to navigate itself relative to its surroundings.

[0002] The ability for a device to quickly orient itself or determine its own position relative to the three-dimensional coordinate space of a physical environment by efficiently loading appropriate map datasets before performing a more fine-grained localization step to accelerate a coarse relocalization step is increasingly valuable. The conventional approach to self-orientation is to load all map datasets of the physical environment into memory before the device begins to orient itself. However, this conventional approach may be associated with practical disadvantages including large memory requirements and long loading times. Summary of the invention

[0003] To solve these problems, a first display device and method are provided to simplify and accelerate the rough repositioning process of the first display device. The first display device may include a processor, a memory operably coupled to the processor, and a client fingerprint program stored in the memory and executed by the processor.

[0004] The client fingerprint program can be configured to receive a session-specific identifier from a second device, the session-specific identifier corresponding to a map data set; transmit the session-specific identifier to a remote system; receive a map data set corresponding to the session-specific identifier from the remote system; and stitch the map data set into local map data of the first display device to create an integrated map.

[0005] This summary is provided to introduce a series of concepts that are further described in the detailed description below in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1A computing device in the form of a head mounted display device according to an example embodiment of the present disclosure is shown.

[0007] Figure 2 In the physical environment Figure 1 Schematic diagram of an example predictive correction algorithm for simultaneous localization and mapping using a computing device.

[0008] Figure 3 is a time series view of the data, which is composed of Figure 1 The various sensors of the computing device capture and process to generate a series of key frames linked by a pose graph.

[0009] Figure 4A is a schematic diagram of a computing device for generating a current fingerprint, comparing the current fingerprint with fingerprint data, and sending a request for a matching map data set according to a first embodiment of the present disclosure.

[0010] Figure 4B It is a depiction Figure 4A A schematic diagram of a computing device of one example of signals and signal sources is shown in FIG.

[0011] Figure 4C yes Figure 4A Detailed schematic diagram of an example of a fingerprint, fingerprint data, and corresponding map dataset shown in .

[0012] Figure 5 is a schematic diagram of a computing device for generating a current fingerprint, comparing the current fingerprint with fingerprint data, and sending a request for a matching map data set according to a second embodiment of the present disclosure.

[0013] Figure 6 Shown is a diagram of a person wearing a mobile phone in a restaurant according to an example of the present disclosure. Figure 1 Users of head mounted display devices.

[0014] Figure 7 is a schematic diagram of an embodiment of map data including a corresponding map data set according to the second embodiment of the present disclosure.

[0015] Fig. 8A , Figure 8B and Figure 8C is a flowchart of a method of generating a current fingerprint and retrieving a corresponding map data set according to three examples of the present disclosure.

[0016] Fig. 9 A computing system according to an embodiment of the present disclosure is shown.

[0017] Fig.10is a schematic diagram of a computing device for obtaining a session-specific identifier and sending a request for a matching map dataset using the session-specific identifier according to a third embodiment of the present disclosure.

[0018] Fig.11A , Fig. 11B , Fig. 11C and Fig.11D is a flowchart of a method of obtaining a session-specific identifier and retrieving a corresponding map dataset according to four examples of the present disclosure. DETAILED DESCRIPTION

[0019] Figure 1 A head mounted display device 10 embodiment of a computing device 10 according to one embodiment of the present disclosure is illustrated, which has been conceived to address the issues discussed above. As shown, the computing device 10 includes a processor 12, a volatile storage device 14, a non-volatile storage device 16, a camera 18, a display 20, and an active depth camera 21. The processor 12 is configured to execute software programs stored in the non-volatile storage device 16 using portions of the volatile storage device 14 to perform the various functions described herein. In one example, the processor 12, the volatile storage device 14, and the non-volatile storage device 16 may be included in a system-on-chip configuration included in the head mounted display device 10. It will be appreciated that the computing device 10 may also take the form of other types of mobile computing devices, such as, for example, a smart phone device, a tablet device, a laptop computer, a machine vision processing unit for an autonomous vehicle, a robot, a drone, or other types of autonomous devices, and the like. In the system described herein, a device in the form of a computing device 10 may be utilized as a display device, as discussed below. Figure 2-8C As shown in the figure.

[0020] The display 20 is configured to be at least partially perspective and includes right and left display areas 120A, 120B configured to display different images to each eye of the user. By controlling the images displayed on these right and left display areas 120A, 120B, the hologram 50 can be displayed in a manner that appears to the user's eyes to be at a distance from the user positioned in the physical environment 9. As used herein, a hologram is an image formed by displaying left and right images on corresponding left and right near-eye displays, which appears to be positioned at a distance from the user due to a stereoscopic effect. Typically, the hologram is anchored to a map of the physical environment by virtual anchors 64, which are placed within the map according to their coordinates and are typically positioned on visual features that are recognizable by image processing of captured images of the environment. These anchors are world-locked, and the hologram is configured to be displayed at a position calculated relative to the anchors. Typically, holograms are defined so as to be displayed within a threshold distance from their corresponding anchors, such as 3 meters, but other threshold distances may also be used.

[0021] exist Figure 1 In the configuration illustrated in , multiple cameras 18 are provided on computing device 10 and are configured to capture images of the surrounding physical environment of computing device 10. In one embodiment, four cameras 18 are provided, but the exact number of cameras 18 may vary. In some configurations, the raw images from cameras 18 may be stitched together with perspective correction to form a 360-degree view of the physical environment. Typically, cameras 18 are visible light cameras. Passive stereo depth estimation techniques may be used to compare images from two or more cameras 18 to provide depth estimates.

[0022] In addition to the visible light camera 18, a depth camera 21 may be provided that uses an active non-visible light illuminator 23 and a non-visible light sensor 22 to emit light in a phased or gated manner and use time-of-flight techniques to estimate depth, or emits light in a structured pattern and uses structured light techniques to estimate depth.

[0023] Computing device 10 also typically includes a six-degree-of-freedom inertial motion unit 19, which includes an accelerometer, a gyroscope, and possibly a magnetometer, configured to measure the positioning of the computing device in six degrees of freedom (ie, x, y, z, pitch, roll, and yaw).

[0024] The data captured by visible light camera 18, depth camera 21, and inertial motion unit 19 can be used to perform simultaneous localization and mapping (SLAM) within physical environment 9, thereby generating a map of the physical environment including a mesh of reconstructed surfaces, and positioning computing device 10 within the map of physical environment 9. The position of computing device 10 is calculated in six degrees of freedom, which is important for displaying world-locked holograms 50 on at least partially see-through display 20. Without accurate identification of the position and orientation of computing device 10, holograms 50 displayed on display 20 may appear to move or vibrate slightly relative to the physical environment when they should remain in place in a world-locked position. This data can also be used to reposition computing device 10 when computing device 10 is turned on, a process that involves determining its position within the map of the physical environment and loading appropriate data from non-volatile memory to volatile memory to display holograms 50 located within the physical environment.

[0025] The IMU 19 measures the positioning and orientation of the computing device 10 in six degrees of freedom, and also measures acceleration and rotational speed. These values ​​can be recorded as a posture map to help track the display device 10. Therefore, even when there are few visual cues to achieve visual tracking, such as in a poorly lit area or a textureless environment, the accelerometer and gyroscope can still achieve spatial tracking by the display device 10 without visual tracking. Other components in the display device 10 may include, but are not limited to, speakers, microphones, gravity sensors, Wi-Fi sensors, temperature sensors, touch sensors, biometric sensors, other image sensors, eye gaze detection systems, energy storage components (such as batteries), communication facilities, etc.

[0026] Figure 2 A general overview of a possible SLAM algorithm for simultaneous localization and mapping by a computing device 10 is illustrated. According to the SLAM algorithm, a predictive correction model 28 is typically applied, which may be, for example, a Kalman filter. In each frame (i.e., time step), the predicted pose 29 in the next frame is calculated by the integration engine 27 based on the ranging data 119A from the IMU 19, and then the correction to the predicted pose 29 is calculated by the predictive correction model (e.g., Kalman filter) 28 based on the estimated and observed positioning of the features sensed in the physical environment 9 by sensors such as the camera 18 and the depth camera 21, and finally the predicted pose 29 is updated and the updated pose 35 is fed back to the integration engine 27 for calculating the predicted pose 29 at subsequent time steps. Additional sensors such as lidar, microphone arrays, etc. may also be used to obtain observations of features in the environment, which may be used to calculate corrections and updated poses of the predictive correction model 28.

[0027] The feature descriptors 111A describe features such as edges, corners, and other patterns detectable by image processing techniques, and the feature descriptors 111A are pre-stored in the feature library 11 in the non-volatile storage device 16. In real time, the image 118A and the depth image 121A are captured by the camera 18 and the depth camera 21, respectively, and processed by the feature matching engine 13 executed by the processor 12 to detect whether there are features matching the feature descriptors 111A in the captured images 118A, 121A by finding areas in the captured images that match the pre-stored feature descriptors 111A. Some detected features may have anchors 64 associated with them, because anchors are often located on such visually identifiable features in the environment. For each detected feature, the location (e.g., coordinate area) and type of the feature are stored as observation data 17 associated with each frame. It will be appreciated that dozens or hundreds of such features may be identified in the image, and the collection of these feature observations 17 may be informally referred to as a point cloud of features detected in the image. In addition, for at least the detected features selected in the image, a patch 15 is obtained from the image around the selected detected features and stored in the memory for later recall. The patch 15 is typically a two-dimensional array of pixels or voxels from the area of ​​the captured image and can be used in future positioning steps - when the computing device 10 captures an image of the selected detected features from another angle, by performing perspective correction on the patch to determine whether (and where) the selected detected features in the perspective-corrected patch appear in the subsequent image. The physical manifestations of these selected detected features in the physical environment are also referred to as anchors. The features matched with the feature descriptors 111A, observations 17, and patches 15 for each frame are collectively referred to as feature matching data 113A. The feature matching data 113A typically does not include a depth image 121A or RGB image data 118A. As discussed below, the feature matching data 113A can be stored in non-volatile or volatile memory for certain frames (referred to as key frames). The pose graph 80, feature matching data 113A, surface reconstruction data 82, and keyframes 84 linked by the pose graph 80 may be collectively referred to as map data 86. As the computing device 10 moves throughout the physical environment 9, it maps the environment and stores its aggregated knowledge of the environment as map data 86. As will be discussed below, sharing a portion of this aggregated map data with another device, either directly or through an intermediary device (such as a server), may enable the other device to more quickly and accurately locate themselves in the physical environment, saving time and processing power for the other device.

[0028] Figure 3An example schematic diagram of data collected by the camera 18 , depth camera 21 , IMU 19 , and GPS receiver 25 in the sensor suite of the computing device 10 and processed by the processor 12 of the computing device 10 is shown. Figure 3 The data points in are shown as arranged along a time axis, with later times being displayed to the right of earlier times. Typically, data is collected periodically from each sensor at a certain sampling rate. For ease of illustration, the sampling rates of the various sensors are depicted in the figure as quantized at the same sampling rate (although the sampling rates may be different in practice), and each time step at which data is sampled from the sensor is referred to as a frame 546. Thus, in the first frame 546A, the second frame 546B, and each frame thereafter, image data 118A is collected by camera 18 (four cameras in this example), depth camera data 121A is collected using depth camera 21, ranging data 119A is collected by IMU 19, and GPS data 125A is collected by GPS receiver 25.

[0029] The processor 12 can generate a pose graph 80, feature matching data 113A, and surface reconstruction data 82 based on the sensor suite inputs including image data 118A, depth image data 121A, range data 119A, and GPS data 125A using the simultaneous localization and mapping (SLAM) techniques discussed above. The pose graph 80 is a directed graph with nodes that are a series of updated poses 35 detected over time. A pose is typically a unit vector with an origin at a predetermined location (x, y, and z) and extending along a predetermined orientation (pitch, yaw, and roll) in physical space, and is represented as a vector with respect to the position of the object. Figure 2 The calculations are performed as described. The origin corresponds to a predetermined fixed location on the computing device, such as equidistant between the left and right displays, and the unit vectors extend outward from the displays in a fixed pose in the direction of the user's gaze. The processor 12 typically generates the gesture graph 80 in each frame, but alternatively the gesture graph 80 may be generated less frequently, such as every other frame. The surface reconstruction data 82 typically includes a mesh of surfaces detected in the physical environment, which are detected using stereo analysis of the depth image data 121A and / or the image data 118A. As discussed above, the feature data 113A typically includes one or more patches, observations, and / or feature descriptors associated with each gesture of the gesture graph 80.

[0030] The processor 12 may generate keyframe data including a plurality of keyframes 84 and store them in a memory. Each keyframe 84 includes a pose of the pose graph 80, so the keyframes 84 are linked by the pose graph 80. Each keyframe 84 also includes feature matching data 113A, which includes one or more (usually multiple) observations 17, features 111A matching feature descriptors, and associated tiles 15 for the frame. The keyframe data may also include metadata, which may include, for example, GPS data 125A applicable to the frame, ranging data 119A, hardware data (e.g., camera lens type), ambient temperature, etc. The keyframes 84 may be generated at regular intervals within a series of consecutive frames, such as every other frame or every 10 frames, etc. Alternatively, the keyframes 84 may be generated at predetermined spatial intervals (such as every 1 or 2 meters) as the computing device 10 moves through the physical environment 9.

[0031] Figure 4A is a schematic diagram of a display device that generates a current signal fingerprint (hereinafter referred to as the "current fingerprint") and interacts with a server computing device according to the first embodiment of the present disclosure. As explained in more detail below, the server computing device 200 can be used to store signal fingerprint data 54 (hereinafter referred to as "fingerprint data") and a map data set 58. The computing device 200 can take the form of a server, a networked computer, a game console, a mobile communication device, a desktop computer, a laptop computer, a tablet computer, a set-top box (such as a cable box, a satellite box), or any other type of suitable computing device. In some examples, the computing device 200 may include an embedded system within a larger electronic or mechanical device or system. In other examples, the computing device 200 may be the same as the display device 30. Additional details about the components and computing aspects of the computing device 200 are described below with respect to Fig. 9 Describe in more detail.

[0032] The server computing device 200 may include a server fingerprint program 214 that may be stored in a mass storage device 218 of the computing device 200. The server fingerprint program 214 may be loaded into the memory 220 and executed by the processor 260 of the server computing device 200 to perform one or more methods and processes for sending fingerprint data 54 to the display device 30 in response to receiving a fingerprint request 52 from the display device 30, and retrieving one or more corresponding map data sets and sending the corresponding map data sets to the display device 30 in response to receiving a map data request 48 from the display device 30, as described in more detail below. The server computing device 200 may be configured with a wireless transceiver 230 that wirelessly communicates with the display device 30 to receive the fingerprint request 52 and the map data request 48 from the display device 30, and transmits the fingerprint data 54 and the matching map data set 66 to the display device 30. It will be appreciated that the matching map data set 66 is one or more map data sets corresponding to one or more closest matching fingerprints. The type of map data used in the present disclosure is not particularly limited, and can be understood as any data set that relates points in a three-dimensional coordinate space in a physical environment to information that helps orient and locate a display device in the three-dimensional space. Figure 6 and Figure 7 One possible embodiment of the map data is described in more detail and includes keyframes 60, a pose graph 62, and anchors in the form of anchor data 64. The format of the anchor data is not particularly limited and may be encoded as a unique string identifier identifying the anchor, and coordinates identifying the location of the anchor within the map.

[0033] The server computing device 200 may be communicatively coupled to one or more other devices via a wired or wireless connection to a network. In some examples, the network may take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and may include the Internet. Figure 4A In some examples, the server computing device 200 is communicatively coupled to the display device 30 via one or more networks. In other examples, the server computing device 200 may be operably connected to additional devices. The display 20 of the display device 30 may display one or more holograms at the virtual location anchor 64 from the vantage point of the display device 30. In some examples, the virtual position of the virtual location anchor 64 may be world-locked to a virtual location that is fixed in a three-dimensional coordinate space overlaid on the real-world three-dimensional environment. In other examples, the virtual position of the virtual location anchor 64 may be world-locked to a virtual location relative to an object in the real-world three-dimensional environment.

[0034] One or more processors of the display device 30 execute the client fingerprint program 38. The display device 30 stores local map data 36 in local memory. The local map data 36 may include recorded rotational and translational motions of the display device 30 tracked by the visual sensors and / or inertial measurement sensors 18 in the display device 30. The display device 30 may be communicatively coupled to one or more other devices via a wired connection or a wireless connection to a network, but the communicative coupling is not particularly limited and may cover any form of data flow, including Wi-Fi, email, external data storage devices, cloud storage devices, etc. In some examples, the network may take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and may include the Internet.

[0035] The sensor 18 in the display device receives a signal 33 from one or more signal sources 32. The signal source 32 may include at least one of a radio frequency source, an electromagnetic source, a light source, an acoustic source, and a heat source. The signal 33 may include at least one of an electromagnetic signal, gravity, a magnetic field, and temperature. The electromagnetic signal may include at least one of an FM signal, a GPS signal, a cellular signal, and a Wi-Fi signal. Figure 4B , the signal source may include one or more wireless access points 132A-C broadcasting beacon frames 133A-C. Each beacon frame 133C includes an Ethernet header 135C, including a MAC address 137C. The body of the beacon frame 133C may include an SSID (not shown). The beacon frame 133C provides information about the type, brand or model of the originating wireless access point 132C.

[0036] return Figure 4A In response to the sensor 18 receiving the signals 33A-C from the signal sources 32A-C, the display device 30 records these detected signals 33A-C and packages them into a current fingerprint associated with the current time. The generation of the current fingerprint can be performed continuously in real time, or can be performed periodically or irregularly in batches. In other embodiments, the current fingerprint can be generated at one or more predetermined events, such as when the display device 30 is turned on. It should be understood that the format of the current fingerprint is similar to the format of the fingerprint 56 in the large-scale storage device 218 of the server computing device, so that the data in the current fingerprint can be compared and contrasted with the fingerprint data 54 of the fingerprint 56.

[0037] Independent of the generation of the current fingerprint, the display device 30 also retrieves fingerprint data 54 from the server computing device 200. The server computing device 200 may send the fingerprint data 54 to the display device 30 in response to the fingerprint request 52 sent by the display device 30. However, it will be appreciated that the server computing device 200 may alternatively or additionally initiate the sending of the fingerprint data 54 to the display device 30 - even in the absence of an instruction or request prompting to do so.

[0038] Steering Figure 4C , depicts a possible example of a fingerprint 56, fingerprint data 54, and a map data set 58, wherein the fingerprints correspond to different, distinguishable areas in a physical environment, and the fingerprint data includes one or more vectors. In this example, two different fingerprints are provided for an office environment and a home environment, respectively, and each vector is a pairing including a MAC address and a signal strength. However, it will be appreciated that SSIDs may also be used instead of MAC addresses. Fingerprint data 54A indicates the MAC address and signal strength of a beacon frame signal that would normally be detected or received by a display device in an office environment. Similarly, fingerprint data 54B indicates the MAC address and signal strength of a beacon frame signal that would normally be received by a display device in a home environment. Each fingerprint is linked to its own corresponding map data set: the office fingerprint 56A is linked to the office map data set 58A, and the home fingerprint 56B is linked to the home map data set 58B. It will be appreciated that the format of the fingerprint data is not particularly limited, and may include only one environment variable (e.g., only a MAC address, only an SSID, or only a signal strength), or multiple environment variables. For example, fingerprint data may include a mix of different types of signals, including a combination of cellular signals and Wi-Fi signals, or a combination of infrared measurements and Wi-Fi signals. There may be overlap in fingerprint data from two or more different fingerprints—for example, office fingerprint 56A and home fingerprint 56B may contain beacon frames with the same MAC address (in this example, MAC address <c>The fingerprint data 54 may be shared by two fingerprints 56A and 56B). In other embodiments, a common fingerprint may be linked to multiple map data sets, and / or a map data set may be linked to multiple fingerprints. The fingerprint data 54 may include a predicted signal detected by the predicted display device 30, or an actual signal that the display device has detected and recorded in the server computing device 200.

[0039] return Figure 4A , the client fingerprint program 38 executed by the display device 30 compares the current fingerprint with the fingerprint data 54 retrieved from the server computing device 200, and then evaluates and sorts the candidate fingerprints in the fingerprint data by the proximity to the current fingerprint. In this example, the candidate fingerprints are the office fingerprint 56A and the home fingerprint 56B. The comparison of the current fingerprint with the fingerprint data 54 can be performed in one dimension or in multiple dimensions. For example, the evaluation of the proximity or disambiguation degree of the candidate fingerprint with the current fingerprint can be performed in one dimension only according to the signal source, or in multiple dimensions according to the signal source and the signal strength. Proximity can be evaluated based on statistical analysis of similarity or disambiguation degree, and can include confidence values ​​and distance functions and distance metrics. Examples of distance functions include cosine distance functions and Ln distance functions. Examples of distance metrics include Hamming distance and Euclidean distance. A confidence algorithm that sorts confidence in descending order and / or a disambiguation degree algorithm combined with a distance function can also be applied to evaluate proximity. The current fingerprint and / or fingerprint data can be filtered to screen out outliers or remove other unwanted data. For example, the client fingerprint program 38 can be configured to filter out data from selected wireless access points, or remove MAC addresses with signal strengths below a predetermined threshold. For example, the client fingerprint program 38 can also impose a threshold or a predetermined upper limit on the number of vectors in the current fingerprint or fingerprint data, such as allowing each fingerprint to have a maximum of three vectors in the fingerprint data. User input can also be solicited during the statistical analysis process. For example, if multiple candidate fingerprints are within a preset similarity threshold, in the absence of a clear determining factor, user input can be solicited to provide input for disambiguation and selection between multiple candidate fingerprints before the evaluation of the candidate fingerprints is concluded. An example of soliciting user input can be presenting two options to the user: a "high confidence" candidate fingerprint, which has a sufficiently high confidence value and can be a reasonable match determined by a predetermined threshold; and a "similar confidence" candidate fingerprint, which is determined to be the second highest match determined by another predetermined threshold, so the user input will be disambiguated between the "high confidence" and "similar confidence" candidate fingerprints.

[0040] After evaluating the proximity of the candidate fingerprints to the current fingerprint, the client fingerprint program 38 determines that one of the candidate fingerprints is the closest match to the current fingerprint, and then generates and sends a map data request 48 for the map data set corresponding to the closest matching fingerprint. For example, if the client fingerprint program 38 determines that the office fingerprint 56A is the closest matching fingerprint, the program 38 generates and sends a map data request for the office map data set 58A. The server computing device 200 receives the map data request 48 and then retrieves the office map data set 58A to send to the display device 30 as a matching map data set 66. The display device 30 receives the office map data set 58A, stitches it into the local map data 36 to create an integrated map, and renders one or more holograms on the display 20 based on the integrated map.

[0041] Figure 5 FIG. 1 is a schematic diagram of a display device that generates a current fingerprint and interacts with a server computing device according to a second embodiment of the present disclosure. Figure 4A and Figure 5 The differences between the embodiments of the present invention mainly relate to differences in the structure of map data sets, fingerprints, and fingerprint data, so for the sake of brevity, a detailed description of the display device 30 and the server computing device 200 is omitted. It should be noted that throughout the detailed description and the drawings, similar parts are indicated by similar reference numerals. In this embodiment, instead of having multiple different, distinct map data sets for different environments, there is only one unified global map data set 158, which represents a unified physical environment containing multiple different physical environments. In some examples, the multiple different physical environments in the global map data set 158 ​​can include the entire world. Each fingerprint 156 is linked to its own corresponding key frame 160 instead of being linked to a corresponding map data set. At least two key frames can be linked to a common fingerprint, especially when the two key frames are physically close to each other. Therefore, the client fingerprint program 38 can determine that multiple candidate fingerprints are the closest match to the current fingerprint, and then generate and send a map data request 48 for the map data set or multiple map data sets corresponding to the key frame linked to the identification of the closest matching candidate fingerprint within the global map data set 158. In some examples, the client fingerprint program 38 may identify a neighborhood around the identified keyframe that is linked to the closest matching candidate fingerprint, which may include anchors, keyframes, and pose graphs that fall within a predetermined distance of the identified keyframe. However, it will be appreciated that a neighborhood may be arbitrarily defined as a three-dimensional space of any shape or size that encompasses the identified keyframes around or near them, including a neighborhood that may not necessarily include at least one of the identified keyframes. One possible embodiment of this neighborhood associated with fingerprints and keyframes is described below with respect to Figure 7 The server computing device 200 then sends one or more matching map data sets 66 corresponding to the identified keyframes to the display device 30 for splicing into the local map data 36 .

[0042] refer to Figure 6 , an example use case illustrating various aspects of the present disclosure will now be presented. Figure 6 As schematically shown in FIG. 1 , the first user 302 may stand in a restaurant 306, wearing a first display device 30. In this example, the first display device 30 may take Figure 1 . However, it will be appreciated that the display device is not particularly limited to an HMD device, and may take any form that allows a user to view a holographic image overlaid on a real-world 3D environment, such as dedicated smart phones and tablet devices, autonomous robots, and the like. As noted above, the first display device 30 (HMD device) and the second display device 34 may include at least partially perspective displays that are configured to visually enhance the views of the first user 302 and the second user 304, respectively, through displays of the real-world three-dimensional environment of the restaurant 306. The first display device 30 may generate a virtual model of the restaurant 306 using a three-dimensional coordinate space overlaid on the real-world restaurant. In Figure 6 In the example of FIG. 4 , such a three-dimensional coordinate space is indicated by x, y, and z axes.

[0043] As described in more detail below, the first display device 30 and the second display device 34 may also include program logic of the client fingerprint program 38, which retrieves one or more map data sets of the restaurant 306. The map data set can be constructed as key frames 60 linked by a gesture graph 62 and anchors 64 associated with the rendering of the hologram 50. In this example, the hologram 50 is projected onto the table 308 using a target anchor 64A located on the picture 310. Another adjacent anchor 64B for another hologram is located in the clock 312 near the picture 310. The first user 302 and the second user 304 roam in the room 306 as they respectively operate the first display device 30 and the second display device 34 to view the hologram 50 from their respective vantage points at various angles in the room 306. As the user navigates the room 306, the sensors 18 within the first display device 30 and the second display device 34 capture visual and / or inertial tracking data, thereby tracking the rotational and translational motion of the display devices via the sensor devices 18, which observe the three-dimensional rotation and translation of the sensor devices 18 to be recorded as poses 62A-G and keyframes 60A-G, which are subsequently stored as local map data 36 in the first display device 30 and local map data in the second display device 34. The local map data 36 may be transmitted to the server computing device 200 to be stored in the mass storage device 218 of the server computing device 200, and later retrieved as one or more matching map data sets 66 if the map data sets correspond to one or more closest matching fingerprints indicated by the map data request 48. Figure 7 Gestures 62A-G and keyframes 60A-G are described in greater detail. Display devices 30 and 34 then use the retrieved matched map data set 66 to orient and locate themselves and determine a more precise position of themselves relative to the physical environment of restaurant 306.

[0044] Steering Figure 7 , discusses in more detail one possible embodiment of map data applicable to the present disclosure. Information for the map data may be generated by at least one sensor device among a plurality of display devices, which sends sensor data including rotational and translational motions tracked by the sensor device to a computing device 200 in the form of a collection of gesture graphs 80 configured as keyframes 60A-G and connecting gestures 62A-H. Here, a display device and its keyframes and gesture graphs are depicted, but other embodiments may feature two or more display devices in proximity to each other, each with their own keyframe and gesture graph tracks. Also included in the map data are a plurality of virtual location anchors 64, including a target anchor 64A and adjacent anchors 64B at a world-locked virtual location with known three-dimensional coordinates in the physical environment. These anchors may include obvious prominent features in the physical environment, such as Figure 6 310 and clock 312 shown in . The gestures 62A-H, depicted as small arrows in the gesture graph 80, are typically unit vectors that point in the direction of a fixed straight-ahead gaze outside the display of the display device, as described above, and the gesture graph records the position of the gestures in three-dimensional space over time. The individual keyframes 60A-G are linked to each other in the gesture graph 80, which links the gestures 62A-H. Thus, the gesture graph includes a plurality of such gestures linked to each other in a directed graph so as to track changes in the gestures as the display device traverses the three-dimensional coordinate space of the physical environment. The gesture graph 80 forms a linear track of map data, which the display device leaves behind to be stored as local map data and subsequently sent to the server computing device 200 as map data for compilation and analysis as the display device traverses the three-dimensional coordinate space over time.

[0045] The keyframes 60A-G contain sets of information that can be used to improve the ability of the display device to determine its position, and thus help render the hologram in a stable position. As discussed above, examples of data included in the keyframes 60A-G include metadata, observations and tiles, and / or image feature descriptors. Metadata can include external data of the camera, the time when the keyframe was taken, gravity data, temperature data, magnetic data, calibration data, global positioning data, etc. Observations and tiles can provide information about feature points detected in the captured image, such as corners and high-contrast color changes, which help correct estimates of the positioning and orientation of the display device, and accordingly help to better align and position the display of the holographic image via the display 20 in three-dimensional space. Image feature descriptors can be feature points, sometimes efficiently represented in small data sets (as small as 32 bytes in some examples), which are used by the above-mentioned feature matching engine 13 to quickly identify features 121A in the real-time captured image 118A and depth image, accurately estimate the position of the display device, and thus accurately render the hologram on the map of the physical environment.

[0046] The pose graph 80 that links the keyframes to each other can be a plurality of continuously connected poses that convey how much rotation and translation the display device has undergone between the keyframes over time. Multiple anchors can be linked to each other via poses 62A-H. It will be appreciated that for a given keyframe, the geometric relationship between the display device and the hologram can be calculated by first calculating the distance between the current pose of the device and the anchor associated with the hologram, and then calculating the distance between the anchor and the hologram itself.

[0047] exist Figure 7 , the client fingerprint program 38 evaluates the candidate fingerprints 56A-G for proximity or disambiguation with the current fingerprint, and generates a ranked list 68 of the candidate fingerprints 56A-G based on the proximity or disambiguation with the current fingerprint based on the statistical analysis. In this example, the top 7 closest candidate fingerprints are included in the ranked list 68, but the number of fingerprints included in the ranked list 68 is not particularly limited and may include one or more candidate fingerprints. After evaluating the proximity or disambiguation, the client fingerprint program 38 determines that multiple candidate fingerprints in the ranked list 68 are the closest match to the current fingerprint, and then generates and sends a map data request for a map data set or multiple map data sets corresponding to the key frame linked to the identification of the closest matching candidate fingerprint within the global map data set 158. In this example, the client fingerprint program 38 has identified a neighborhood 67 around the key frame 60D linked to the identification of the closest matching candidate fingerprint 56D. Alternatively, the client fingerprint program 38 may identify neighborhoods around the key frames 60D and 60E that are linked to multiple identifications of the closest matching candidate fingerprints 56D and 56E, respectively. In this example, this neighborhood 67 includes anchor 64A, keyframes 60B-E, and poses 62C-F that fall within a predetermined distance of keyframe 60D linked to the identification of the closest matching fingerprint 56D, and neighborhood 67 is retrieved and sent to display device 30 as matching map dataset 66.

[0048] However, it will be appreciated that the neighborhood 67 may alternatively be arbitrarily defined as a three-dimensional space of any shape or size encompassing the identified keyframes around or near them, including a neighborhood that may not necessarily include at least one of the identified keyframes. The matching map dataset 66 may also include instructions for the sorting order in which the keyframes are loaded into the display device 30. For example, the instructions may instruct the display device 30 to sequentially load keyframes 60D, 60E, 60C, 60B, 60F, 60F, 60A, and 60G in the sorting order of fingerprints 56D, 56E, 56C, 56B, 56F, 56A, and 56G, respectively. Alternatively, if multiple matching map datasets are sent to the display device 30, the instructions may include the sorting order in which the individual map datasets are loaded into the memory of the display device 30. This sorting order of fingerprints or map datasets may also be used by the display device 30 to determine the order in which keyframes or map datasets are discarded or evicted when the display device 30 encounters memory limitations because the display device 30 is constantly merging new map datasets. In this example, when the display device 30 clears its memory of old map data sets, the display device 30 will discard key frames 60G, 60A, 60F, 60B, 60C, 60E and 60D in sequence. It can be understood that the sorting order of fingerprints can also be used to determine the order in which other data sets (such as pose graphs and anchors) are evicted from the memory of the display device 30.

[0049] Fig. 8A FIG. 6 is a flowchart of a first configuration of a method 600 for performing coarse positioning according to an example of the present disclosure. The following description of the method 600 is based on the Figure 1-Figure 7 It is understood that the method 600 may also be performed in other contexts using other suitable hardware and software components.

[0050] refer to Fig. 8A , at 602, method 600 may include a sensor of a display device receiving a signal from one or more signal sources, such as a wireless access point. At 604, the display device generates a current fingerprint based on the received signal. At 606, the display device sends a request for fingerprint data to a server computing device. At 608, the server computing device receives the request for fingerprint data. At 610, the server computing device sends the fingerprint data to the display device. At 612, the display device receives the fingerprint data from the server computing device. At 614, the display device compares the current fingerprint with the fingerprint data. At 616, the display device evaluates the candidate fingerprints in the fingerprint data by proximity to the current fingerprint and sorts the candidate fingerprints. At 618, the display device sends a request for one or more map data sets corresponding to one or more closest matching fingerprints. At 620, the server computing device receives a request for one or more map data sets corresponding to one or more closest matching fingerprints. At 622, the server computing device retrieves one or more map data sets corresponding to the one or more closest matching fingerprints and sends them to the display device. At 624, the display device receives one or more corresponding map data sets. At 626, the display device stitches the corresponding map data sets into the local map data to create an integrated map. At 628, the display device renders one or more holograms based on the integrated map.

[0051] Figure 8B 1 is a flow chart illustrating a second configuration of a method 700 for performing coarse positioning according to an example of the present disclosure. The following description of the method 700 is based on the Figure 1-Figure 7 It is provided by the software and hardware components shown in and described above. It is understood that the method 700 can also be performed in other contexts using other suitable hardware and software components.

[0052] refer to Figure 8B , at 702, method 700 may include a sensor of a display device receiving a signal from one or more signal sources, such as a wireless access point. At 704, the display device generates a current fingerprint based on the received signal. At 706, the display device sends a request for fingerprint data to a server computing device. At 708, the server computing device receives the request for fingerprint data. At 710, the server computing device compares the current fingerprint with the fingerprint data. At 712, the server computing device evaluates and sorts candidate fingerprints in the fingerprint data by proximity to the current fingerprint. At 714, the server computing device retrieves one or more map data sets corresponding to one or more closest matching fingerprints and sends them to the display device. At 716, the display device receives one or more corresponding map data sets. At 718, the display device splices the corresponding map data sets into the local map data to create an integrated map. At 720, the display device renders one or more holograms based on the integrated map.

[0053] Figure 8C A flowchart of a third configuration of a method 800 for performing coarse positioning according to an example of the present disclosure is illustrated. The following description of the method 800 is based on the Figure 1-Figure 7 It is understood that the method 800 may also be performed in other contexts using other suitable hardware and software components.

[0054] refer to Figure 8C , at 802, method 800 may include a sensor of a display device receiving a signal from one or more signal sources. At 804, the display device generates a current fingerprint based on the received signal. At 806, the display device compares the current fingerprint with the fingerprint data. At 808, the display device evaluates candidate fingerprints in the fingerprint data by proximity to the current fingerprint and sorts the candidate fingerprints. At 810, the display device retrieves one or more map data sets corresponding to one or more closest matching fingerprints. At 812, the display device splices the corresponding map data sets into the local map data to create an integrated map. At 814, the display device renders one or more holograms based on the integrated map.

[0055] It will be appreciated that methods 600, 700, and 800 are provided as examples and are not meant to be limiting. Fig. 8A , Figure 8B and Figure 8C . In addition, it will be appreciated that methods 600, 700, and 800 may be performed in any suitable order. In addition, it will be appreciated that one or more steps may be omitted from methods 600, 700, and 800 without departing from the scope of the present disclosure. For example, the comparison of the current fingerprint with the fingerprint data and the evaluation and ranking of the candidate fingerprints may be performed by a display device, a server computing device, or another computing device. In addition, splicing the corresponding map data set into the local map data may alternatively be performed by a server computing device or another computing device. The map data set to be retrieved may be stored in a display device, a server computing device, or another computing device.

[0056] In summary, when the display device retrieves one or more map data sets to orient and locate itself, it uses signal data detected by sensors in the display device to generate a current fingerprint. This current fingerprint is compared to the existing fingerprint data and statistically analyzed for proximity or disambiguation with the candidate fingerprints of the fingerprint data. A sorted list of the closest matching fingerprints is generated, and then the map data corresponding to the closest matching fingerprints is sent to the display device so that the display device can efficiently load the map data and merge it into the local map data stored in the display device memory to effectively orient and locate itself in the physical environment. Then, as 6-DoF tracking continues to run on the display device to leave a track of linear map data stored as local map data, the device can discard or evict map data based on the reverse order of the sorted list of the closest matching fingerprints.

[0057] Thus, the coarse relocalization step is accelerated so that the display device can more quickly and reliably orient and position itself relative to the physical environment around it, allowing the user to easily physically move between different environments and automatically adapt the discovered relevant map data for the new environment. The memory requirements of the display device are also reduced, so that larger environmental data can be incrementally loaded without adversely affecting the user experience. Various statistical analysis tools can be applied to optimize for different scenarios, such as calibration to facilitate loading speed or reduce memory requirements. Various existing map data formats can be implemented to retain the flexibility of keyframe-based map data, which, for example, allows flexible adjustment of sampling distances.

[0058] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0059] Fig. 9 A non-limiting embodiment of a computing system 900 that can perform one or more of the above methods and processes is schematically illustrated. The computing system 900 is shown in simplified form. The computing system 900 may embody Figure 1 The computing device 10 or Figure 4A and Figure 5 The computing system 900 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0060] The computing system 900 includes a logic processor 902, a volatile memory 904, and a non-volatile storage device 906. The computing system 900 may optionally include a display subsystem 908, an input subsystem 910, a communication subsystem 912, and / or Fig. 9 Other components not shown.

[0061] Logical processor 902 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, convert the state of one or more components, implement a technical effect, or otherwise achieve a desired result.

[0062] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 902 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the logical processor may optionally be distributed among two or more separate devices, which may be located at a distance and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In this case, it will be understood that these virtualized aspects run on different physical logical processors of various different machines.

[0063] The non-volatile storage device 906 includes one or more physical devices configured to store instructions executable by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 906 may be transformed—for example, to hold different data.

[0064] The non-volatile storage device 906 may include a removable and / or built-in physical device. The non-volatile storage device 906 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 906 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It will be appreciated that the non-volatile storage device 906 is configured to retain instructions even when power to the non-volatile storage device 906 is cut off.

[0065] The volatile memory 904 may include physical devices including random access memory. The volatile memory 904 is typically utilized by the logical processor 902 to temporarily store information during the processing of software instructions. It will be appreciated that when the volatile memory 904 is powered off, the volatile memory 904 typically does not continue to store instructions.

[0066] Aspects of the logic processor 902, volatile memory 904, and non-volatile storage device 906 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program-specific and application-specific integrated circuits (PASIC / ASIC), program-specific and application-specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0067] The terms "module", "program", and "engine" may be used to describe an aspect of the computing system 900, typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves transformative processing that specifically configures the processor to perform that function. Thus, a module, program, or engine may be instantiated via the logical processor 902 executing instructions held by the non-volatile storage device 906 using portions of the volatile memory 904. It will be appreciated that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may include a single or a group of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0068] When included, the display subsystem 908 can be used to present a visual representation of the data held by the non-volatile storage device 906. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data held by the non-volatile storage device, and thus convert the state of the non-volatile storage device, the state of the display subsystem 908 can also be converted to visually represent the change of the underlying data. The display subsystem 908 can include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logical processor 902, the volatile memory 904, and / or the non-volatile storage device 906 in a shared shell, or such a display device can be a peripheral display device.

[0069] When included, the input subsystem 910 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transformation and / or processing of input actions may be processed on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0070] When included, the communication subsystem 912 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple to other devices. The communication subsystem 912 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or a wide area network (such as HDMI and Bluetooth connected via Wi-Fi). In some embodiments, the communication subsystem can allow the computing system 900 to send and / or receive messages to and / or from other devices via a network such as the Internet.

[0071] Fig.10 is a schematic diagram of a display device acquiring a session specific identifier and interacting with a cloud service of a server computing device according to a third embodiment of the present disclosure. Figure 5 and Fig.10 The differences between the embodiments primarily relate to the differences in the manner in which the display device requests map data from the server computing device, and therefore a detailed description of the display devices 30a and 30b and the server computing device 200 is omitted for the sake of brevity. It should be noted that similar parts are designated by similar reference numerals throughout the detailed description and the accompanying drawings. In this example, the first display device 30a captures visual and / or inertial tracking data, thereby tracking the rotational and translational motion of the display device 30a via the sensor device 18a, which observes the three-dimensional rotation and translation of the sensor device 18a, which is recorded as poses and keyframes, and then stores the poses and keyframes as local map data 36a in the first display device 30a. The first display device 30a also generates a session-specific identifier 155 associated with the local map data 36a. If the map data set corresponds to the session-specific identifier 155 indicated by the map data request 48, the first display device 30a then transmits the session-specific identifier 155 and the local map data 36a to the server computing device 200 to be stored in the mass storage device 218 of the server computing device 200 and subsequently retrieved as one or more matching map data sets 66. The corresponding map data may include adjacent map data for a neighborhood around one or more keyframes linked to the session-specific identifier 155.

[0072] The session-specific identifier 155 is an identifier associated with a predetermined session. For example, the session-specific identifier 155 can be a unique code or a temporary non-unique code that is valid only for the duration of the predetermined session. The session-specific identifier 155 can be based on a beacon frame of a wireless signal 33A-C broadcast by one or more wireless access points 32A-C. It will be appreciated that the session-specific identifier 155 is not limited to a MAC address and can be based on one or more features within the beacon frame of the wireless signal 33A-C. The session-specific identifier 155 can indicate a specific category of device, such as an augmented reality device. It will be appreciated that the session-specific identifier 155 preferably does not contain privacy information of a user of the first display device 30a or the second display device 30b in order to protect the privacy of the user's personal data.

[0073] In this embodiment, the first display device 30a is a device that has entered the environment 306 before the second display device 30b. When the second display device 30b enters the environment 306, the second display device 30b begins the process of orienting itself by acquiring local map data 36b corresponding to the new environment 306. The second display device 30b can transmit an identifier request 51 to surrounding display devices (such as the first display device 30a) to determine whether there are other devices subscribed to the same remote system 215, so that they can access the server fingerprint program 214 of the remote system 215 of the server computing device 200. The identifier request 51 may include an identifier recognized as a request by the client fingerprint program 38a of the first display device 30a, which can utilize a low-energy beacon (such as The transmission of the identifier request 51 is performed using a WIFI beacon (beacon) to detect nearby devices that are physically close to the second display device 30b. WIFI beacon frames can also be used. The client fingerprint program 38a executed by the first display device 30a and the client fingerprint program 38b executed by the second display device 30b are configured to use the same cloud service 215, which includes a server fingerprint program 214 hosted on the server computing device 200. Therefore, both the client fingerprint program 38a and the client fingerprint program 38b are configured to recognize the identifier request 51 and the session-specific identifier 155 because these parameters are defined according to a pattern known to each client fingerprint program 38a, 38b and the cloud service 215 and communicated between them.

[0074] It is understood that the server computing device 200 can take the form of a server, a networked computer, a game console, a mobile communication device, an edge computing device, a desktop computer, a laptop computer, a tablet computer, a set-top box (e.g., a cable box, a satellite box), or any other type of suitable computing device that is configured to execute server software in response to requests from clients. In some examples, the server computing device 200 can be an embedded system within a larger electronic or mechanical device or system.

[0075] It is understood that the remote system 215 (hereinafter described as a cloud service) is not particularly limited and can be a private, public, hybrid or community cloud service, deployed internally or externally, which can be accessed by the first display device 30a and the second display device 30b. When the cloud service 215 is configured as a private cloud operated solely for an organization, the location of the second display device 30b based on the session-specific identifier 155 can be enabled, while still allowing the organization to maintain control of all data generated between the display devices 30a and 30b and the server computing device 200. When the cloud service 215 is configured as a public cloud available to the public or a large industrial group, it is understood that additional data security infrastructure can be deployed to maintain control of the data generated between the server computing device 200 and the public's multiple display devices operating within the environment 306.

[0076] The first display device 30a detects or receives the identifier request 51. In response, the first display device 30a transmits the session-specific identifier 155 to the second display device 30b. The second display device 30b receives the session-specific identifier 155 from the first display device 30a. In response to receiving the session-specific identifier 155 from the first display device 30a, the second display device 30b transmits a map data request 48 including the session-specific identifier 155 to the cloud service 215 of the server computing device 200, requesting a map data set corresponding to the session-specific identifier 155 within the global map data set 158. The server computing device 200 receives the map data request 48 including the session-specific identifier 155, and then retrieves and sends a matching map data set corresponding to the session-specific identifier 155 to the second display device 30b as a matching map data set 66. The second display device 30b receives the matching map data set 66 corresponding to the session-specific identifier 155 from the cloud service 215. The second display device 30b then stitches the corresponding matching map data set 66 into the local map data 36b to create an integrated map. The second display device 30b then renders one or more holograms based on the integrated map, so that the display of the second display device 30b displays the one or more holograms based on the retrieved map dataset 66 corresponding to the session-specific identifier 155. As discussed above, the type of retrieved map dataset 66 is to be understood as any dataset that associates points in a three-dimensional coordinate space in a physical environment with information that helps orient and locate a display device in the three-dimensional space. Figure 6 and Figure 7 As described in, the retrieved map dataset 66 may include the keyframe 60, the gesture graph 62, and the anchor 64 in the form of anchor data. In some embodiments, it is understood that the retrieved map dataset 66 may not include the keyframe 60, the gesture graph 62, or the anchor 64; instead, the retrieved map dataset 66 may include the location data of one or more holograms associated with the anchor in the matching map dataset 66 corresponding to the session-specific identifier 155. According to this configuration, by retrieving the map dataset 66 corresponding to the session-specific identifier 155, the memory requirements of the second display device are also reduced, so that larger environmental data can be incrementally loaded without adversely affecting the user experience, and the sharing of map data or holographic location data can be simplified by utilizing near field communication between devices, thereby shortening the loading time.

[0077] refer to Fig.11A , which illustrates a flow chart of a fourth configuration of a method 1000 for performing coarse positioning according to an example of the present disclosure. The following description of the method 1000 is based on the Figure 1-Figure 7 1 and described above. It will be appreciated that method 1000 may also be performed in other contexts using other suitable hardware and software components. Although only two display devices are depicted in the flowchart, it will be appreciated that method 1000 may also be applied to multiple pairs of devices in an environment including several devices subscribed to or registered with the same cloud service.

[0078] refer to Fig.11A , at 1006, the first display device transmits a wireless signal including a session-specific identifier. At 1008, the second display device receives the wireless signal including the session-specific identifier corresponding to the map data set. At 1010, the second display device sends the session-specific identifier and a request for the map data set to the cloud service. At 1012, the cloud service receives the session-specific identifier and the map data request from the second display device. At 1014, in response to receiving the session-specific identifier and the map data request, the cloud service retrieves the map data set corresponding to the session-specific identifier and sends it to the second display device. At 1016, the second display device receives the map data set corresponding to the session-specific identifier. At 1018, the second display device stitches the corresponding map data set into the local map data to create an integrated map. At 1020, the second display device renders one or more holograms based on the integrated map.

[0079] refer to Fig. 11B , which illustrates a flowchart of a fifth configuration of a method 1100 for performing coarse positioning according to an example of the present disclosure. The following description of the method 1100 is based on the Figure 1-Figure 7 The method 1100 is provided by the software and hardware components shown in and described above. It will be appreciated that the method 1100 may also be performed in other contexts using other suitable hardware and software components. Although only two display devices are depicted in the flowchart, it will be appreciated that the method 1100 may also be applied to multiple pairs of devices in an environment including several devices subscribed to or registered with the same cloud service.

[0080] refer to Fig. 11B At 1102, the second display device transmits a wireless signal requesting a session-specific identifier. At 1104, the first display device receives the request for the session-specific identifier from the second display device. At 1106, the first display device transmits a wireless signal including the session-specific identifier. At 1108, the second display device receives a wireless signal from the first display device, the wireless signal including the session-specific identifier corresponding to the map data set. At 1110, the second display device sends the session-specific identifier and the request for the map data set to the cloud service. At 1112, the cloud service receives the session-specific identifier and the map data request from the second display device. At 1114, in response to receiving the session-specific identifier and the map data request, the cloud service retrieves the map data set corresponding to the session-specific identifier and sends it to the second display device. At 1116, the second display device receives the map data set corresponding to the session-specific identifier. At 1118, the second display device stitches the corresponding map data set into the local map data to create an integrated map. At 1120, the second display device renders one or more holograms based on the integrated map.

[0081] Fig. 11C A flowchart of a sixth configuration of a method 1200 for performing coarse positioning according to an example of the present disclosure is illustrated. In the sixth configuration of method 1200, the display device initially attempts to receive a signal from one or more signal sources to create a current fingerprint. If it is determined that the quality of the received signal is sufficient to generate the current fingerprint, the display device makes a map data set request based on the current fingerprint. Otherwise, if it is determined that the quality of the received signal is insufficient to generate the current fingerprint, the display device requests and receives a session-specific identifier from a neighboring device, and makes a map data set request based on the session-specific identifier. The following description of method 1200 is with reference to Figure 1-Figure 7 It is understood that the method 1200 may also be performed in other contexts using other suitable hardware and software components.

[0082] refer to Fig. 11C At 1202, method 1200 may include a sensor of a display device receiving a signal from one or more signal sources. At 1204, method 1200 determines whether the received signal is sufficient to generate a current fingerprint. When it is determined that the received signal is insufficient to generate the current fingerprint, method 1200 proceeds to 1206. The number of received wireless signals below a predetermined threshold or the signal-to-noise ratio below a predetermined threshold may indicate that the received signal is insufficient to generate the current fingerprint.

[0083] At 1206, the display device transmits a wireless signal requesting a session-specific identifier from a neighboring device. At 1208, the display device receives a wireless signal including the session-specific identifier from the neighboring device. At 1210, the display device sends the session-specific identifier and a request for a map data set to the cloud service. At 1212, the display device receives a map data set corresponding to the session-specific identifier from the cloud service. At 1214, the display device stitches the corresponding map data set into the local map data to create an integrated map. At 1216, the display device renders one or more holograms based on the integrated map.

[0084] At 1204, when it is determined that the received signal is sufficient to generate the current fingerprint, method 1200 proceeds to 1218. A number of received wireless signals above a predetermined threshold or a signal-to-noise ratio above a predetermined threshold may indicate that the received signal is sufficient to generate the current fingerprint.

[0085] At 1218, the display device generates a current fingerprint based on the received signal. At 1220, the display device compares the current fingerprint to the fingerprint data. At 1222, the display device evaluates and sorts candidate fingerprints in the fingerprint data by proximity to the current fingerprint. At 1224, the display device retrieves one or more map data sets corresponding to one or more closest matching fingerprints. At 1226, the display device splices the corresponding map data sets into the local map data to create an integrated map. At 1228, the display device renders one or more holograms based on the integrated map.

[0086] Fig.11D A flowchart of a seventh configuration of method 1300 for performing coarse positioning according to an example of the present disclosure is illustrated. In the seventh configuration of method 1300, the display device initially attempts to contact a neighboring device subscribed to the same cloud service, which provides a session-specific identifier to the display device. When the display device successfully obtains the session-specific identifier from the neighboring device, the display device then makes a map dataset request based on the session-specific identifier. Otherwise, when the display device does not successfully obtain the session-specific identifier from the neighboring device, the display device receives a signal from one or more signal sources to create a current fingerprint, and then makes a map dataset request based on the current fingerprint. The following description of method 1300 is with reference to Figure 1-Figure 7 It is provided by the software and hardware components shown in and described above. It is understood that the method 1300 can also be performed in other contexts using other suitable hardware and software components.

[0087] At 1302, the display device transmits a wireless signal requesting a session-specific identifier from a neighboring device. At 1304, when the display device receives a wireless signal including the session-specific identifier from the neighboring device, the method proceeds to 1310. At 1310, the display device sends the session-specific identifier and a request for a map data set to a cloud service. At 1312, the display device receives a map data set corresponding to the session-specific identifier from the cloud service. At 1314, the display device stitches the corresponding map data set into the local map data to create an integrated map. At 1316, the display device renders one or more holograms based on the integrated map.

[0088] At 1304, when the display device is unable to receive a wireless signal including a session-specific identifier from a neighboring device, method 1300 proceeds to 1317. At 1317, a sensor of the display device receives a signal from one or more signal sources. At 1318, the display device generates a current fingerprint based on the received signal. At 1320, the display device compares the current fingerprint with the fingerprint data. At 1322, the display device evaluates candidate fingerprints in the fingerprint data by proximity to the current fingerprint and sorts the candidate fingerprints. At 1324, the display device retrieves a map data set corresponding to one or more closest matching fingerprints. At 1326, the display device splices the corresponding map data set into the local map data to create an integrated map. At 1328, the display device renders one or more holograms based on the integrated map.

[0089] The following paragraphs provide additional support for the claims of the present application. One aspect provides a first display device, comprising: a processor; a display; a memory operably coupled to the processor and the display; a client fingerprint program stored in the memory and executed by the processor of the first display device, the client fingerprint program being configured to: receive a session-specific identifier from a second device, the session-specific identifier corresponding to a map data set; transmit the session-specific identifier to a remote system; receive at least a portion of the map data set corresponding to the session-specific identifier from the remote system; and stitch the map data set into local map data of the first display device to create an integrated map. In this aspect, additionally or alternatively, the client fingerprint program may also be configured to cause the display of the first display device to display one or more holograms based on the map data corresponding to the session-specific identifier. In this aspect, additionally or alternatively, the map data set may include keyframes; and the keyframes and the gesture graph may record rotational and translational motion of the first display device through a real-world three-dimensional environment. In this aspect, additionally or alternatively, at least two of the keyframes may be linked to a common fingerprint. In this aspect, additionally or alternatively, the map data set corresponding to the session-specific identifier may include adjacent map data of a neighborhood around one or more key frames linked to the session-specific identifier. In this aspect, additionally or alternatively, when the client fingerprint program fails to successfully receive the session-specific identifier from the second device, the client fingerprint program may be configured to receive one or more signals from one or more signal sources by at least one sensor of the first display device; generate a current signal fingerprint based on the received signal; determine the map data to be loaded by comparing the current signal fingerprint with the signal fingerprint data to generate a sorted list of matching fingerprints, including performing statistical analysis to evaluate the proximity or disambiguation degree of the current signal fingerprint with the signal fingerprint data; and based on the sorted list, retrieve the corresponding map data corresponding to the matching fingerprint in the sorted list. In this aspect, additionally or alternatively, the client fingerprint program may generate the session-specific identifier based on a signal from a signal source, the signal source including at least one of a wireless access point, a radio frequency source, an electromagnetic source, a light source, an acoustic source, and a heat source. In this aspect, additionally or alternatively, the signal may include a beacon frame. In this aspect, additionally or alternatively, the session-specific identifier may be received from the second device via a low-energy beacon. In this aspect, additionally or alternatively, the client fingerprint program may transmit an identifier request; and the client fingerprint program may receive the session-specific identifier from the second device that received the transmitted identifier request.

[0090] Another aspect provides a method, including: receiving a session-specific identifier from a second device at a first display device, the session-specific identifier corresponding to a map data set; transmitting the session-specific identifier from the first display device to a remote system; receiving at least a portion of the map data set corresponding to the session-specific identifier from the remote system at the first display device; and stitching the map data set into local map data at the first display device to create an integrated map. In this aspect, additionally or alternatively, the method may also include displaying one or more holograms at the first display device based on the map data set corresponding to the session-specific identifier. In this aspect, additionally or alternatively, the map data set may include keyframes; and the keyframes and the gesture graph may record rotational and translational motion of the first display device through a real-world three-dimensional environment. In this aspect, additionally or alternatively, at least two of the keyframes may be linked to a common fingerprint. In this aspect, additionally or alternatively, the map data set corresponding to the session-specific identifier may include adjacent map data of a neighborhood around one or more keyframes linked to the session-specific identifier. In this aspect, additionally or alternatively, the session-specific identifier may be generated at the second device based on a beacon frame of a signal from a signal source. In this aspect, additionally or alternatively, a session-specific identifier may be generated at the second device based on a signal from a signal source, the signal source comprising at least one of a wireless access point, a radio frequency source, an electromagnetic source, a light source, an acoustic source, and a heat source. In this aspect, additionally or alternatively, the session-specific identifier may be received at the first display device from the second device via a low-energy beacon. In this aspect, additionally or alternatively, the method may further include transmitting an identifier request at the first display device; and, at the first display device, receiving the session-specific identifier from the second device that received the transmitted identifier request.

[0091] On the other hand, a server computing device is provided, including a processor; a memory operably coupled to the processor; signal fingerprint data stored in the memory; a server fingerprint program stored in the memory and executed by the processor of the computing device, the server fingerprint program being configured to: receive from a first mobile device one or more signals received by a second mobile device based on one or more signals from one or more signal sources received by at least one sensor of the second mobile device; and send to the first mobile device a corresponding map data set corresponding to a session-specific identifier, the corresponding map data set including adjacent map data of a neighborhood around one or more key frames linked to the session-specific identifier, the adjacent map data being captured and recorded by the second mobile device.

[0092] It is understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific procedures or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions illustrated and / or described may be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process may be changed.

[0093] The subject matter of the present disclosure includes all novel and nonobvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents.< / c>

Claims

1. A first display device in an environment, comprising: processor; monitor; a memory operatively coupled to the processor and the display; as well as a first client fingerprint program, the client fingerprint program being stored in the memory and executed by the processor of the first display device, The first client fingerprint program is configured as follows: transmitting the identifier request to the second device; In response to the transmitting step, a session-specific identifier is received from the second device, the session-specific identifier corresponding to a map data set, wherein the session-specific identifier is an identifier associated with a predetermined session, the session-specific identifier is valid only for the duration of the predetermined session, wherein the second device is a second display device within the environment that is close to the first display device, and the second display device has entered the environment before the first display device, the second display device includes a second client fingerprint program, both the second client fingerprint program and the first client fingerprint program are configured to recognize the identifier request and the session-specific identifier, and wherein the map data set corresponds to local map data of the second display device corresponding to the environment, the local map data is associated with the session-specific identifier generated by the second display device, and the session-specific identifier and the local map data are subsequently transmitted to a remote system including a storage device for storing the map data set; transmitting a map data request including the session specific identifier to the remote system; receiving, from the remote system, the map data set corresponding to the session specific identifier; as well as The map data set is stitched into local map data corresponding to the first display device of the environment to create an integrated map.

2. The first display device according to claim 1, wherein The client fingerprint program is further configured to cause the display of the first display device to display one or more holograms based on the map data corresponding to the session specific identifier.

3. The first display device according to claim 2, wherein A map dataset is constructed as keyframes linked by a pose graph and anchors associated with a rendering of the hologram; and the keyframes and the pose graph record rotational and translational movement of the first display device through a real-world three-dimensional environment.

4. The first display device according to claim 3, wherein The map data set corresponding to the session-specific identifier includes neighboring map data of a neighborhood around one or more keyframes linked to the session-specific identifier.

5. The first display device according to claim 1, wherein When the client fingerprint program fails to successfully receive the session specific identifier from the second device, the client fingerprint program is configured to: receiving, by at least one sensor of the first display device, one or more signals from one or more signal sources; generating a current signal fingerprint based on the received signal; determining map data to load by comparing the current signal fingerprint to signal fingerprint data to generate a ranked list of matching fingerprints, including performing a statistical analysis to assess the closeness or disambiguation of the current signal fingerprint to the signal fingerprint data; as well as Based on the sorted list, map data corresponding to the matching fingerprint in the sorted list is retrieved.

6. The first display device according to claim 1, wherein The session specific identifier is received from the second device via a low energy beacon.

7. A method comprising: transmitting an identifier request from a first display device to a second device in an environment; In response to the transmitting step, receiving at the first display device from the second device a session-specific identifier, the session-specific identifier corresponding to a map data set, wherein the session-specific identifier is an identifier associated with a predetermined session, the session-specific identifier being valid only for the duration of the predetermined session, and wherein the second device is a second display device in the environment proximate to the first display device, and the second device has entered the environment before the first display device, the second display device includes a second client fingerprint program, both the second client fingerprint program and the first client fingerprint program of the first display device are configured to recognize the identifier request and the session-specific identifier, and wherein the map data set corresponds to local map data of the second display device corresponding to the environment, the local map data being associated with the session-specific identifier generated by the second display device, the session-specific identifier and the local map data being subsequently transmitted to a remote system including a storage device for storing the map data set; transmitting a map data request including the session specific identifier from the first display device to the remote system; receiving, from the remote system, the map data set corresponding to the session specific identifier at the first display device; as well as The map data set is stitched into local map data corresponding to the environment to create an integrated map at the first display device.

8. The method according to claim 7, further comprising: At the first display device, one or more holograms are displayed based on the map data set corresponding to the session-specific identifier.

9. The method according to claim 8, wherein A map dataset is structured as keyframes linked by a pose graph and anchors associated with said hologram rendering; and The key frames and pose graph record rotational and translational movement of the first display device through a real-world three-dimensional environment.

10. The method according to claim 9, wherein The map data set corresponding to the session-specific identifier includes neighboring map data of a neighborhood around one or more keyframes linked to the session-specific identifier.

11. The method according to claim 7, wherein The session specific identifier is generated at the second device based on a beacon frame of a signal from a signal source.

12. The method according to claim 7, wherein The session specific identifier is received at the first display device from the second device via a low energy beacon.

Images