Custom landmark progressive scanning

Abstract

A method for generating an updated locator map of a reference scene is provided. The method may comprise: obtaining a preliminary locator map of a reference scene, the preliminary locator map comprising preliminary frames, each preliminary frame being associated with a set of preliminary data points; capturing a plurality of new data points on the reference scene; determining a pose of a capture device related to capture of the plurality of new data points; creating at least one new frame; selecting one or more target frames from the preliminary frame and the new frame; and generating an updated locator map based on the one or more target frames.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 368,600, filed on July 15, 2022, entitled “INCREMENTAL SCANNING FOR CUSTOM LANDMARKERS,” which is incorporated herein by reference in its entirety. Background Art

[0003] Augmented reality (AR) can include the use of computer-generated augmentations to add new information to an image in real time or near real time. For example, an image of a real-world object on a device's display can be enhanced by displaying details that do not exist on the object but are generated by the AR system to appear to be on or part of the object. AR systems require a complex blending of integrated and matched image capture information with AR information that is added to the captured scene in an attempt to seamlessly present the final image from the perspective determined by the image capture device.

[0004] Various technologies can be used for AR rendering, including optical projection systems, monitors, handheld devices, and display systems worn on the body, such as glasses, contact lenses, or heads-up displays (HUDs). BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0006] Figure 1 and Figure 2 Pictures of reference scenes captured under different lighting conditions are shown according to some examples.

[0007] Figure 3 Pictures of a reference scene captured from different perspectives are shown according to some examples.

[0008] Figure 4 A diagram showing a sparse map corresponding to a reference scene according to some examples.

[0009] Figure 5 A diagram showing an old sparse map and multiple new frames according to some examples.

[0010] Figure 6 Diagram showing a transformation from a new coordinate system to an old coordinate system according to some examples.

[0011] Figure 7 and Figure 8 Two pictures covering different areas of a reference scene are shown, according to some examples.

[0012] Fig. 9 and Fig.10 Two sparse maps created corresponding to a reference scene according to some examples are shown.

[0013] Fig.11 and Fig.12 Two updated sparse maps according to some examples are shown.

[0014] Fig.13 A diagram showing generation of an updated sparse map according to some examples.

[0015] Fig.14 A flow chart illustrating a method of adding a customer landmark to a landmark menu is shown, according to some examples.

[0016] Fig.15 A flow chart illustrating a method for performing a progressive scan, according to some examples, is shown.

[0017] Fig.16 A flow chart illustrating a method for generating an updated locator map of a reference scene, according to some examples, is shown.

[0018] Fig.17 A flow chart illustrating a method of aligning an updated locator map with a preliminary 3D data representation of a reference scene is shown, according to some examples.

[0019] Fig.18 is a diagrammatic representation of a networking environment in which the present disclosure may be deployed, according to some examples.

[0020] Fig.19 is a diagrammatic representation of a messaging system having both client-side and server-side functionality according to some examples.

[0021] Fig. 20 is a diagrammatic representation of data structures as maintained in a database according to some examples.

[0022] Fig.21 is a graphical representation of messages according to some examples.

[0023] Fig. 22 Systems in which a head wearable device is provided according to some examples are shown.

[0024] Fig.23is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0025] Fig.24 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0026] In the context of AR (augmented reality) technology, an AR landmark ("landmarker") can refer to a digital representation (e.g., a 3D model) of a physical landmark, which is a feature or object in the physical world that is recognized and tracked by an AR system. Landmarks are used as reference points or anchors for placing virtual content or overlays in the user's augmented reality experience. AR systems typically use computer vision algorithms and sensors (such as cameras) to detect and identify specific landmarks in the real-world environment. Therefore, landmarks can be anything from landmarks, buildings, and signs to everyday objects such as tables, chairs, and even people. Once a landmark is identified, the AR system can accurately position and align virtual objects or information relative to the physical world.

[0027] Landmarks play a role in creating immersive and interactive AR experiences. They enable AR systems to understand the user's surroundings and provide relevant virtual content based on the location, orientation or other characteristics of detected landmarks. This technology can be used in a variety of AR applications such as gaming, navigation, education, retail and industrial training.

[0028] Examples described herein relate to methods and systems that allow users to incrementally update custom AR landmarks. Such updates may be particularly useful in some example scenarios including:

[0029] Use AR models in different lighting conditions (see e.g. Figure 1 and Figure 2 );

[0030] The AR models cover a wide range of reference scenarios (see e.g. Figure 7 and Figure 8 );as well as

[0031] The surroundings of the reference scene have changed.

[0032] For example, consider that a custom AR landmark built during daytime is unlikely to be located / tracked at night, and vice versa. It is desirable to add the ability for custom landmarks to provide the same AR experience even under different lighting conditions than when the landmark was created. A user may have built a custom landmark during daytime. When a user subsequently revisits a custom landmark during nighttime, the user may not be able to get the same AR experience as they did during daytime. This is because, due to changes in lighting, the AR system cannot determine the position and orientation of the user's capture device relative to the previously created map. Similar problems arise when old custom landmarks are used for a large range on a reference scene, or when the surrounding environment of the reference scene has changed, such as when a piece of interior furniture has been moved around in the reference scene. It should be noted that in the examples provided in the present disclosure, the term "reference scene" may refer to an object, a portion of an object (e.g., a surface), or an environment containing multiple objects.

[0033] The described example progressive landmarking techniques provide enhancement or digital effect creators (e.g., "lens" creators) with the ability to progressively update the locator map to add a nighttime map to an existing daytime map to support various lighting conditions or to expand the locator map to cover a larger area nearby, etc. However, the progressive map building feature may result in the following issues:

[0034] 1. The map is too large - beyond the system support range;

[0035] 2. Delays in tracking - especially if too many landmarks are clustered on the same viewpoint; or

[0036] 3. Duplicate data – Progressive maps may contain too much similar content.

[0037] The examples provided in this disclosure also provide a method for clearing frames to be less than a preset number to limit the size of the locator map. It should be noted that in the different scenarios described above, the same or different methods of selecting / clearing frames can be used. It should also be noted that in the examples provided in this disclosure, the terms "locator map" and "sparse map" can be used interchangeably.

[0038] Figure 1 and Figure 2 Figure 2 shows pictures of reference scenes captured under different lighting conditions according to some examples. Figure 1 and Figure 2As shown, pictures 100 and 200 correspond to the same reference scene, but are captured under different lighting conditions. For example, picture 100 is captured during the day, and picture 200 is captured at night. The reference scene in pictures 100 and 200 includes table 102, two balls 104, four containers 106, plant 108, and other items. Due to the change in lighting conditions, some objects (e.g., four containers 106) may not be recognizable.

[0039] One way to enable AR experiences at any time of the day is to create two separate custom landmarks (as an example of a 3D model): one during the day and one during the night. However, this can be inconvenient, especially for lens creators, who would have to create two lenses, one for the daytime landmark and another for the night.

[0040] The examples in the present disclosure enable a single custom landmark to provide an AR experience at any time of the day and under a variety of lighting conditions. The example process includes: 1. Create a custom landmark during daytime - this operation generates a 3D mesh (as an example of a 3D data representation) and a locator map (as an example of a computer vision model, a position tracking data representation) of the landmark (e.g., as a reference surface or reference object) that is suitable for positioning and tracking during daytime (e.g., when it is created); 2. Revisit the landmark during dusk time. Check whether the landmark can be located, and then add the keyframes (as examples of frames, poses) and 3D points at dusk to the existing / daytime locator map. During this operation, the existing 3D mesh can be recreated or remain unchanged. 3. Optionally revisit the landmark during nighttime and perform the tasks discussed in 2, but using nighttime data.

[0041] In some examples, old or pre-existing frames in the original locator map and new keyframes at dusk or nighttime make the map size too large. Therefore, the example system selects / clears one or more target frames from the old and new keyframes. The method of selecting / clearing is described elsewhere in this disclosure. See, e.g. Fig.16 Operation 1610 of method 1600 in the embodiment of the present disclosure. It should be noted that in the examples provided in the present disclosure, the terms "selection" and "clearance" can be used interchangeably. For example, clearing a frame from a plurality of frames is similar to selecting other frames from a plurality of frames.

[0042] More specifically, in some examples, as new keyframes are added to the locator map during progressive updates, the total keyframe count may exceed the maximum size allowed for optimal positioning and tracking performance. To prevent the locator map from growing too large, a "keyframe selection" process is used to determine which existing keyframes can be removed without significantly affecting performance.

[0043] The keyframe selection process can analyze each keyframe in the locator map to determine a "utility score" that indicates how useful it is for positioning and tracking. Keyframes with low utility scores may be good candidates for removal. The utility score is calculated based on the following factors:

[0044] • Timestamp: Keyframes captured earlier in the past are less likely to represent the current state of the reference scene and receive a lower score. A preset time period (such as 6 to 12 months) may be used to determine whether a keyframe is too old.

[0045] Coverage: Keyframes that capture a portion of the reference scene that is already covered by other, newer keyframes receive lower scores. Coverage is determined by estimating the 3D area observed in each keyframe.

[0046] Quality: Keyframes that are blurry, poorly lit, or otherwise low quality receive lower scores. Image quality metrics such as sharpness, brightness, and feature detectability are used to determine keyframe quality.

[0047] Redundancy: Groups of keyframes that observe the same part of the reference scene from nearly the same device position receive lower scores. The degree of similarity between keyframe views is estimated using feature matching and epipolar geometry.

[0048] Before the total keyframe count reaches the maximum allowed, keyframes are selected for removal based on their utility scores. If adding new keyframes during a progressive update causes the total count to exceed the maximum again, additional keyframes are removed.

[0049] The keyframe selection process helps ensure that useful, high-quality keyframes that provide good coverage of the reference scene are retained in the locator map. By removing redundant, low-quality, or outdated keyframes, the locator map is kept at a size that is useful for fast, accurate positioning and tracking.

[0050] Figure 3 Pictures of reference scenes captured from different perspectives according to some examples are shown. Picture 300 may be captured at a common reference scene with pictures 100 and 200 but from a wider and higher perspective. In addition to table 102, ball 104, and plant 108, bottle 302 and balloon 304 are also captured.

[0051] Figure 4 A diagram of a locator map corresponding to a reference scene according to some examples is shown. Figure 4 As shown, the sparse map 400 includes a plurality of 3D sparse points 402 and a plurality of key frames 404. The sparse map 400 may correspond to Figure 3300 in the sparse map 400. Clusters of sparse points 402 in the sparse map 400 may correspond to objects in the image 300. For example, clusters 406, 408, and 410 correspond to balloon 304, bottle 302, and ball 104, respectively.

[0052] Each of keyframes 404 can correspond to a pose (e.g., position and / or orientation) of a capture device (e.g., a mobile device) and is associated with a set of sparse points 402. In some examples, Figure 4 The sparse map 400 in is created at a first time point t1.

[0053] Figure 5 A schematic diagram of an old sparse map and multiple new frames according to some examples is shown. As previously described, an old sparse map 400 including multiple 3D sparse points 402 and key frames 404 is created at a first time point t1. At a second time point t2, one or more new key frames 502 can be captured. Each new key frame in the new key frame 502 includes the position and / or orientation of the capture device. Pictures 504, 506, and 508 are examples of pictures captured by the capture device at the new key frame 502. For example, pictures 504, 506, and 508 correspond to balloon 304, ball 104, and bottle 302, respectively. The items in pictures 504, 506, and 508 may look different from the items in picture 300 because the new key frame 502 is different from the old key frame 404. However, it should not be restrictive. Some or all of the new key frames 502 may overlap with the old key frame 404.

[0054] In some examples, the new keyframe 502 has a coordinate system that is different from the coordinate system of the old keyframe 404 (e.g., the origin 512 of the new coordinate system is different from the origin 510 of the old coordinate system). Therefore, in order to add information about the new keyframe 502 to the old sparse map 400, the keyframe 502 is transformed from the new coordinate system 512 to the old coordinate system 510. Once the positioning model of the landmark is obtained, the locator / tracker system (e.g., simultaneous localization and mapping (SLAM) system, natural feature tracking (NFT) system) can determine the position and orientation of the camera when the image was captured based on the picture of the scene. Figure 6 Describe the transformation of coordinate system in detail.

[0055] Figure 6A schematic diagram of a transformation from a new coordinate system to an old coordinate system according to some examples is shown. In a 3D Cartesian coordinate system, a point (X, Y, Z) is defined relative to the origin (0, 0, 0) of the coordinate system. For example, a custom landmark constructed at time t1 is defined in its own coordinate system, where the origin 510 (0, 0, 0) is defined as the origin of its coordinate system. When the user revisits the custom landmark at a different time t2, the position and orientation of the user's capture device relative to the AR system can also be determined (at time t2) when the AR system of the user's capture device is turned on. If the locator / tracker can match the features / descriptors in the picture with the features / descriptors in the locator model of the custom landmark constructed at time t1, the position and orientation of the capture device can be determined relative to the locator model t1. The pose (e.g., position and orientation) of the camera (e.g., the camera that captures the balloon 304) at time t2 relative to the AR system (e.g., the SLAM tracking system of the camera) is defined as Pose2. If the same camera is successfully positioned using the localizer model t1, its pose relative to the model coordinate system is defined as Pose1.

[0056] From this, the relationship between the two coordinate systems Pose1 and Pose2 can be derived, because they belong to the same camera but are defined in different coordinate systems. The pose of the camera consists of the rotation R and the translation vector T defined by the reference coordinate system. R can be defined as a 3×3 matrix and T can be defined as a 3×1 matrix. The pose of the camera can be represented as the following 4×4 matrix:

[0057]

[0058] The pose of the camera can be defined as a transformation from a reference coordinate system to the camera coordinate system. For the example above, Pose1 defines the transformation from the localizer model to the camera, and Pose2 defines the transformation from the AR system to the camera. If the user wishes to compute the transformation from the AR system coordinate system at time t2 to the localizer model coordinate system at time t1, this can be done by computing the transformation between the two, as shown below:

[0059] (Pose1) -1 *Pose2

[0060] It should be noted that “transformation of sparse points between two coordinate systems” is used interchangeably with “alignment of sparse points generated from one coordinate system with another coordinate system” in the present disclosure.

[0061] Figure 7 and Figure 8 Two pictures covering different areas of a reference scene according to some examples are shown. Figure 7 and Figure 8As shown, images 700 and 800 are street views of the same street 702. For example, restaurant 704 appears in both images 700 and 800. Image 700 covers restaurant 704 and flower shop 706 to the left of restaurant 704, and image 800 covers restaurant 704 and office building 802 to the right of restaurant 704. In some examples, the method of the present disclosure can be used to create a wide locator map covering all flower shops 706, restaurants 704, and office buildings 802. For example, the above can be used to create a wide locator map covering all flower shops 706, restaurants 704, and office buildings 802. Figure 6 The method mentioned in determines the pose of the camera when capturing picture 800 and transforms it to the pose when capturing picture 700. The localizer map corresponding to picture 800 can then be transformed and combined with the localizer map corresponding to picture 700 to generate a wider localizer map. Details on how to use pictures 700 and 800 to generate a wider localizer map can be found in Figures 9 to 13 and its description.

[0062] Fig. 9 and Fig.10 Two sparse maps created corresponding to reference scenes according to some examples are shown. Similar to pictures 700 and 800 both including restaurant 704, sparse points 902 in sparse map 900 and sparse points 1002 in sparse map 1000 correspond to the same object, and each of sparse maps 900 and 1000 covers more items.

[0063] Fig.11 and Fig.12 Two illustrative updated sparse maps according to some examples are shown. Sparse map 1100 is a direct combination of sparse maps 900 and 1000 without a coordinate system transformation. Sparse map 1200 is a combination of sparse maps 900 and 1000 with a coordinate system transformation. Coordinate system transformation methods can be found elsewhere in this disclosure, see e.g. Figure 6 and its description. Fig.11 As shown, sparse points 1102 are mixed, and it is not possible to identify items that appear in both sparse maps 900 and 1000. In contrast, Fig.12 For example, sparse point 1202 reflects sparse map 1000, and sparse point 1204 reflects sparse map 900. Overlap region 1206 includes sparse points that reflect items that appear in both sparse maps 900 and 1000.

[0064] Fig.13 Schematic diagram of generating an updated sparse map according to some examples is shown. Fig.13As shown, picture 1302 includes multiple reference scenes, such as balloons, bottles, tables, shelves, and grid plant pots 1304. Sparse maps 1306 and 1308 are generated corresponding to the reference scenes in picture 1302. For example, sparse points 1310 in sparse map 1306 and sparse points 1314 in sparse map 1308 correspond to grid plant pots 1304. In some examples, sparse maps 1308 and 1306 are aligned with each other (e.g., by transformation of coordinate systems) so that sparse points 1310 and 1314 of the same grid plant pot 1304 overlap in the generated sparse map 1312 (collectively shown as sparse points 1316).

[0065] Fig.14 A flow chart showing a method 1400 of adding a customer landmark to a landmark menu according to some examples is shown. In some examples, the method 1400 is performed by a creation application described herein (such as a scan utility, studio application, other application 1806).

[0066] In operation 1402, the creation application presents an interface to instruct the user to add a landmark in its landmark menu. The landmark menu is a list of landmarks created, uploaded, or added by the user.

[0067] In operation 1404, the user interacts with the interface to start adding a landmark. For example, the interface includes an "add landmark" button, and the user clicks the "add landmark" button to start the remaining operations.

[0068] In operation 1406, the creation application presents an interface to instruct the user to input an ID and / or name of the landmark. The interface may include an input field for the user to input the ID and / or name.

[0069] In operation 1408, the user enters an ID and / or name of the landmark. The ID of the landmark may be unique and randomly assigned when the landmark is created, and the name of the landmark is given by the creator of the landmark.

[0070] In operation 1410, the creation application determines whether the ID and / or name is valid. If it is determined that the ID and / or name is invalid, the method 1400 proceeds to operation 1412 and prompts a message "Invalid ID / Name"; otherwise, the method 1400 proceeds to operation 1414, and the creation application saves the landmark to cloud storage. The cloud storage can be a public cloud storage or a private cloud storage. Alternatively or additionally, the landmark is downloaded locally to the user's device. In operation 1416, the creation application displays the landmark in the user's landmark menu.

[0071] Fig.15A flow chart illustrating a method 1500 for performing a progressive scan according to some examples is shown. In some examples, the method 1500 is performed by a creation application described herein (such as a scan utility, studio application, other application 1806).

[0072] In operation 1502 , an application presentation interface is created to instruct a user to begin progressive scanning while the user is viewing an existing landmark.

[0073] In operation 1504, the user interacts with the interface to start the progressive scan. In some examples, the interface includes a "progressive scan" button, and the user clicks the "progressive scan" button to start the remaining operations.

[0074] In operation 1506, the creation application downloads the landmarks and their keyframes to construct and display a 3D visual effect of the features and perspective of the reference scene. The 3D visual effect may include sparse points and / or a 3D grid.

[0075] In operation 1508, the creation application generates a progress meter. The progress meter indicates the progress of the progressive scan, for example, how many sparse points have been captured or updated, how many areas in the reference scene have been revisited. In some examples, the progress meter is updated during the progressive scan.

[0076] In operation 1510, the user starts scanning under new lighting or other physical conditions. For example, the user moves around the reference scene to take multiple pictures or videos with his capture device. In operation 1512, the creation application updates the 3D visual effects and progress meter.

[0077] In operation 1514, the creation application determines that a termination condition is satisfied. The termination condition includes, but is not limited to: a progress meter reaches a threshold, a termination instruction is received from a user, a duration has passed, a count of total sparse points collected or new sparse points reaches a threshold, a count of new keyframes generated reaches a threshold.

[0078] In operation 1516, the creation application generates a new landmark (or an updated landmark) based on the updated 3D vision. In operation 1518, the creation application presents an interface to instruct the user to test the new landmark or complete the progressive scan.

[0079] User experience can include the following:

[0080] A user may wish to incrementally update an existing custom landmark including a 3D model and locator map by capturing new keyframes and 3D points under different conditions. The user launches the custom landmark creation app and views a list of their existing custom landmarks. The user selects the custom landmark they want to update by entering its ID. It should be noted that the user who updates a custom landmark does not have to be the user who originally created it.

[0081] The custom landmark creation application prompts the user to capture new data for the selected custom landmark in order to begin the progressive update process. The user is first asked to position their capture device, such as a camera, relative to the existing custom landmark. This initializes the application to the correct position and orientation so that new keyframes and 3D points can be aligned correctly. Positioning is performed by detecting sparse points in the current camera image and matching them to sparse points in the existing locator map.

[0082] Once located, the existing 3D mesh of the custom landmark is displayed to the user to confirm that it appears to be aligned correctly. The user can choose to create a new 3D mesh for the custom landmark, which will replace the existing mesh, or leave the existing mesh unchanged.

[0083] The application then prompts the user to specify lighting conditions for capturing new data, such as good lighting, poor lighting, or no lighting. The user can then start capturing new keyframes by moving around the reference scene and capturing images or videos. As new keyframes are captured, the pose of the capture device is determined for each keyframe. 3D points are detected in each new keyframe and matched to determine their 3D positions. The new keyframes and 3D points are added to the locator map, and the 3D mesh can be recreated or left unchanged.

[0084] During the progressive update of a custom landmark, a decision can be made whether to reuse an existing 3D model (e.g., a 3D mesh) or generate a new 3D model from the latest keyframes and 3D points. Reusing an existing 3D model can be more efficient, while generating a new 3D model can produce higher quality results. The approach taken depends on the analysis of the existing 3D model and the latest keyframes.

[0085] An existing 3D model is suitable for reuse if the following conditions are met:

[0086] It is generated from a keyframe that captured the reference scene under similar conditions (e.g., same lighting and environment) as the latest keyframe. This indicates that the 3D model is still an accurate representation of the reference scene.

[0087] Comparing the sparse points from the latest keyframe to the 3D model shows minimal deviations. Small deviations indicate that the geometry of the reference scene has not changed significantly. Large deviations indicate that the 3D model needs to be updated.

[0088] The latest keyframe does not observe any new part of the reference scene that is not covered by the existing 3D model. If new areas are observed, the 3D model needs to be extended.

[0089] If the above conditions are met, the existing 3D model can be reused as is. The latest keyframe is aligned with the 3D model, but no changes are required to the model geometry. This approach is the most efficient because the 3D model does not need to be recreated.

[0090] However, a new 3D model can be generated if:

[0091] The latest keyframes were captured under very different conditions than those used to generate the existing 3D model. For example, an existing daytime 3D model will not accurately represent a nighttime keyframe.

[0092] There are large deviations between the sparse points in the latest keyframe and the existing 3D model, indicating that the reference scene geometry has changed significantly.

[0093] The latest keyframe observes new parts of the reference scene that are not covered by the existing 3D model. The 3D model needs to be extended to include these new areas.

[0094] Generating a new 3D model from the latest aligned keyframes and 3D points produces the highest quality results at the expense of additional processing time. The new 3D model can be a very good fit to the current state of the reference scene.

[0095] When the user is finished capturing new data, they instruct the user to complete the scan. The updated custom landmark (including the updated locator map and potentially the new 3D mesh) can then be tested to confirm that it is correctly positioned even in the new conditions. If satisfied with the results, the user can publish the updated custom landmark. The updated custom landmark is automatically starred so the user can easily revisit and test it again.

[0096] Progressive update processing allows a single custom landmark to be improved over time by capturing new keyframes and 3D points under different conditions. This provides an efficient way for custom landmarks to continue to provide a high-quality augmented reality experience as conditions change or more area is covered.

[0097] Fig.16 A flow chart showing a method 1600 for generating an updated locator map of a reference scene according to some examples is shown. In some examples, the method 1600 is performed by a creation application described herein (such as a scan utility, studio application, other application 1806).

[0098] In operation 1602 , a creation application obtains a preliminary sparse map of a reference scene, the preliminary sparse map comprising at least one preliminary keyframe, each preliminary keyframe being associated with a set of preliminary sparse points.

[0099] In operation 1604, the creation application instructs the user to capture a plurality of new sparse points on the reference scene using a capture device. In some examples, the new sparse points are captured at different times, under different lighting conditions, and / or cover different ranges.

[0100] In operation 1606 , the creation application determines at least one pose (eg, position and orientation) of the capture device associated with the capture of the plurality of new sparse points.

[0101] In operation 1608 , the creation application creates at least one new keyframe, wherein each new keyframe of the at least one new keyframe corresponds to one of the at least one gesture and is associated with a set of new sparse points of the plurality of new sparse points.

[0102] In operation 1610, the creation application selects one or more target frames from the at least one preliminary frame and the at least one new frame. In some examples, after determining that the count of the selected one or more target frames exceeds the maximum number of frames, the creation application clears the target frames from the selected one or more target frames so that the count of the selected target frames after clearing is less than or equal to the maximum number of frames.

[0103] One or more target frames may be randomly selected from at least one preliminary frame and at least one new frame. Alternatively, the selection of the one or more target frames may include: determining a timestamp corresponding to each of the at least one preliminary frame; removing old preliminary frames from the at least one preliminary frame to generate one or more remaining preliminary frames, wherein the timestamps of the old frames are earlier than a preset time point; and selecting one or more target frames from the one or more remaining preliminary frames and the at least one new frame.

[0104] In some examples, operation 1610 includes: clustering at least one preliminary frame and at least one new frame to generate one or more target clusters; and selecting at least one target frame from each of the one or more target clusters as one or more target frames. More specifically, there may be an isolated target cluster among the one or more target clusters, wherein the count of frames in the isolated target cluster is less than a threshold. Operation 1610 may also include removing the isolated target cluster from the one or more target clusters to generate at least one remaining target cluster; and selecting at least one target frame from each of the at least one remaining target clusters as one or more target frames.

[0105] In some examples, operation 1610 includes: displaying a visual representation related to at least one preliminary frame and at least one new frame on a graphical user interface (GUI) of a user device; receiving a selection operation from a user on the GUI; and selecting one or more target frames based on the selection operation from the user. Additionally or alternatively, operation 1610 includes determining a quality index for each of the at least one preliminary frame and each of the at least one new frame; and selecting one or more target frames from the at least one preliminary frame and the at least one new frame based on the quality index.

[0106] In operation 1612, the creation application generates an updated locator map based on the one or more target frames and the associated set of preliminary or new data points.

[0107] Fig.17 A flow chart illustrating a method 1700 of aligning an updated locator map with a preliminary 3D data representation of a reference scene is shown according to some examples. In some examples, the method 1700 is performed by a creation application described herein, such as a scan utility, studio application, other application 1806.

[0108] In operation 1702, the creation application obtains a preliminary 3D data representation (e.g., a 3D mesh) of a reference scene. In operation 1704, the creation application aligns a preliminary locator map with the preliminary 3D data representation. In operation 1706, the creation application transforms at least one new frame from a new coordinate system to an old coordinate system, wherein the old coordinate system corresponds to at least one preliminary frame in the preliminary locator map. In operation 1708, the creation application generates a 3D model of the reference scene based on the updated locator map and the preliminary 3D data representation.

[0109] Example alignment process:

[0110] When a user performs a progressive update on an existing custom landmark, new keyframes and 3D points are captured from the reference scene. These new keyframes and 3D points are initially in the capture device's coordinate system, called the "new coordinate system". Existing locator maps for custom landmarks have keyframes and 3D points in their own "old coordinate system".

[0111] In order to add new keyframes and 3D points to an existing locator map, they can be transformed from the new coordinate system to the old coordinate system. This is done through a process called "coordinate system alignment" or "spatial alignment".

[0112] The alignment process begins by detecting common sparse points, such as corners or edges, between at least one new keyframe and an existing keyframe in the locator map. The positions of these common sparse points in the new and old coordinate systems are used to calculate a spatial transformation, such as a rigid transformation, between the two coordinate systems.

[0113] The spatial transformation is then applied to all new keyframes and 3D points to transform their positions to the old coordinate system. The transformed new keyframes and 3D points can then be added to the existing locator map, and all data is now in a common coordinate system.

[0114] Since all keyframes and 3D points are in the same coordinate system, the updated localizer map can be used for localization and tracking as normal. By detecting sparse points in the current view and matching them with sparse points from the original localizer map or progressively updated keyframes, the capture device position and orientation can be accurately determined relative to the reference scene.

[0115] The alignment process allows incremental updates to be built on top of an existing locator map, without the need to generate an entirely new map from scratch. Over multiple updates, a single locator map accumulates keyframes and 3D points from different conditions, improving its ability to handle changes in lighting, environment, or device position. The end result is a custom landmark that continues to provide a high-quality experience even as the reference scene evolves over time.

[0116] Network computing environment

[0117] Fig.18 1 is a block diagram illustrating an example interactive system 1800 for facilitating interactions over a network (e.g., exchanging text messages, making text, audio, and video calls, or playing games). The interactive system 1800 includes a plurality of user systems 1802, each of which hosts a plurality of applications including an interactive client 1804 and other applications 1806, including a creation application as described above.

[0118] Each interactive client 1804 is communicatively coupled to other instances of the interactive client 1804 (e.g., hosted on respective other user systems 1802), the interactive server system 1810, and the third-party server 1812 via one or more communication networks including a network 1808 (e.g., the Internet). The interactive client 1804 may also communicate with a locally hosted application 1806 using an application program interface (API).

[0119] The custom landmark creation application provides an interface for users to create and update custom landmarks. To create a new custom landmark, the creation application presents an "Add New Landmark" option. Once selected, the creation application prompts the user to enter the ID and name of the new landmark. The creation application determines whether the ID is valid and unique. If so, the new landmark is saved to the memory and displayed in the user's landmark list. The creation application then instructs the user to capture an image or video of the reference scene. For each frame captured, the application detects 3D points that can be matched across frames to determine their 3D positions. The creation application also determines the pose of the capture device for each frame. The 3D points and device pose are used to generate a sparse map or locator map, which can determine the position and orientation of the capture device relative to the scene. 3D models, such as 3D meshes, are also generated based on the captured data.

[0120] Users can revisit and update custom landmarks over time by capturing new data under different conditions. The creation application presents an "Update Landmark" option for the selected custom landmark. The user is prompted to capture additional images or video, and new keyframes and 3D points are added to the locator map. The 3D model can be recreated or left unchanged.

[0121] To limit the size of the locator map, create an application that selects keyframes to remove so that the total keyframe count remains below a maximum value. The keyframes selected can be random, timestamp-based, clustered, quality-based, or selected by user input. An interface is provided for the user to review and modify the keyframe selection.

[0122] The updated placemark can then be tested to confirm it is correctly positioned. If satisfied, the user can publish the update. The updated placemark is marked with a star for easy revisiting.

[0123] Create apps that provide a progress indicator when generating or updating custom placemarks. The progress meter displays the number of keyframes, 3D points, or regions captured. Provide users with interface options to review, test, or recapture data for any part of the reference scene.

[0124] The custom landmark creation app implements an efficient workflow for generating and updating custom landmarks to provide high-quality augmented reality experiences in different conditions. Progressive update processing allows landmarks to be improved over time based on user feedback and new data.

[0125] Each user system 1802 may include multiple user devices, such as a mobile device 1814, a head wearable device 1816, and a computer client device 1818, which may be communicatively connected to exchange data and messages.

[0126] The interactive clients 1804 interact with other interactive clients 1804 and with an interactive server system 1810 via the network 1808. Data exchanged between the interactive clients 1804 (e.g., interactions 1820) and between the interactive clients 1804 and the interactive server system 1810 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0127] The interactive server system 1810 provides server-side functions to the interactive client 1804 via the network 1808. Although certain functions of the interactive system 1800 are described herein as being performed by the interactive client 1804 or the interactive server system 1810, it may be a design choice whether certain functions are located within the interactive client 1804 or within the interactive server system 1810. For example, it may be technically preferred to initially deploy certain technologies and functions within the interactive server system 1810, but later migrate the technologies and functions to the interactive client 1804 where the user system 1802 has sufficient processing power.

[0128] The interactive server system 1810 supports various services and operations provided to the interactive clients 1804. Such operations include sending data to the interactive clients 1804, receiving data from the interactive clients 1804, and processing data generated by the interactive clients 1804. The data may include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, entity relationship information, and live event information. The data exchange within the interactive system 1800 is activated and controlled by functions available through the user interface (UI) of the interactive client 1804.

[0129] Turning now specifically to the interaction server system 1810, an application program interface (API) server 1822 is coupled to and provides a programming interface to the interaction server 1824, making the functionality of the interaction server 1824 accessible to the interaction clients 1804, other applications 1806, and third-party servers 1812. The interaction server 1824 is communicatively coupled to a database server 1826, thereby facilitating access to a database 1828 that stores data associated with interactions processed by the interaction server 1824. Similarly, a web server 1830 is coupled to the interaction server 1824 and provides a web-based interface to the interaction server 1824. To this end, the web server 1830 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0130] The application program interface (API) server 1822 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 1824 and the user system 1802 (and, for example, the interaction client 1804 and other applications 1806) and the third-party server 1812. Specifically, the application program interface (API) server 1822 provides a set of interfaces (e.g., routines and protocols) that the interaction client 1804 and other applications 1806 can call or query to activate the functions of the interaction server 1824. The application program interface (API) server 1822 exposes various functions supported by the interaction server 1824, including account registration; login functionality; sending interaction data from a particular interaction client 1804 to another interaction client 1804 via the interaction server 1824; transferring media files (e.g., images or videos) from the interaction client 1804 to the interaction server 1824; setting up a media data collection (e.g., a story); retrieving a friend list of a user of the user system 1802; retrieving messages and content; adding and deleting entities (e.g., friends) to an entity relationship graph (e.g., entity graph 2010); locating friends in the entity relationship graph; and opening application events (e.g., related to the interaction client 1804).

[0131] Interactive server 1824 hosts multiple systems and subsystems, see below Fig.19 Give a description.

[0132] System Architecture

[0133] Fig.19 1800 is a block diagram showing additional details about the interactive system 1800 according to some examples. Specifically, the interactive system 1800 is shown as including an interactive client 1804 and an interactive server 1824. The interactive system 1800 includes a plurality of subsystems that are supported on the client side by the interactive client 1804 and on the server side by the interactive server 1824. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) can have components that enable the microservice subsystem to operate independently and communicate with other services. Example components of a microservice subsystem can include:

[0134] Functional logic: Functional logic implements the functionality of a microservice subsystem and represents the specific capabilities or functions provided by a microservice.

[0135] API interface: Microservices can communicate with each other through a well-defined API or interface using a lightweight protocol such as REST or messaging. The API interface defines the input and output of the microservice subsystem and how the microservice subsystem interacts with other microservice subsystems of the interactive system 1800.

[0136] Data storage: The microservice subsystem may be responsible for its own data storage, which may be in the form of a database, cache, or other storage mechanism (e.g., utilizing database server 1826 and database 1828). This enables the microservice subsystem to operate independently of other microservices of interactive system 1800.

[0137] Service discovery: A microservice subsystem can find and communicate with other microservice subsystems of the interactive system 1800. The service discovery mechanism enables a microservice subsystem to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0138] Monitoring and logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable tracking the health and performance of microservice subsystems.

[0139] In some examples, the interactive system 1800 can adopt a monolithic architecture, a service-oriented architecture (SOA), a function as a service (FaaS) architecture, or a modular architecture:

[0140] Example subsystems are discussed below.

[0141] Image processing system 1902 provides various functions that enable a user to capture and enhance (eg, annotate or otherwise modify or edit) media content associated with a message.

[0142] The camera system 1904 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system 1802 (e.g., directly or via operating system control) to modify and enhance the real-time images captured and displayed via the interactive client 1804.

[0143] The enhancement system 1906 provides functionality related to the generation and publication of enhancements (e.g., media overlays) for images captured in real time by the camera of the user system 1802 or retrieved from the memory of the user system 1802. For example, the enhancement system 1906 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for the interactive client 1804 for enhancing the real-time image received via the camera system 1904 or the stored image retrieved from the memory 2202 of the user system 1802. These enhancements are selected and presented to the user of the interactive client 1804 by the enhancement system 1906 based on some input and data, such as:

[0144] The geolocation of the user system 1802; and

[0145] Entity relationship information of users of the user system 1802.

[0146] Enhancement can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user systems 1802 for transmission in messages, or applied to video content such as video content streams or feeds sent from interactive clients 1804. Therefore, image processing system 1902 can interact with various subsystems of communication system 1908, and supports various subsystems of communication system 1908, such as messaging system 1910 and video communication system 1912.

[0147] Media overlays may include text or image data that can be superimposed on a photo taken by user system 1802 or a video stream produced by user system 1802. In some examples, media overlays may be location overlays (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In other examples, image processing system 1902 uses the geolocation of user system 1802 to identify media overlays including the name of a business at the geolocation of user system 1802. Media overlays may include other tags associated with the business. Media overlays may be stored in database 1828 and accessed by database server 1826.

[0148] The image processing system 1902 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify a context in which a particular media overlay should be provided to other users. The image processing system 1902 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0149] The augmented creation system 1914 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive clients 1804. The augmented creation system 1914 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0150] In some examples, the enhancement creation system 1914 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with a geographic location via a bidding process. For example, the enhancement creation system 1914 associates the media overlay of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

[0151] The communication system 1908 is responsible for enabling and processing various forms of communication and interaction within the interactive system 1800, and includes a messaging system 1910, an audio communication system 1916, and a video communication system 1912. The messaging system 1910 is responsible for enforcing temporary or time-limited access to content by the interactive clients 1804. The messaging system 1910 includes multiple timers (e.g., in a transient timer system) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive clients 1804 based on duration and display parameters associated with a message or a collection of messages (e.g., a story). The audio communication system 1916 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 1804. Similarly, the video communication system 1912 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 1804.

[0152] The user management system 1918 is operationally responsible for managing user data and profiles, and maintaining entity information (eg, stored in the entity table 2008, entity graph 2010, and profile data 2002) about users of the interactive system 1800 and relationships between users.

[0153] The collection management system 1920 is operationally responsible for managing a collection or collection of media (e.g., a collection of text, image video, and audio data). A collection of content (e.g., messages, including images, videos, text, and audio) can be organized into an "event gallery" or "event story". Such a collection can be made available within a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 1920 can also be responsible for publishing an icon that provides a notification of a specific collection to the user interface of the interactive client 1804. The collection management system 1920 includes a curation function that enables collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate a collection of content related to a specific event (e.g., deleting inappropriate content or redundant messages). In addition, the collection management system 1920 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such cases, the collection management system 1920 operates to automatically make payments to such users for use of their content.

[0154] The mapping system 1922 provides various geographic location (e.g., geolocation) functions and supports the presentation of map-based media content and messages by the interactive client 1804. For example, the mapping system 1922 enables the display of a user icon or avatar (e.g., stored in the profile data 2002) on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content (e.g., a collection of messages including photos and videos) generated by these friends. For example, on the map interface of the interactive client 1804, a message posted by the user to the interactive system 1800 from a particular geographic location can be displayed to the "friends" of a particular user within the context of that particular location on the map. The user can also share his or her location and status information with other users of the interactive system 1800 via the interactive client 1804 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to the selected user within the context of the map interface of the interactive client 1804.

[0155] The gaming system 1924 provides various gaming functions within the context of the interactive client 1804. The interactive client 1804 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the interactive client 1804 and played with other users of the interactive system 1800. The interactive system 1800 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation to such other users from the interactive client 1804. The interactive client 1804 also supports voice, video, and text messaging (e.g., chat) within the context of game play, provides leaderboards for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0156] The external resource system 1926 provides an interface for the interactive client 1804 to communicate with a remote server (e.g., a third-party server 1812) to launch or access an external resource (i.e., an application or applet). Each third-party server 1812 hosts, for example, an application based on a markup language (e.g., HTML5) or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application). The interactive client 1804 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 1812 associated with a web-based resource. The application hosted by the third-party server 1812 is programmed in JavaScript using a software development kit (SDK) provided by the interactive server 1824. The SDK includes an application program interface (API) with functions that can be called or activated by a web-based application. The interactive server 1824 hosts a JavasScript library that provides access to a given external resource to specific user data of the interactive client 1804. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0157] To integrate the functionality of the SDK into a web-based resource, the SDK is downloaded from the interactive server 1824 by the third-party server 1812, or the SDK is received by the third-party server 1812 in other ways. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 1804 into the web-based resource.

[0158] The SDK stored on the interactive server system 1810 effectively provides a bridge between external resources (e.g., applications 1806 or applet) and the interactive client 1804. This gives the user a seamless experience of communicating with other users on the interactive client 1804, while also retaining the look and feel of the interactive client 1804. In order to bridge the communication between the external resources and the interactive client 1804, the SDK facilitates the communication between the third-party server 1812 and the interactive client 1804. The bridging script running on the user system 1802 establishes two one-way communication channels between the external resources and the interactive client 1804. Messages are sent asynchronously between the external resources and the interactive client 1804 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0159] By using the SDK, not all information from the interactive client 1804 is shared with the third-party server 1812. The SDK limits which information is shared based on the needs of the external resource. Each third-party server 1812 provides an HTML5 file corresponding to a web-based external resource to the interactive server 1824. The interactive server 1824 can add a visual representation (e.g., a box design or other graphics) of the web-based external resource in the interactive client 1804. Once the user selects the visual representation or instructs the interactive client 1804 to access the features of the web-based external resource through the GUI of the interactive client 1804, the interactive client 1804 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0160] The interactive client 1804 presents a graphical user interface (e.g., a login page or title screen) for an external resource. During, before, or after presenting the login page or title screen, the interactive client 1804 determines whether the launched external resource has been previously authorized to access the user data of the interactive client 1804. In response to determining that the launched external resource has been previously authorized to access the user data of the interactive client 1804, the interactive client 1804 presents another graphical user interface of the external resource including the functions and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access the user data of the interactive client 1804, after displaying the login page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interactive client 1804 slides up a menu (e.g., animating the menu to emerge from the bottom of the screen to the middle or other part of the screen) for authorizing the external resource to access the user data. The menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, the interactive client 1804 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 1804. External resources are authorized by the interactive client 1804 to access user data under the OAuth 2 framework.

[0161] The interaction client 1804 controls the type of user data shared with the external resource based on the type of external resource that is authorized. For example, an external resource including a full-scale application (e.g., application 1806) is provided with access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics). As another example, an external resource including a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of a user, a three-dimensional avatar of a user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the look and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0162] The advertising system 1928 operationally enables third parties to purchase advertisements for presentation to end users via the interactive client 1804, and also handles the delivery and presentation of these advertisements.

[0163] The artificial intelligence and machine learning system 1930 provides various services to different subsystems within the interactive system 1800. For example, the artificial intelligence and machine learning system 1930 operates with the image processing system 1902 and the camera system 1904 to analyze images and extract information, such as objects, text, or faces. This information can then be used by the image processing system 1902 to enhance, filter, or manipulate images. The artificial intelligence and machine learning system 1930 can be used by the enhancement system 1906 to generate enhanced content and augmented reality experiences, such as adding virtual objects or animations to images of the real world. The communication system 1908 and the messaging system 1910 can use the artificial intelligence and machine learning system 1930 to analyze communication patterns and provide insights into how users interact with each other, as well as provide intelligent message classification and tagging, such as classifying messages based on emotions or topics. The artificial intelligence and machine learning system 1930 can also provide chatbot functionality to the message interactions 1820 between user systems 1802 and between user systems 1802 and the interactive server system 1810. The artificial intelligence and machine learning system 1930 may also cooperate with the audio communication system 1916 to provide speech recognition and natural language processing capabilities, enabling a user to interact with the interactive system 1800 using voice commands.

[0164] Data Architecture

[0165] Fig. 20 2 is a diagram illustrating a data structure 2000 that may be stored in a database 2004 of an interactive server system 1810 according to some examples. Although the contents of the database 2004 are illustrated as including a plurality of tables, it should be understood that data may be stored in other types of data structures (e.g., object-oriented databases).

[0166] Database 2004 includes message data stored in message table 2006. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Fig. 20 Describes additional details regarding information that may be included in a message and included within the message data stored in message table 2006.

[0167] The entity table 2008 stores entity data and is linked (e.g., referenced) to the entity graph 2010 and profile data 2002. Entities for which records are maintained within the entity table 2008 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity for which the interactive server system 1810 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0168] The entity graph 2010 stores information about relationships and associations between entities. By way of example only, such relationships may be social, professional (e.g., working at a common company or organization), interest-based, or activity-based. Some relationships between entities may be one-way, such as a subscription by an individual user to digital content of a business or publishing user (e.g., a newspaper or other digital media organization, or a brand). Other relationships may be two-way, such as a "friend" relationship between various users of the interactive system 1800.

[0169] Certain permissions and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friend relationship between various users) can include authorization for the publication of digital content items between various users, but certain restrictions or filters (e.g., based on content characteristics, location data, or time of day data) can be imposed on the publication of these digital content items. Similarly, a subscription relationship between a personal user and a business user can impose varying degrees of restrictions on the publication of digital content from a business user to a personal user, and can significantly limit or prevent the publication of digital content from a personal user to a business user. A specific user, as an example of an entity, can record certain restrictions in a record for the entity in the entity table 2008 (e.g., by means of privacy settings). Such privacy settings can be applied to all types of relationships in the context of the interactive system 1800, or can be selectively applied to certain types of relationships.

[0170] Profile data 2002 stores multiple types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, profile data 2002 can be selectively used and presented to other users of the interactive system 1800. In the case where the entity is a person, profile data 2002 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of a message transmitted via the interactive system 1800 and on a map interface displayed to other users by the interactive client 1804. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can choose to transmit at a particular time.

[0171] Where the entity is a group, profile data 2002 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (eg, notifications) for the relevant group.

[0172] Database 2004 also stores enhancement data, such as overlays or filters, in enhancement table 2012. Enhancement data is associated with and applied to videos (data for videos is stored in video table 2014) and images (data for images is stored in image table 2016).

[0173] In some examples, filters are displayed as overlays on an image or video during presentation to a recipient user. Filters can be of various types, including filters selected by a user from a set of filters presented to a sending user by interactive client 1804 when the sending user is composing a message. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a sending user based on geolocation. For example, geolocation filters specific to a nearby or special location can be presented by interactive client 1804 within a user interface based on geolocation information determined by a global positioning system (GPS) unit of user system 1802.

[0174] Another type of filter is a data filter, which can be selectively presented to the sending user by the interaction client 1804 based on other input or information collected by the user system 1802 during the message creation process. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user system 1802, or the current time.

[0175] Other augmented data that may be stored in the image table 2016 include augmented reality content items (eg, corresponding to an application "lens" or augmented reality experience). Augmented reality content items may be real-time special effects and sounds that may be added to an image or video.

[0176] The collection table 2018 stores data about a collection of messages and associated images, videos, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 2008). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the interactive client 1804 may include an icon that the user can select to enable the sending user to add specific content to his or her personal story.

[0177] The collection may also constitute a "live story", which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a co-located event at a particular time may be presented with the option of contributing content to a particular live story, for example, via the user interface of the interactive client 1804. Live stories may be identified to the user by the interactive client 1804 based on his or her location. The end result is a "live story" told from a group perspective.

[0178] Another type of content collection is called a "location story" that enables users whose user systems 1802 are located within a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributions to location stories may employ secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0179] As mentioned above, video table 2014 stores video data that, in some examples, is associated with messages for which records are maintained within message table 2006. Similarly, image table 2016 stores image data associated with messages whose message data is stored in entity table 2008. Entity table 2008 may associate various enhancements from enhancement table 2012 with various images and videos stored in image table 2016 and video table 2014.

[0180] Data communication architecture

[0181] Fig.21 is a schematic diagram illustrating the structure of a message 2100 according to some examples, the message 2100 being generated by an interactive client 1804 for transmission to another interactive client 1804 via an interactive server 1824. The content of a particular message 2100 is used to populate a message table 2006 stored in a database 2004 accessible by the interactive server 1824. Similarly, the content of the message 2100 is stored in memory as "in-transit" or "in-flight" data of the user system 1802 or the interactive server 1824. The message 2100 is shown as including the following example components:

[0182] Message identifier 2102 : A unique identifier that identifies the message 2100 .

[0183] Message text payload 2104 : Text to be generated by the user via the user interface of the user system 1802 and included in the message 2100 .

[0184] Message Image Payload 2106: Image data captured by the camera component of the user system 1802 or retrieved from the memory component of the user system 1802 and included in the message 2100. Image data for a message 2100 sent or received may be stored in the Image Table 2016.

[0185] Message Video Payload 2108: Video data captured by the camera component or retrieved from the memory component of the user system 1802 and included in the message 2100. The video data for a message 2100 sent or received may be stored in the Image Table 2016.

[0186] Message audio payload 2110 : Audio data captured by a microphone or retrieved from a memory component of the user system 1802 and included in the message 2100 .

[0187] Message enhancement data 2112: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 2106, message video payload 2108, or message audio payload 2110 of the message 2100. The enhancement data for a sent or received message 2100 may be stored in the enhancement table 2012.

[0188] · Message duration parameter 2114: A parameter value indicating the amount of time in seconds that the content of the message (e.g., message image payload 2106, message video payload 2108, message audio payload 2110) is to be presented to the user via the interactive client 1804 or made accessible to the user.

[0189] Message geolocation parameters 2116: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 2116 values ​​may be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within a message image payload 2106 or a specific video in a message video payload 2108).

[0190] Message story identifier 2118: An identifier value that identifies one or more content collections (e.g., "Stories" identified in collection table 2018) associated with a particular content item in message image payload 2106 of message 2100. For example, multiple images within message image payload 2106 may each be associated with multiple content collections using an identifier value.

[0191] Message tags 2120: Each message 2100 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 2106 depicts an animal (e.g., a lion), a tag value may be included within the message tags 2120 indicating the relevant animal. The tag value may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0192] • Message sender identifier 2122: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the user system 1802 on which the message 2100 was generated and from which the message 2100 was sent.

[0193] • Message recipient identifier 2124: Indicates an identifier of the user of user system 1802 to which message 2100 is addressed (eg, a messaging system identifier, an email address, or a device identifier).

[0194] The content (e.g., value) of each component of message 2100 may be a pointer to a location in a table where the content data value is stored. For example, the image value in message image payload 2106 may be a pointer (or address) to a location in image table 2016. Similarly, the value in message video payload 2108 may point to data stored in image table 2016, the value stored in message enhancement data 2112 may point to data stored in enhancement table 2012, the value stored in message story identifier 2118 may point to data stored in collection table 2018, and the values ​​stored in message sender identifier 2122 and message recipient identifier 2124 may point to user records stored in entity table 2008.

[0195] System with head wearable device

[0196] Fig. 22 A system 2200 is shown including a head wearable device 1816 with a selector input device according to some examples. Fig. 22 is a high-level functional block diagram of an example head wearable device 1816 communicatively coupled to a mobile device 1814 and various server systems 2204 (e.g., an interactive server system 1810) via various networks 1808.

[0197] The head wearable device 1816 includes one or more cameras, each of which may be, for example, a visible light camera 2206 , an infrared transmitter 2208 , and an infrared camera 2210 .

[0198] The mobile device 1814 is connected to the head wearable device 1816 using both the low power wireless connection 2212 and the high speed wireless connection 2214. The mobile device 1814 is also connected to the server system 2204 and the network 2216.

[0199] The head wearable device 1816 also includes two image displays in the image display 2218 of the optical assembly. The two image displays 2218 of the optical assembly include an image display associated with the left lateral side of the head wearable device 1816 and an image display associated with the right lateral side of the head wearable device 1816. The head wearable device 1816 also includes an image display driver 2220, an image processor 2222, a low power circuit 2224, and a high speed circuit 2226. The image display 2218 of the optical assembly is used to present images and videos to a user of the head wearable device 1816, including images that may include a graphical user interface.

[0200] The image display driver 2220 commands and controls the image display 2218 of the optical assembly. The image display driver 2220 can deliver the image data directly to the image display 2218 of the optical assembly for presentation or can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., while the static image data can be formatted according to a compression format such as Portable Network Graphics Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF) or Exchangeable Image File Format (EXIF), etc.

[0201] The head wearable device 1816 includes a frame and rods (or temples) extending from lateral sides of the frame. The head wearable device 1816 also includes a user input device 2228 (e.g., a touch sensor or a push button) including an input surface on the head wearable device 1816. The user input device 2228 (e.g., a touch sensor or a push button) is used to receive input selections from a user for manipulating a graphical user interface of a presented image.

[0202] Fig. 22The components of the head wearable device 1816 shown in are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in chunks, frames, hinges, or nose bridges of the head wearable device 1816. The left visible light camera and the right visible light camera 2206 may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0203] The head wearable device 1816 includes a memory 2202 that stores instructions for performing a subset or all of the functions described herein. The memory 2202 may also include a storage device.

[0204] like Fig. 22 As shown, the high-speed circuit 2226 includes a high-speed processor 2230, a memory 2202, and a high-speed wireless circuit 2232. In some examples, the image display driver 2220 is coupled to the high-speed circuit 2226 and is operated by the high-speed processor 2230 to drive the left image display and the right image display in the image display 2218 of the optical assembly. The high-speed processor 2230 can be any processor capable of managing high-speed communications and operations of any general computing system required by the head wearable device 1816. The high-speed processor 2230 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) on the high-speed wireless connection 2214 using the high-speed wireless circuit 2232. In some examples, the high-speed processor 2230 executes an operating system (e.g., a LINUX operating system) or other such operating system of the head wearable device 1816, and the operating system is stored in the memory 2202 for execution. In addition to any other duties, the high-speed processor 2230 that executes the software architecture of the head wearable device 1816 is used to manage data transmission with the high-speed wireless circuit 2232. In some examples, high-speed wireless circuitry 2232 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communications standard, also referred to herein as In some examples, high-speed wireless circuitry 2232 may implement other high-speed communication standards.

[0205] The low power wireless circuit 2234 and the high speed wireless circuit 2232 of the head wearable device 1816 may include a short range transceiver (Bluetooth TM ) and a wireless WAN transceiver, a wireless LAN transceiver, or a WAN transceiver (e.g., cellular or ). Mobile device 1814, including a transceiver for communicating via low power wireless connection 2212 and high speed wireless connection 2214, may be implemented using details of the architecture of head wearable device 1816, as may other elements of network 2216.

[0206] The memory 2202 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 2206, the infrared camera 2210, and the image processor 2222, and the images generated to be displayed on the image display 2218 in the optical assembly through the image display driver 2220. Although the memory 2202 is shown as being integrated with the high-speed circuit 2226, in some examples, the memory 2202 can be a separate independent element of the head wearable device 1816. In some such examples, electrical wiring can provide a connection from the image processor 2222 or the low-power processor 2236 to the memory 2202 through a chip including the high-speed processor 2230. In some examples, the high-speed processor 2230 can manage the addressing of the memory 2202 so that the low-power processor 2236 will start the high-speed processor 2230 at any time when a read or write operation involving the memory 2202 is required.

[0207] like Fig. 22 As shown, the low-power processor 2236 or the high-speed processor 2230 of the head wearable device 1816 can be coupled to a camera device (a visible light camera device 2206, an infrared emitter 2208 or an infrared camera device 2210), an image display driver 2220, a user input device 2228 (for example, a touch sensor or a push button), and a memory 2202.

[0208] The head wearable device 1816 is connected to a host computer. For example, the head wearable device 1816 is paired with the mobile device 1814 via a high-speed wireless connection 2214 or connected to the server system 2204 via a network 2216. The server system 2204 can be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the mobile device 1814 and the head wearable device 1816 through the network 2216.

[0209] The mobile device 1814 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via a network 2216, a low power wireless connection 2212, or a high speed wireless connection 2214. The mobile device 1814 may further store at least a portion of the instructions in a memory of the mobile device 1814 memory to implement the functions described herein.

[0210] The output components of the head wearable device 1816 include visual components, such as a display (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The image display of the optical assembly is driven by the image display driver 2220. The output components of the head wearable device 1816 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the head wearable device 1816, the mobile device 1814, and the server system 2204 (e.g., user input devices 2228) may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touch pads, tracking balls, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0211] The head wearable device 1816 may also include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 1816. For example, the peripheral elements may include any I / O components, including output components, motion components, position components, or any other such components described herein.

[0212] For example, biometric components include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The biometric components may include a brain-computer interface (BMI) system that allows communication between the brain and an external device or machine. This can be achieved by recording brain activity data, converting the data into a format that can be understood by a computer, and then using the resulting signals to control the device or machine.

[0213] Example types of BMI technologies include:

[0214] Electroencephalogram (EEG)-based BMI, which uses electrodes placed on the scalp to record electrical activity in the brain.

[0215] • Invasive BMIs, which use electrodes surgically implanted in the brain.

[0216] Optogenetic BMIs, which use light to control the activity of specific nerve cells in the brain.

[0217] Any biometric data collected by the biometric component is captured and stored only with the user's approval and is deleted upon user request. In addition, such biometric data may be used for very limited purposes (e.g., identification verification). In order to ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if any). Any use of biometric data may be strictly limited to identification verification purposes, and the biometric data is not shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0218] The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The position component includes a positioning sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a Wi-Fi or Bluetooth receiver for generating positioning system coordinates, and a positioning sensor component for generating a positioning system coordinate. TM transceiver, altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), orientation sensor component (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the mobile device 1814 through the low-power wireless connection 2212 and the high-speed wireless connection 2214 via the low-power wireless circuit 2234 or the high-speed wireless circuit 2232.

[0219] Machine Architecture

[0220] Fig.232300 within which instructions 2302 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 2300 to perform any one or more of the methods discussed herein. For example, the instructions 2302 may cause the machine 2300 to perform any one or more of the methods described herein. The instructions 2302 convert a general purpose, unprogrammed machine 2300 into a specific machine 2300 that is programmed to perform the functions described and illustrated in the manner described. The machine 2300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 2300 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 2300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 2302 specifying actions to be taken by the machine 2300. In addition, although only a single machine 2300 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 2302 to perform any one or more of the methods discussed herein. For example, the machine 2300 may include any of the user system 1802 or a plurality of server devices that form part of the interactive server system 1810. In some examples, the machine 2300 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side, and certain operations of the particular method or algorithm are performed on the client side.

[0221] The machine 2300 may include a processor 2304, a memory 2306, and an input / output (I / O) component 2308 that may be configured to communicate with each other via a bus 2310. In an example, the processor 2304 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 2312 and a processor 2314 that execute instructions 2302. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.23 Multiple processors 2304 are shown, but the machine 2300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0222] The memory 2306 includes a main memory 2316, a static memory 2318, and a storage unit 2320, which can be accessed by the processor 2304 via the bus 2310. The main memory 2306, the static memory 2318, and the storage unit 2320 store instructions 2302 that implement any one or more of the methods or functions described herein. The instructions 2302 may also reside, completely or partially, within the main memory 2316, within the static memory 2318, within the machine-readable medium 2322 within the storage unit 2320, within at least one of the processors 2304 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 2300.

[0223] I / O components 2308 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 2308 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be appreciated that I / O components 2308 may include Fig.2323. In various examples, the I / O components 2308 may include a user output component 2324 and a user input component 2326. The user output component 2324 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 2326 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0224] In other examples, the I / O component 2308 may include a biometric component 2328, a motion component 2330, an environmental component 2332, or a position component 2334, as well as a wide range of other components. For example, the biometric component 2328 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The biometric component may include a brain-computer interface (BMI) system that allows communication between the brain and an external device or machine. This can be achieved by recording brain activity data, converting the data into a format that can be understood by a computer, and then using the resulting signal to control the device or machine.

[0225] Example types of BMI technologies include:

[0226] Electroencephalogram (EEG)-based BMI, which uses electrodes placed on the scalp to record electrical activity in the brain.

[0227] • Invasive BMIs, which use electrodes surgically implanted in the brain.

[0228] Optogenetic BMIs, which use light to control the activity of specific nerve cells in the brain.

[0229] Any biometric data collected by the biometric component is captured and stored only with the user's approval and is deleted upon user request. In addition, such biometric data may be used for very limited purposes (e.g., identification verification). In order to ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if any). Any use of biometric data may be strictly limited to identification verification purposes, and the data is not shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0230] The motion component 2330 includes an acceleration sensor component (eg, an accelerometer), a gravity sensor component, and a rotation sensor component (eg, a gyroscope).

[0231] Environmental components 2332 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0232] With respect to cameras, user system 1802 may have a camera system including, for example, a front-facing camera on a front surface of user system 1802 and a rear-facing camera on a rear surface of user system 1802. The front-facing camera may, for example, be used to capture still images and videos of a user of user system 1802 (e.g., “selfies”), which may then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera mode, which images are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, user system 1802 may also include a 360° camera for capturing 360° photos and videos.

[0233] Additionally, the camera system of the user system 1802 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the user system 1802. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0234] The location component 2334 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0235] Various technologies may be used to implement communications. I / O components 2308 also include communications components 2336 that are operable to couple machine 2300 to network 2338 or device 2340 via corresponding couplings or connections. For example, communications components 2336 may include a network interface component or another suitable device that interfaces with network 2338. In other examples, communications components 2336 may include wired communications components, wireless communications components, cellular communications components, near field communications (NFC) components, Parts (e.g. Low energy consumption), Components and other communication components for providing communication via other forms. Device 2340 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0236] In addition, the communication component 2336 can detect an identifier, or include a component operable to detect an identifier. For example, the communication component 2336 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting a one-dimensional barcode such as a Universal Product Code (UPC) barcode, a barcode such as a Quick Response (QR) code, an Aztec code, a Data Matrix, a Dataglyph, or a similar barcode). TM ), MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcode, multi-dimensional barcode and other optical codes) or acoustic detection components (e.g., microphones for audio signals for identifying tags). In addition, various information can be obtained via communication component 2336, such as positioning via Internet Protocol (IP) geolocation, Positioning obtained by signal triangulation, positioning obtained by detecting NFC beacon signals that can indicate a specific position, etc.

[0237] Various memories (e.g., main memory 2316, static memory 2318, and memory of processor 2304) and storage unit 2320 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 2302) when executed by processor 2304 cause various operations to implement the disclosed examples.

[0238] Instructions 2302 may be sent or received via a network interface device (e.g., a network interface component included in communications component 2336) using a transmission medium and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 2338. Similarly, instructions 2302 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with device 2340 using a transmission medium.

[0239] Software Architecture

[0240] Fig.24 2400 is a block diagram illustrating a software architecture 2402 that may be installed on any one or more of the devices described herein. The software architecture 2402 is supported by hardware such as a machine 2404 including a processor 2406, a memory 2408, and an I / O component 2410. In this example, the software architecture 2402 may be conceptualized as a stack of layers, each of which provides specific functionality. The software architecture 2402 includes layers such as an operating system 2412, a library 2414, a framework 2416, and an application 2418. In operation, the application 2418 invokes an API call 2420 through the software stack and receives a message 2422 in response to the API call 2420.

[0241] The operating system 2412 manages hardware resources and provides public services. The operating system 2412 includes, for example, a kernel 2424, a service 2426, and a driver 2428. The kernel 2424 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 2424 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. The service 2426 can provide other public services to other software layers. The driver 2428 is responsible for controlling or interfacing with the underlying hardware. For example, the driver 2428 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0242] The libraries 2414 provide a common low-level infrastructure used by the applications 2418. The libraries 2414 may include a system library 2430 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 2414 may include an API library 2432, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 2414 may also include various other libraries 2434 to provide many other APIs to the application 2418 .

[0243] The framework 2416 provides a common high-level infrastructure used by the applications 2418. For example, the framework 2416 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 2416 can provide a wide range of other APIs that can be used by the applications 2418, some of which can be specific to a particular operating system or platform.

[0244] In an example, applications 2418 may include a home application 2436, a contacts application 2438, a browser application 2440, a book reader application 2442, a positioning application 2444, a media application 2446, a messaging application 2448, a game application 2450, and a variety of other applications such as third-party applications 2452. Applications 2418 are programs that execute functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 2418 constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C language or assembly language). In a specific example, third-party applications 2452 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the mobile phone or another mobile operating system. In this example, the third-party application 2452 can activate the API call 2420 provided by the operating system 2412 to facilitate the functions described in this article.

[0245] in conclusion

[0246] Examples provided in this disclosure relate to methods and systems for progressively updating custom landmarks to improve positioning and tracking performance. Custom landmarks can refer to 3D models and locator maps built for real-world reference scenes (such as rooms, buildings, or objects). Custom landmarks can be used to determine the position and orientation of a capture device (such as a camera) relative to the reference scene.

[0247] The described methods and systems allow for progressive updating of custom landmarks when revisiting a reference scene. This can be useful when viewing a reference scene from a wider range of locations under different conditions (such as different lighting), or if the surroundings have changed. Progressively updating custom landmarks adds new keyframes and 3D points to the existing locator map to expand its coverage and improve positioning under various conditions.

[0248] To generate an updated locator map, new keyframes and 3D points are captured from the reference scene. The pose of the capture device is determined for each new keyframe. One or more target keyframes are then selected from the new keyframes and the existing keyframes in the preliminary locator map. The selected target keyframes and their associated 3D points are used to generate an updated locator map. The updated locator map can then be aligned with the existing 3D model of the reference scene.

[0249] The example method also describes controlling the size of the updated locator map by purging keyframes from the preliminary locator map and / or new keyframes to keep the total keyframe count below a maximum number. Keyframes may be randomly selected for purging based on timestamps, by clustering keyframes, based on quality, or by user input.

[0250] An updated 3D model of the reference scene can be generated based on the aligned updated locator map and the existing 3D model. The updated 3D model and the locator map can then be used to provide improved positioning and tracking of the capture device relative to the reference scene.

[0251] Example

[0252] Example 1 is a method for generating an updated locator map of a reference scene, the method comprising: obtaining a preliminary locator map of the reference scene, the preliminary locator map comprising preliminary frames, each preliminary frame being associated with a set of preliminary data points; capturing a plurality of new data points on the reference scene using a capture device; determining a pose of the capture device associated with the capture of the plurality of new data points; creating a new frame, the new frame corresponding to the pose and associated with a set of new data points among the plurality of new data points; selecting one or more target frames from the preliminary frames and the new frames; and generating an updated locator map based on the one or more target frames and the associated preliminary data points or groups of new data points.

[0253] In Example 2, the subject matter described in Example 1 includes determining that a count of one or more selected target frames exceeds a maximum number of frames; and clearing target frames from the selected one or more target frames so that the count of the selected target frames after clearing is less than or equal to the maximum number of frames.

[0254] In Example 3, the subject matter of Examples 1-2 includes, wherein selecting one or more target frames from the preliminary frame and the new frame includes: randomly selecting one or more target frames from the preliminary frame and the new frame.

[0255] In Example 4, the subject matter described in Examples 1 to 3 includes, wherein selecting one or more target frames from the preliminary frames and the new frames includes: determining a timestamp corresponding to each of the preliminary frames; removing old preliminary frames from the preliminary frames to generate one or more remaining preliminary frames, wherein the timestamps of the old frames are earlier than a preset time point; and selecting one or more target frames from the one or more remaining preliminary frames and the new frames.

[0256] In Example 5, the subject matter described in Examples 1 to 4 includes, wherein selecting one or more target frames from the preliminary frames and the new frames includes: clustering the preliminary frames and the new frames to generate one or more target clusters; and selecting a target frame from each of the one or more target clusters as one or more target frames.

[0257] In Example 6, the subject matter described in Example 5 includes determining an isolated target cluster from one or more target clusters, wherein a count of frames in the isolated target cluster is less than a threshold; removing the isolated target cluster from the one or more target clusters to generate remaining target clusters; and selecting a target frame from each of the remaining target clusters as one or more target frames.

[0258] In Example 7, the subject matter described in Examples 1 to 6 includes, wherein selecting one or more target frames from the preliminary frames and the new frames includes: determining a quality index for each of the preliminary frames and each of the new frames; and selecting one or more target frames from the preliminary frames and the new frames based on the quality index.

[0259] In Example 8, the subject matter described in Examples 1 to 7 includes, wherein selecting one or more target frames from the preliminary frames and the new frames includes: displaying a visual representation related to the preliminary frames and the new frames on a graphical user interface (GUI) of a user device; receiving a selection operation from a user on the GUI; and selecting one or more target frames based on the selection operation from the user.

[0260] In Example 9, the subject matter of Examples 1 to 8 includes acquiring a 3D data representation of a reference scene; and generating a 3D model of the reference scene based on the 3D data representation of the reference scene and the updated locator map.

[0261] In Example 10, the subject matter of Example 9 includes the 3D data representation of the reference scene and the updated locator map further comprising: aligning the updated locator map with the 3D data representation; and generating a 3D model of the reference scene based on the aligned updated locator map and 3D data representation.

[0262] In Example 11, the subject matter of Example 10 includes the 3D data representation comprising: aligning the preliminary locator map with the 3D data representation; and transforming the new frame from the new coordinate system to the old coordinate system, wherein the old coordinate system corresponds to the preliminary frame.

[0263] In Example 12, the subject matter of Examples 1 to 11 includes wherein determining a pose of the capture device associated with capturing the plurality of new data points comprises recording the pose of the capture device using a recording device.

[0264] In Example 13, the subject matter of Examples 1 to 12 includes, wherein determining a pose of the capture device associated with capturing the plurality of new data points comprises: implementing a preliminary locator map to determine a pose of the capture device based on the plurality of new data points.

[0265] Example 14 is a computing device comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the device to: obtain a preliminary locator map of a reference scene, the preliminary locator map comprising preliminary frames, each preliminary frame being associated with a set of preliminary data points; capture multiple new data points on the reference scene using a capture device; determine a pose of the capture device associated with the capture of the multiple new data points; create new frames, each of the new frames corresponding to one of the poses and associated with a set of new data points in the multiple new data points; select one or more target frames from the preliminary frames and the new frames; and generate an updated locator map based on the one or more target frames and the associated preliminary data points or groups of new data points.

[0266] In Example 15, the subject matter described in Example 14 includes, wherein the instructions further configure the device to: determine that the count of one or more selected target frames exceeds a maximum number of frames; and clear target frames from the selected one or more target frames so that the count of the selected target frames after clearing is less than or equal to the maximum number of frames.

[0267] In Example 16, the subject matter described in Examples 14 to 15 includes, wherein, in order to select one or more target frames from preliminary frames and new frames, the instructions configure the device to: determine a timestamp corresponding to each of the preliminary frames; remove old preliminary frames from the preliminary frames to generate one or more remaining preliminary frames, wherein the timestamps of the old frames are earlier than a preset time point; and select one or more target frames from the one or more remaining preliminary frames and the new frames.

[0268] In Example 17, the subject matter of Example 16 includes, wherein the instructions further configure the apparatus to: obtain a 3D data representation of a reference scene; and generate a 3D model of the reference scene based on the 3D data representation of the reference scene and the updated locator map.

[0269] In Example 18, the subject matter of Example 17 includes a 3D data representation of a reference scene and an updated locator map, and the instructions further configure the device to: align the updated locator map with the 3D data representation; and generate a 3D model of the reference scene based on the aligned updated locator map and 3D data representation.

[0270] In Example 19, the subject matter of Example 18 includes the 3D data representation, the instructions configuring the device to: align the preliminary locator map with the 3D data representation; and transform the new frame from the new coordinate system to the old coordinate system, wherein the old coordinate system corresponds to the preliminary frame.

[0271] Example 20 is a non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: obtain a preliminary locator map of a reference scene, the preliminary locator map comprising preliminary frames, each preliminary frame being associated with a set of preliminary data points; capture a plurality of new data points on the reference scene using a capture device; determine a pose of the capture device associated with the capture of the plurality of new data points; create new frames, each of the new frames corresponding to one of the poses and associated with a set of new data points in the plurality of new data points; select one or more target frames from the preliminary frames and the new frames; and generate an updated locator map based on the one or more target frames and the associated preliminary data points or groups of new data points.

[0272] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations for implementing any one of Examples 1 to 20.

[0273] Example 22 is a device comprising a device implementing any one of Examples 1 to 20.

[0274] Example 23 is a system implementing any one of Examples 1 to 20.

[0275] Example 24 is a method of implementing any one of Examples 1 to 20.

[0276] Glossary

[0277] A “locator map” may refer to a map used for positioning and tracking. Examples of locator maps may include sparse maps, dense maps, and point cloud maps.

[0278] “CV model” may refer to a computer vision model. Examples of CV models may include a simultaneous localization and mapping (SLAM) model, a visual inertial odometry (VIO) model, and a structure from motion (SfM) model.

[0279] A "3D data representation" may refer to a 3D representation of an object or environment. Examples of 3D data representations may include a 3D mesh, a point cloud, a depth map.

[0280] A "reference scene" may refer to a scene that includes an object, surface, or environment. Examples of reference scenes may include a room, a building facade, a landmark.

[0281] A "capture device" may refer to a device used to capture data. Examples of capture devices may include cameras (RGB, RGB-D, stereo), 3D scanners.

[0282] "Pose" may refer to the position and orientation of a capture device. Examples of poses may include translation parameters and rotation parameters.

[0283] "Frame" may refer to data captured at a point in time. Examples of frames may include image frames, point cloud frames.

[0284] "Component" may refer to a logic or device, a physical entity with the following boundaries, which is defined by a function or subroutine call, a branch point, an API, or other technical definitions that provide partitioning or modularization for a specific processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and a part of a program that generally performs a specific function of a related function. A component may constitute a software component (e.g., a code implemented on a machine-readable medium) or a hardware component. "Hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured by software (e.g., an application or an application part) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine), which is uniquely customized to perform the function of the configuration and is no longer a general-purpose processor. It will be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit to implement the hardware component. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. The software configures a specific one or more processors accordingly, such as to constitute a specific hardware component at one time and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., by appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described in this article can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operations in the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some of the operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographic locations.

[0285] "Computer-readable storage media" may refer to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier waves / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0286] "Ephemeral message" may refer to a message that is accessible for a time-limited duration. An ephemeral message may be text, an image, a video, etc. The access time of an ephemeral message may be set by the message sender. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

[0287] "Machine storage media" may refer to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media", "device storage media", and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media", "computer storage media", and "device storage media" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media".

[0288] “Non-transitory computer-readable storage medium” may refer to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0289] "Signal medium" may refer to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A method for generating an updated locator map of a reference scene, the method comprising: obtaining a preliminary locator map of the reference scene, the preliminary locator map comprising preliminary frames, each of the preliminary frames being associated with a set of preliminary data points; capturing a plurality of new data points on the reference scene using a capture device; determining a pose of the capture device associated with capture of the plurality of new data points; creating a new frame corresponding to the pose and associated with a set of new data points from the plurality of new data points; selecting one or more target frames from the preliminary frames and the new frames; as well as The updated locator map is generated based on the one or more target frames and the associated set of preliminary or new data points.

2. The method according to claim 1, further comprising: determining that a count of the selected one or more target frames exceeds a maximum number of frames; as well as Target frames are cleared from the selected one or more target frames so that a count of the selected target frames after clearing is less than or equal to the maximum number of frames.

3. The method according to claim 1, wherein: Selecting the one or more target frames from the preliminary frame and the new frame comprises: The one or more target frames are randomly selected from the preliminary frames and the new frames.

4. The method according to claim 1, wherein: Selecting the one or more target frames from the preliminary frame and the new frame comprises: determining a timestamp corresponding to each of the preliminary frames; removing old preliminary frames from the preliminary frames to generate one or more remaining preliminary frames, wherein the time stamps of the old frames are earlier than a preset time point; and The one or more target frames are selected from the one or more remaining preliminary frames and the new frame.

5. The method according to claim 1, wherein: Selecting the one or more target frames from the preliminary frame and the new frame comprises: clustering the preliminary frame and the new frame to generate one or more target clusters; and A target frame is selected from each of the one or more target clusters as the one or more target frames.

6. The method according to claim 5, further comprising: determining an isolated target cluster from the one or more target clusters, wherein a count of frames in the isolated target cluster is less than a threshold; removing the isolated target cluster from the one or more target clusters to generate remaining target clusters; and The target frame is selected from each of the remaining target clusters as the one or more target frames.

7. The method according to claim 1, wherein: Selecting the one or more target frames from the preliminary frame and the new frame comprises: determining a quality index for each of the preliminary frames and each of the new frames; and The one or more target frames are selected from the preliminary frames and the new frames based on the quality index.

8. The method according to claim 1, wherein: Selecting the one or more target frames from the preliminary frame and the new frame comprises: displaying a visual representation associated with the preliminary frame and the new frame on a graphical user interface (GUI) of a user device; receiving a selection operation on the GUI from a user; and The one or more target frames are selected based on the selection operation from the user.

9. The method according to claim 1, further comprising: Acquire a 3D data representation of the reference scene; as well as A 3D model of the reference scene is generated based on the 3D data representation of the reference scene and the updated localizer map.

10. The method according to claim 9, wherein: Generating a 3D model of the reference scene based on the 3D data representation of the reference scene and the updated localizer map further comprises: aligning the updated locator map with the 3D data representation; and A 3D model of the reference scene is generated based on the aligned updated locator map and the 3D data representation.

11. The method according to claim 10, wherein: Aligning the updated locator map with the 3D data representation comprises: aligning the preliminary locator map with the 3D data representation; and The new frame is transformed from a new coordinate system to an old coordinate system, wherein the old coordinate system corresponds to the preliminary frame.

12. The method according to claim 1, wherein: Determining a pose of the capture device associated with capturing the plurality of new data points includes: The gesture of the capture device is recorded using a recording device.

13. The method according to claim 1, wherein: Determining a pose of the capture device associated with capturing the plurality of new data points includes: The preliminary locator map is executed to determine a pose of the capture device based on the plurality of new data points.

14. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to: obtaining a preliminary localizer map of a reference scene, the preliminary localizer map comprising preliminary frames, each of the preliminary frames being associated with a set of preliminary data points; capturing a plurality of new data points on the reference scene using a capture device; determining a pose of the capture device associated with capture of the plurality of new data points; creating new frames, each of the new frames corresponding to one of the gestures and associated with a set of new data points of the plurality of new data points; selecting one or more target frames from the preliminary frames and the new frames; as well as An updated locator map is generated based on the one or more target frames and the associated set of preliminary or new data points.

15. The computing device of claim 14, wherein: The instructions further configure the apparatus to: determining that a count of the selected one or more target frames exceeds a maximum number of frames; as well as Target frames are cleared from the selected one or more target frames so that a count of the selected target frames after clearing is less than or equal to the maximum number of frames.

16. The computing device of claim 14, wherein: To select the one or more target frames from the preliminary frame and the new frame, the instructions configure the apparatus to: determining a timestamp corresponding to each of the preliminary frames; removing old preliminary frames from the preliminary frames to generate one or more remaining preliminary frames, wherein the time stamps of the old frames are earlier than a preset time point; and The one or more target frames are selected from the one or more remaining preliminary frames and the new frame.

17. The computing device of claim 16, wherein: The instructions further configure the apparatus to: acquiring a 3D data representation of the reference scene; and A 3D model of the reference scene is generated based on the 3D data representation of the reference scene and the updated localizer map.

18. The computing device of claim 17, wherein: To generate a 3D model of the reference scene based on the 3D data representation of the reference scene and the updated localizer map, the instructions further configure the apparatus to: aligning the updated locator map with the 3D data representation; as well as A 3D model of the reference scene is generated based on the aligned updated locator map and the 3D data representation.

19. The computing device of claim 18, wherein: To align the updated locator map with the 3D data representation, the instructions configure the apparatus to: aligning the preliminary locator map with the 3D data representation; and The new frame is transformed from a new coordinate system to an old coordinate system, wherein the old coordinate system corresponds to the preliminary frame.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: obtaining a preliminary localizer map of a reference scene, the preliminary localizer map comprising preliminary frames, each of the preliminary frames being associated with a set of preliminary data points; capturing a plurality of new data points on the reference scene using a capture device; determining a pose of the capture device associated with capture of the plurality of new data points; creating new frames, each of the new frames corresponding to one of the gestures and associated with a set of new data points of the plurality of new data points; selecting one or more target frames from the preliminary frames and the new frames; as well as An updated locator map is generated based on the one or more target frames and the associated set of preliminary or new data points.

Images