Key frame downsampling for reducing memory usage in SLAM
By optimizing keyframes in the SLAM system through descriptor downsampling and standard elimination, the problems of high computational cost and low positioning accuracy are solved, achieving a more efficient positioning and mapping process.
Patent Information
- Application Number
- CN202280094476.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-06
- Publication Date
- 2024-11-19
AI Technical Summary
Existing SLAM systems suffer from high computational costs during localization and mapping due to the excessive number of keyframes, and are unable to perform effective triangulation when dealing with significant rotational motion, thus affecting localization accuracy and efficiency.
By downsampling keyframes and selecting some descriptors to map landmarks, the number of stored descriptors is reduced. Furthermore, descriptor elimination criteria and image elimination criteria are applied to optimize the storage and mapping process of keyframes.
It effectively reduces the storage requirements of SLAM systems, improves positioning accuracy and efficiency, especially when dealing with cameras with multiple descriptors, ensures the overall accuracy of mapping data, and reduces unnecessary computational burden.
Smart Images

Figure CN118985007A_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to data processing technology, including but not limited to methods, systems, and non-transitory computer-readable media for selectively storing key frames for positioning cameras and mapping environments in extended reality applications. Background Art
[0002] Simultaneous localization and mapping (SLAM) is widely used in virtual reality (VR), augmented reality (AR), autonomous driving, and navigation. In SLAM, high-frequency pose estimation is achieved through sensor fusion. Asynchronous time warping (ATW) is often applied to AR systems along with SLAM to warp images before they are sent to the display to correct for head movements that occur after the image is rendered. In SLAM and ATW, relevant image data and inertial sensor data are synchronized and used to estimate and predict the camera pose. For localization, many SLAM systems suppress the accumulated errors of inertial sensor data by detecting corners and extracting image descriptors on keyframe images. Each keyframe image is associated with a descriptor derived using a bag of visual words data structure, and the corresponding descriptors of a new image are compared with those of existing keyframe images to localize the new image. The computational cost increases with the number of existing keyframe images, and becomes out of control when the number of existing keyframe images reaches a huge or global scale, such as 1 billion.
[0003] Traditional solutions use the length of optical flow between two consecutive images to reduce the number of keyframe images. When the average length of different optical flows is short, traditional solutions wait for new images to find more optical flow lengths and set the next keyframe when it is sufficient. In addition, when the camera has significant rotational motion relative to the translational motion, this significant rotational motion cannot be triangulated for 3D calculations, and the associated keyframes are invalid and removed. Smaller six degree-of-motion (6DOF) motions are used to identify close objects, but are not conducive to downsampling of keyframes. It would be beneficial to establish a more efficient SLAM mechanism using fewer keyframes than current practices. Summary of the invention
[0004] Various embodiments of the present application relate to SLAM techniques for mapping virtual spaces with descriptors. A camera is positioned and used to capture an image in a scene, and the scene is mapped to a virtual space including a plurality of landmarks. Each image (also referred to as a keyframe) corresponds to a camera pose (i.e., camera position and camera orientation) of the camera, and a set of landmarks in the scene is recorded. Each keyframe is processed to provide a set of descriptors for the set of landmarks of the scene, and the set of descriptors is associated with the camera pose. From the perspective of each landmark, each landmark is recorded in one or more keyframes, and the one or more keyframes provide one or more descriptors associated with one or more corresponding camera poses to describe the corresponding landmarks. As the number of keyframes captured by the camera increases, memory usage also increases. In some embodiments, the number of keyframes or the number of associated landmarks in each keyframe is downsampled to reduce the number of descriptors that SLAM needs to store. For example, when a camera pose has multiple descriptors, a portion of these descriptors are selected to map landmarks because other unselected descriptors are substantially similar to the descriptors of other keyframes. When the selected portion of the descriptor is small, the contribution of the camera pose to the overall accuracy of the mapping data of the virtual space is limited, and the camera pose and the corresponding keyframe are completely prohibited from providing any descriptor including the selected portion to map the set of landmarks. This keyframe downsampling mechanism can be applied to various SLAM-based products, such as AR glasses, robotic systems, autonomous driving, drones, or mobile devices implementing AR applications.
[0005] In one aspect, a method is implemented in an electronic system having a camera. The method includes obtaining image data, the image data including a plurality of images captured by a camera in a scene, and each of the plurality of images includes a first landmark. The method also includes generating a plurality of landmark descriptors for the first landmark from the image data, and identifying a plurality of camera poses for the plurality of landmark descriptors. Each landmark descriptor is generated from a different image including the first landmark and captured at a different camera pose. The method also includes selecting a first landmark descriptor corresponding to a first camera pose of two of the plurality of camera poses to map the first landmark in the scene based on determining that two of the plurality of camera poses satisfy a descriptor elimination criterion.
[0006] In some embodiments, the method further comprises deselecting a second different landmark descriptor corresponding to a second camera pose of the two camera poses in the plurality of camera poses to map the first landmark in the scene based on determining that the two camera poses in the plurality of camera poses satisfy the descriptor elimination criteria. In some embodiments, the method further comprises mapping the first landmark with a subset of the plurality of landmark descriptors. The subset of the plurality of landmark descriptors corresponds to a subset of different camera poses that captured the subset of the plurality of images. The subset of the plurality of landmark descriptors is selected based on determining that any two camera poses in the subset of different camera poses do not satisfy the descriptor elimination criteria.
[0007] In another aspect, some embodiments include an electronic system comprising one or more processors and a memory having instructions stored thereon, which, when executed by the one or more processors, cause the processors to perform any of the above methods.
[0008] In yet another aspect, some embodiments include a non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the processors to perform any of the above methods.
[0009] These exemplary embodiments and implementations are not mentioned to limit or define the present disclosure, but to provide examples to aid understanding of the present disclosure. Other embodiments are discussed in the detailed description, and further description is provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals represent corresponding parts throughout the drawings.
[0011] Figure 1 is an example data processing environment having one or more servers communicatively coupled to one or more client devices in accordance with some embodiments.
[0012] Figure 2 is a block diagram illustrating an electronic system according to some embodiments.
[0013] Figure 3 is a flow chart of a process for processing inertial sensor data and image data of an electronic system (eg, a server, a client device, or a combination of both) using a SLAM module according to some embodiments.
[0014] FIG. 4A to FIG. 4C are three simplified diagrams of a virtual space with multiple landmarks captured by different keyframes, according to some embodiments.
[0015] Figure 5Ais a diagram of a virtual space mapped with a plurality of landmarks associated with a first set of keyframes according to some embodiments, and Figure 5B is another diagram of a virtual space mapped with a plurality of landmarks associated with a second set of keyframes in accordance with some embodiments.
[0016] Figure 6 is a flow chart of a method of simultaneous localization and mapping (SLAM) according to some embodiments.
[0017] Like reference numerals designate corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION
[0018] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, many non-limiting specific details are listed to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic systems having digital video capabilities.
[0019] The present application relates to positioning a camera and mapping a scene for rendering extended reality content (e.g., virtual, augmented, or mixed reality content) on an electronic device. In the prior art, the current image is compared with an existing keyframe to identify the camera position, which may be very inefficient because a large number of keyframes need to be created to map the scene. In various embodiments of the present application, the number of keyframes or the number of associated landmarks in each keyframe are downsampled to reduce the number of descriptors that SLAM needs to store. For example, when a camera pose has multiple descriptors, a portion of these descriptors is selected to map the landmarks because the other unselected descriptors are substantially similar to the descriptors of other keyframes. When the selected portion of the descriptor is small, the contribution of the camera pose to the overall accuracy of the mapping data of the virtual space is limited, and the camera pose and the corresponding keyframe are completely prohibited from providing any descriptors including the selected portion to map the group of landmarks.
[0020] Figure 11 is an example data processing environment 100 having one or more servers 102 communicatively coupled to one or more client devices 104 according to some embodiments. The one or more client devices 104 may be, for example, a laptop 104A, a tablet 104B, a mobile phone 104C, or a smart multi-sensing connected home device (e.g., a surveillance camera 104E, a smart TV device, a drone). In some implementations, one or more client devices 104 include a head-mounted display 104D for rendering extended reality content. Each client device 104 may collect data or user input, execute a user application, and present output on a user interface of the client device. The collected data or user input may be processed locally on the client device 104 and / or remotely by the server 102. The one or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client devices 104, and in some embodiments, process data and user input received from the client devices 104 when the user application is executed on the client devices 104. In some embodiments, data processing environment 100 also includes memory 106 for storing data related to server 102, client device 104, and applications executing on client device 104. For example, memory 106 may store video content (including visual and audio content), static visual content, and / or inertial sensor data.
[0021] One or more servers 102 can communicate real-time data with client devices 104 that are remote from each other or remote from one or more servers 102. In addition, in some embodiments, one or more servers 102 can implement data processing tasks that the client device 104 cannot or preferably does not complete locally. For example, the client device 104 includes a game console (e.g., formed by a head-mounted display 104D) that executes an interactive online game application. The game console receives user instructions and sends the user instructions to the game server 102 together with the user data. The game server 102 generates a video data stream based on the user instructions and the user data, and provides the video data stream for display on the game console and other client devices participating in the same game session with the game console.
[0022] One or more servers 102, one or more client devices 104, and storage 106 are communicatively coupled to each other via one or more communication networks 108, which are media for providing communication links between these devices and connected computers within data processing environment 100. One or more communication networks 108 may include connections such as wired connections, wireless communication links, or fiber optic cables. Examples of one or more communication networks 108 include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. The one or more communication networks 108 are optionally implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, universal serial bus (USB), FireWire, long term evolution (LTE), global system for mobile communications (GSM), enhanced data GSM environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet protocol (VoIP), Wi-MAX, or any other suitable communication protocol. The connection to the one or more communication networks 108 can be established directly (e.g., using a 3G / 4G connection to a wireless carrier), or through a network interface 110 (e.g., a router, switch, gateway, hub, or smart dedicated whole-home control node), or through any combination thereof. Thus, one or more communication networks 108 may represent the Internet, a global collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols. At the heart of the Internet lies a backbone of high-speed data communication lines between major nodes or host computers, including thousands of commercial, government, educational, and other electronic systems that route data and messages.
[0023] The head mounted display 104D (also referred to as AR glasses 104D) includes one or more cameras (e.g., visible light cameras, depth cameras), microphones, speakers, one or more inertial sensors (e.g., gyroscopes, accelerometers), and displays. The camera and microphone are used to capture video and audio data from the scene in which the AR glasses 104D are located, while the one or more inertial sensors are used to capture inertial sensor data. In some cases, the camera captures the gestures of the user wearing the AR glasses 104D. In some cases, the microphone records ambient sounds, including the user's voice commands. In some cases, both the video or static visual data captured by the visible light camera and the inertial sensor data measured by one or more inertial sensors are used to determine and predict the device pose. The video, static image, audio, or inertial sensor data captured by the AR glasses 104D is processed by the AR glasses 104D, the server 102, or both to identify the device pose. The device pose is used to control the AR glasses 104D itself, or to interact with an application (e.g., a game application) executed by the AR glasses 104D. In some embodiments, the display of the AR glasses 104D displays a user interface, and the recognized or predicted device pose is used to interact with user-selectable display items on the user interface or to render virtual objects with high fidelity.
[0024] In some embodiments, SLAM technology is applied in the data processing environment 100 to process video data, static image data, or depth data captured by the AR glasses 104D together with the inertial sensor data. Identify and predict the device pose, and map and update the scene in which the AR glasses 104D are located. Optionally, the SLAM technology is implemented independently by the AR glasses 104D, or jointly by the server 102 and the AR glasses 104D.
[0025] Figure 2 1 is a block diagram illustrating an electronic system 200 for processing content data (eg, image data) according to some embodiments. The electronic system 200 includes a server 102, a client device 104 (eg, Figure 1The electronic system 200 typically includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as chipsets). The electronic system 200 includes one or more input devices 210 for facilitating user input, such as a keyboard, a mouse, a voice command input unit or a microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. In addition, in some embodiments, the client device 104 of the electronic system 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard. In some embodiments, the client device 104 includes one or more optical cameras 260 (e.g., RGB cameras), scanners, or light sensor units for capturing images, such as images of graphic serial codes printed on electronic devices. The electronic system 200 also includes one or more output devices 212 capable of presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays.
[0026] Optionally, the client device 104 includes a location detection device, such as a GPS (Global Positioning System) or other geolocation receiver, for determining the location of the client device 104. Optionally, the client device 104 includes an inertial measurement unit (IMU) 280 that integrates sensor data captured by multi-axis inertial sensors to provide an estimate of the position and orientation of the client device 104 in space. The one or more inertial sensors of the IMU 280 include, but are not limited to, a gyroscope, an accelerometer, a magnetometer, and an inclinometer.
[0027] Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Memory 206 optionally includes one or more storage devices located away from one or more processing units 202. Memory 206 or optionally non-volatile memory within memory 206 includes non-transitory computer-readable storage media. In some embodiments, memory 206 or non-transitory computer-readable storage media of memory 206 stores the following programs, modules, and data structures or a subset or superset of the above: Operating system 214, including processes for handling various basic system services and for performing hardware-related tasks; A network communication module 216 for connecting each server 102 or client device 104 to other devices (e.g., server 102, client device 104, or storage 106) via one or more network interfaces 204 (wired or wireless) and one or more communication networks 108 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.); A user interface module 218 for enabling presentation of information (e.g., a graphical user interface of an application 224, a widget, a website and web pages therein, and / or games, audio and / or video content, text, etc.) on each client device 104 via one or more output devices 212 (e.g., a display, a speaker, etc.); An input processing module 220 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected inputs or interactions; A web browser module 222 for navigating, requesting (e.g., via HTTP), and displaying websites and web pages of the websites, including a web interface for logging into a user account associated with the client device 104 or another electronic device, controlling the client or electronic device if the client or electronic device is associated with a user account, and editing and viewing settings and data associated with the user account; One or more user applications 224 executed by the electronic system 200 (e.g., a game, a social networking application, a smart home application, and / or other web-based or non-web applications for controlling another electronic device and viewing data captured by such a device); A model training module 226 for receiving training data and building a data processing model for processing content data (e.g., video, image, audio, or text data) to be collected or obtained by the client device 104; a data processing module 228 for processing the content data using the data processing model 250 to identify information contained in the content data, match the content data with other data, categorize the content data, or synthesize related content data, wherein in some embodiments, the data processing module 228 is associated with one of the user applications 224 to process the content data in response to user instructions received from the user application 224; a pose determination and prediction module 230 for determining and predicting a pose of a client device 104 (e.g., AR glasses 104D), wherein, in some embodiments, the pose is jointly determined and predicted by the pose determination and prediction module 230 and the data processing module 228, and in some embodiments, the module 230 further includes a SLAM module 232 for mapping a scene in which the client device 104 is located and identifying the pose of the client device 104 within the scene using images or IMU sensor data; a pose-based rendering module 238 for rendering virtual objects in the field of view of the camera 260 of the client device 104, or creating mixed, virtual, or augmented reality content using images captured by camera 260, wherein virtual objects are rendered and the mixed, virtual, or augmented reality content created is from the perspective of camera 260 based on the camera pose of camera 260; and One or more databases 240, used to store data including at least one or more of the following: o Device settings 242, including general device settings of one or more servers 102 or client devices 104 (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.); o User account information 244 for one or more user applications 224, such as user name, security questions, account history data, user preferences, and preset account settings; o Network parameters 246 of one or more communication networks 108, such as IP addresses, subnet masks, default gateways, DNS servers, and host names; o training data 248 for training one or more data processing models 250; o Data processing model 250 for processing content data (e.g., video, image, audio, or text data) using deep learning techniques; o a pose data database 252 for storing pose data of the camera 260, wherein, in some embodiments, the descriptors and associated camera poses are compressed according to a descriptor or image elimination criterion and stored in association with landmarks in the scene; and o Content data and results 254, the content data is obtained by the client device 104 of the electronic system 200, and the content results are output to the client device 104 of the electronic system 200, wherein the content data is processed locally on the client device 104 by the data processing model 250 or remotely on the server 102 to provide associated results to be presented on the client device 104, and the content data includes candidate images.
[0028] Optionally, one or more databases 240 are stored in one of the server 102, the client device 104, and the memory 106 of the electronic system 200. Optionally, one or more databases 240 are distributed in multiple of the server 102, the client device 104, and the memory 106 of the electronic system 200. In some embodiments, multiple copies of the above data are stored in different devices, for example, two copies of the data processing model 250 are stored in the server 102 and the memory 106 respectively.
[0029] Each element in the element of the above-mentioned identification can be stored in one or more of the aforementioned memory devices, and corresponds to the instruction set for performing the above-mentioned functions. The module or program (that is, instruction set) identified above does not need to be implemented as a separate software program, process, module or data structure, and therefore each subset of these modules can be combined or otherwise rearranged in various embodiments. In certain embodiments, memory 206 optionally stores the module identified above and the subset of data structure. In addition, memory 206 optionally stores additional modules and data structure not described above.
[0030] Figure 3 2 is a flow chart of a process 300 for processing inertial sensor data and image data of an electronic system (e.g., a server 102, a client device 104, or a combination of both) using a visual-inertial SLAM module 232 according to some embodiments. The process 300 includes measurement preprocessing 302, initialization 304, local visual-inertial odometry (VIO) 306 with relocalization, and global pose graph optimization 308. In measurement preprocessing 302, the RGB camera 260 captures image data of a scene at an image rate (e.g., 30 FPS), and features are detected and tracked (310) from the image data. The IMU 280 measures inertial sensor data at a sampling frequency (e.g., 1000 Hz) while the RGB camera 260 captures image data, and the inertial sensor data is pre-integrated (312) to provide data on changes in device pose 340. In initialization 304, image data captured by the RGB camera 260 and inertial sensor data measured by the IMU 280 are temporally aligned (314). A vision-only structure from motion (SfM) technique 314 is applied (316) to couple the image data and inertial sensor data, estimate the three-dimensional structure, and map the scene of the RGB camera 260.
[0031] After initialization 304 and during relocalization 306, the VIO is optimized (322) using a sliding window 318 and associated states 320 from loop closures. When the VIO corresponds to (324) a keyframe for a smooth video transition and a corresponding loop is detected (326), features are retrieved (328) and used for the associated states generated from the loop closure 320. In a global pose graph optimization 308, a multi-degree-of-freedom (multi-DOF) pose graph is optimized (330) based on the states from the loop closure 320, and a keyframe database 332 is updated with the keyframes associated with the VIO. Specifically, in some embodiments, each keyframe includes a set of landmarks and is processed to generate landmark descriptors to map the landmarks. The descriptors are optionally compressed according to a descriptor elimination criterion, and the corresponding keyframe mapping landmarks are optionally eliminated according to an image elimination criterion.
[0032] In addition, the detected and tracked (310) features are used to monitor (334) the motion of the object in the image data and to estimate an image-based pose 336 (e.g., based on the image rate). In some embodiments, the pre-integrated (234) inertial sensor data can be propagated (338) based on the motion of the object and used to estimate an inertial-based pose 340 (e.g., based on the sampling frequency of the IMU 280). The image-based pose 336 and the inertial-based pose 340 are stored in the database 240 and used by the module 230 to estimate and predict poses used by the real-time video rendering system 234. Optionally, in some embodiments, the module 232 receives the inertial sensor data measured by the IMU 280 and obtains the image-based pose 336 to estimate and predict more poses 340 that are further used by the temporal video rendering system 234.
[0033] In SLAM, high-frequency pose estimation is achieved through sensor fusion, which relies on data synchronization between imaging sensors and IMU 280. Imaging sensors (e.g., RGB camera 260, LiDAR scanner) provide image data required for pose estimation and typically operate at a lower frequency (e.g., 30 frames per second) and longer latency (e.g., 30 milliseconds) than IMU 280. In contrast, IMU 280 can measure inertial sensor data and operate at a very high frequency (e.g., 1000 samples per second) and negligible latency (e.g., <0.1 milliseconds). Asynchronous time warping (ATW) is often applied in AR systems to warp images before they are sent to the display, thereby correcting head movements and pose changes that occur after the image is rendered. The ATW algorithm reduces image latency, increases or maintains frame rate, or reduces jitter caused by missing images. In SLAM and ATW, relevant image data and inertial sensor data are stored locally so that these data can be synchronized and used for pose estimation / prediction. In some embodiments, the image and inertial sensor data are stored in one of a plurality of standard tessellation language (STL) containers such as std::vector, std::queue, std::list, etc., or other custom containers. These containers are generally convenient to use. The image and inertial sensor data are stored in the STL container along with the timestamps of the data, which are used for data searching, data insertion, and data organization.
[0034] FIG. 4A to FIG. 4C are three simplified diagrams of a virtual space 400 having multiple landmarks 402 captured by different keyframes 404 according to some embodiments. The electronic device has a camera 260 and is located in a scene. The camera 260 captures multiple images (also referred to as keyframes) 404, and each keyframe 404 is captured when the camera 260 has a camera pose 406 (i.e., is located at a camera position and oriented toward a camera orientation). The scene includes multiple objects associated with multiple landmarks 402. Each keyframe 404 records a portion of the scene that includes a subset of objects, and therefore includes a subset of landmarks 402. For each landmark in the subset of landmarks 402, the corresponding keyframe 404 is processed by a data processing model (e.g., a convolutional neural network (CNN)) to extract a landmark descriptor for the corresponding landmark 402. For example, reference Figure 4A, the first keyframe 404A includes three landmarks 402A to 402C and is processed to extract three corresponding landmark descriptors of the three landmarks 402A to 402C. Similarly, each of the second keyframe 404B and the third keyframe 404C includes three landmarks 402A to 402C and is processed to extract three corresponding landmark descriptors of the three landmarks 402A to 402C. In this way, the keyframes 404A to 404C provide 9 landmark descriptors associated with the three landmarks 402A to 402C.
[0035] The first landmark 402A is associated with at least three landmark descriptors determined from the first keyframe 404A, the second keyframe 404B, and the third keyframe 404C. The first landmark 402A is present in each of the keyframes 404A to 404C, each of which is captured at a different camera pose 406A to 406C. A descriptor elimination criterion is applied to determine whether to select each of the landmark descriptors determined from the keyframes 404A to 404C to map the first landmark 402A. For example, referring to Figure 4B , based on a descriptor elimination criterion, a first landmark descriptor determined by a first keyframe 404A is selected over a second landmark descriptor determined by a second keyframe 404B to map a first landmark 402A in a scene. A first camera pose 406A associated with the first keyframe 404A includes a first camera position and a first camera pose, and a second camera pose 406B associated with the second keyframe 404B includes a second camera position and a second camera pose. For the first landmark 402A, a first image ray 408A connects the first camera position to the first landmark 402A, and a second image ray 408B connects the second camera position to the first landmark 402A. The first image ray 408A and the second image ray 408B form a ray angle 410 connecting the first camera position and the second camera position to the first landmark 402A. The descriptor elimination criterion defines a ray angle threshold (e.g., 5-10 degrees). If the ray angle 410 is less than the ray angle threshold, the first image ray 408A and the second image ray 408B and the first camera position and the second camera position meet the descriptor elimination criteria. That is, according to the descriptor elimination criteria, one of the landmark descriptors associated with the first keyframe 404A and the second keyframe 404B needs to be eliminated to map the first landmark 402A in the scene.
[0036] Assume that the first key frame 404A is captured earlier in time than the second key frame 404B. Figure 4BIn some of the illustrated embodiments, when one of the landmark descriptors associated with the first key frame 404A and the second key frame 404B mapping the first landmark 402A needs to be eliminated based on the descriptor elimination criteria, the first landmark descriptor associated with the first key frame 404A captured earlier in time is selected, and the second landmark descriptor associated with the second key frame 404B captured later in time is prohibited from mapping the first landmark 402A. In other words, based on the descriptor elimination criteria, the second landmark descriptor associated with the second key frame 404B captured later in time is eliminated with a higher priority than the first landmark descriptor. Conversely, in Figure 4B In some embodiments not shown in FIG. 1 , based on the descriptor elimination criteria, the first landmark descriptor associated with the first key frame 404A captured earlier in time is eliminated with priority over the second landmark descriptor. The second landmark descriptor associated with the second key frame 404B captured later in time is selected, and the first landmark descriptor associated with the first key frame 404A captured earlier in time is prohibited from landmark mapping.
[0037] Similarly, for the second landmark 402B, the third image ray 408C connects the second camera position to the second landmark 402B, and the fourth image ray 408D connects the first camera position to the second landmark 402B. The third image ray 408C and the fourth image ray 408D form a ray angle 412 that connects the second camera position and the first camera position to the second landmark 402B. Since the ray angle 410 is less than the ray angle threshold, the third image ray 408C and the fourth image ray 408D and the first camera position and the second camera position meet the descriptor elimination criteria. It is necessary to eliminate one of the landmark descriptors associated with the first keyframe 404A and the second keyframe 404B mapping the second landmark 402B in the scene. In Figure 4B In some embodiments shown, a landmark descriptor associated with a first keyframe 404A captured earlier in time is selected to map a second landmark 402B in the scene, and another landmark descriptor associated with a second keyframe 404B captured later in time is disabled for landmark mapping. Figure 4B In some embodiments not shown in FIG. 4 , a landmark descriptor associated with a second keyframe 404B captured later in time is selected to map a landmark 402B in the scene, and a first landmark descriptor associated with a first keyframe 404A captured earlier in time is disabled for landmark mapping.
[0038] In some embodiments, after applying the descriptor elimination criteria, the first landmark 402A, the second landmark 402B, and the third landmark 402C are mapped with a set of landmark descriptors determined from a set of keyframes 404 including the first keyframe 404A, the second keyframe 404B, and the third keyframe 404C. The set of landmark descriptors includes three landmark descriptors provided by the first keyframe 404A, three landmark descriptors provided by the third keyframe 404C, but only one landmark descriptor provided by the second keyframe 404B. Any two landmark descriptors of each landmark 402 meet the descriptor elimination criteria. In other words, the set of landmark descriptors is selected based on determining that any two camera poses in a subset of different camera poses do not meet the descriptor elimination criteria. In the example, the ray angle formed by any two image rays connected to the same landmark 402 is greater than the ray angle threshold defined by the descriptor elimination criteria.
[0039] In some embodiments, the descriptor elimination criteria eliminates one of the two landmark descriptors determined from the two keyframes 404 for the same landmark 402 based on the distance between the landmark 402 and each of the two camera positions corresponding to the two keyframes 404. For example, for the first landmark 402A, if the ray angle 410 is less than the ray angle threshold, the second landmark descriptor determined from the second keyframe 404B is eliminated based on determining that the first distance between the first landmark 402A and the first camera position of the first keyframe 404A is greater than the second distance between the first landmark 402A and the second camera position of the second keyframe 404B. Conversely, in some embodiments, based on determining that the ray angle 410 is less than the ray angle threshold and the second distance is less than the first distance, the first landmark descriptor determined from the first keyframe 404A is eliminated and the second landmark descriptor determined from the second keyframe 404B is selected.
[0040] refer to Figure 4CIn some embodiments, a target keyframe 404 (e.g., a second keyframe 404B) satisfies an image elimination criterion and is completely eliminated from mapping a plurality of landmarks 402 in a scene. The target keyframe 404 is captured to include a first number of landmarks 402 (e.g., 100 landmarks) corresponding to a first number of landmark descriptors. After applying the descriptor elimination criterion, a second number of landmark descriptors corresponding to a subset of landmarks are prohibited from mapping the subset of landmarks. Based on determining that the first number or the second number satisfies the image elimination criterion, the target keyframe 404 is completely eliminated, and the first number of landmark descriptors associated with the target keyframe 404 are eliminated from mapping a plurality of landmarks 402 in the scene. In addition, in some embodiments, the image elimination criterion requires that the ratio of the second number to the first number exceeds a predetermined threshold (e.g., 90%). This means that if a majority of the landmark descriptors provided by the target keyframe 404 are eliminated, the target keyframe 404 cannot be effectively used to map the scene and needs to be eliminated, leaving space to store information of the keyframe 404 that is more effectively utilized.
[0041] Additionally, when eliminating the target keyframe 404 , if a landmark descriptor associated with the target keyframe has been selected to map one of the multiple landmarks based on the descriptor elimination criteria, selecting the selected landmark descriptor to map one of the multiple landmarks is discontinued.
[0042] In the example, the target keyframe 404 is a second keyframe 404B that is processed to provide three landmark descriptors to three landmarks 402A to 402C. Based on the descriptor elimination criteria, two of the three landmark descriptors determined from the second keyframe 404B are eliminated, and only one of the three landmark descriptors is associated with the third landmark 402C. Assuming that two of the three landmark descriptors provided by the keyframe 404B are eliminated, the second keyframe 404B is completely eliminated from the mapping scene. Although one of the three landmark descriptors is not eliminated by the descriptor elimination criteria, the landmark descriptor of the three landmark descriptors is still eliminated from the mapping third landmark 402C together with the second keyframe 404B.
[0043] Figure 5A is a diagram of a virtual space 500 mapped with a plurality of landmarks 402 associated with a first set of keyframes 520 according to some embodiments, and Figure 5B540 is another diagram of a virtual space 500 mapped with a plurality of landmarks 402 associated with a second set of key frames 540 according to some embodiments. The second set of key frames 540 is simplified from the first set of key frames 520, and the first set of key frames 520 includes the second set of key frames 540. In addition to the second set of key frames 540, the first set of key frames 520 also has one or more additional key frames. One or more additional key frames are eliminated, not selected, or deselected based on a combination of descriptor elimination criteria and image elimination criteria. In these ways, the landmark descriptors determined from the second set of key frames 540 can fully map the scene and identify the current camera pose of the current frame 504C, while requiring less storage space to store information of the second set of key frames 540 including the landmark descriptors determined from the second set of key frames 540.
[0044] The camera 260 is placed in a scene and captures a plurality of images. The plurality of images are applied as keyframes 404 to map the scene to a virtual space 500. The scene includes a plurality of objects 502 associated with a plurality of landmarks 402. For example, each landmark 402 is associated with a corner or edge of a corresponding object 502. Each keyframe 404 records a portion of a scene including a subset of objects 502, and therefore includes a subset of landmarks 402. Each keyframe 404 is captured when the camera 260 has a camera pose 406 (i.e., located at a camera position and facing a camera orientation). The subset of landmarks 402 in each keyframe 404 is determined based on the camera pose 406 corresponding to the corresponding keyframe 404. For each landmark in the subset of landmarks 402, the corresponding keyframe 404 is processed to extract a landmark descriptor of the corresponding landmark 402. From the perspective of each landmark 402 , the respective landmark 402 is recorded in one or more keyframes 404 , each keyframe 404 providing a landmark descriptor associated with a respective camera pose 406 to map the respective landmark in the virtual space 500 .
[0045] The first group of key frames 520 provides a plurality of landmark descriptors. Each key frame in the first group of key frames 404-1 provides one or more corresponding landmark descriptors to map one or more corresponding landmarks 402. A subset of the plurality of landmark descriptors is eliminated by applying a descriptor elimination criterion. In some embodiments, the first subset of key frames 404-1 is completely eliminated because all one or more corresponding landmark descriptors provided by each key frame 404-1 are prohibited from landmark mapping based on the descriptor elimination criterion. In some embodiments, the second subset of key frames 404-2 is not eliminated by the descriptor elimination criterion. Each key frame 404-2 provides a plurality of corresponding landmark descriptors, and at least one landmark descriptor in the corresponding landmark descriptors provided by each key frame 404-2 is prohibited from landmark mapping based on the descriptor elimination criterion. In some embodiments, the third subset of key frames 404-3 is not affected by the descriptor elimination criterion, and all one or more corresponding landmark descriptors provided by each key frame 404-3 are selected to map the corresponding landmark 402.
[0046] In some embodiments, the image elimination criteria are further applied to determine whether to eliminate each key frame 404 in the second subset of key frames 404-2. The image elimination criteria do not affect a third subset of key frames 404-3 that do not correspond to any landmark descriptors eliminated by the descriptor elimination criteria. Each key frame in the second subset of key frames 404-2 provides a corresponding first number of landmark descriptors to a corresponding first number of landmarks 402, and due to the descriptor elimination criteria, a corresponding second number of landmark descriptors are not selected. In some embodiments, the second subset of key frames 404-2 includes one or more key frames 404-2A. For each key frame 404-2A, the corresponding first number or the second number satisfies the image elimination criteria (e.g., the criteria requires that a ratio of the corresponding second number to the first number exceeds a predetermined threshold), and the corresponding key frame 404-2A is eliminated and not selected. Figure 5B In some embodiments, the second subset of key frames 404-2 includes one or more key frames 404-2B. For each key frame 404-2B, the corresponding first number or the second number does not meet the image elimination criteria, and the corresponding key frame 404-2B is not eliminated, so that Figure 5B The second set of key frames 540 is shown in FIG.
[0047] In some embodiments, the second set of key frames 540 includes at least one key frame 404-2B and at least one key frame 404-3. Optionally, in some embodiments not shown, the second set of key frames 540 does not include any key frame 404-2B, and after applying the image elimination criteria, any key frame 404-2 with an unselected landmark descriptor is eliminated. Optionally, in some embodiments not shown, the second set of key frames 540 does not include any key frame 404-3, and all key frames 404 are affected by the descriptor elimination criteria. Based on the descriptor or image elimination criteria, the first set of key frames 520 is simplified to the second set of key frames 540. For each landmark 402 in the scene, one or more landmark descriptors are selected based on determining that any two of the one or more associated camera poses do not meet the descriptor elimination criteria. The mapping data of the scene is generated based on the information of the second set of key frames 540, and the second set of key frames 540 includes one or more landmark descriptors corresponding to each landmark 402 in the scene and one or more associated camera poses 406. Each landmark descriptor corresponds to a corresponding keyframe in the second set of keyframes 540 and a corresponding camera pose 406. The first set of keyframes 520 is compressed into the second set of keyframes 540, and the second set of keyframes 540 includes fewer keyframes than the first set of keyframes 520, thereby saving storage space required to store keyframe related information.
[0048] refer to Figure 5B , after creating the mapping data of the scene, the current frame 504C is used for SLAM, that is, to identify the current camera pose of the camera 260 and update the mapping of the scene in a synchronized manner. After the electronic device (e.g., AR glasses 104D) obtains the current frame 504C, the electronic device (e.g., based on CNN) extracts a plurality of feature points 506 from the current frame. Each feature point in the plurality of feature points 506 corresponds to an image descriptor determined from the current frame 504C. For each feature point in the plurality of feature points 506, the image descriptor is compared with the mapping data to identify the matching landmark 402 based on the second set of key frames 540. Based on the corresponding camera pose corresponding to the matching landmark 402, the current camera pose that captured the current frame 504C is determined.
[0049] In some embodiments, the current camera pose is interpolated from two or more camera poses corresponding to two or more key frames 404 in the second set of key frames 540. The plurality of feature points 506 of the current frame 504C includes a first feature point 506A. The electronic device determines that the first image descriptor of the first feature point 506A is a combination of two landmark descriptors of the matching landmark 402. The two landmark descriptors of the matching landmark 402 are determined from two different key frames 404. Based on the two camera poses corresponding to the two different key frames 404 that determine the two landmark descriptors, the current camera pose of the current frame 504C is determined. For example, the first image descriptor of the first feature point 506A in the current frame 504C is a combination of the two landmark descriptors of the first landmark 402A. The two landmark descriptors of the first landmark 402A are determined by the two key frames 508 and 510. Based on the two camera poses corresponding to the two key frames 508 and 510, the current camera pose of the current frame 504C is determined. For example, the current camera pose of the current frame 504C is equal to the weighted average of the two camera poses of the two key frames 508 and 510 .
[0050] Additionally, in some embodiments, a second feature point 506B is extracted from the current frame 504C and corresponds to a second image descriptor determined from the current frame 504C. Based on determining that the second image descriptor does not match the landmark descriptor in the mapping data, the current frame 504C is not included in the combination of the second set of key frames 540 and cannot be determined from the combination of the second set of key frames 540. The mapping data is updated by associating the second landmark 402B with the second image descriptor of the second feature point 506B and the current camera pose associated with the current frame 504C. Conversely, based on determining that the second image descriptor matches a subset of the landmark descriptors in the mapping data, the mapping data is not updated using information associated with the second image descriptor.
[0051] Figure 6 6 is a flowchart of a method 600 of simultaneous localization and mapping (SLAM) according to some embodiments. In some embodiments, the method is applied to AR glasses 104D, a robotic system, or an autonomous vehicle. For convenience, method 600 is described as being implemented by an electronic system 200 (e.g., a client device 104). In an example, method 600 is applied to determine and predict poses, map scenes, and simultaneously render virtual and real content in an extended reality (e.g., VR, AR). Method 600 is optionally governed by instructions stored in a non-temporary computer-readable storage medium and executed by one or more processors of an electronic system. Figure 6 Each operation shown in may correspond to a program stored in a computer memory or a non-transitory computer-readable storage medium (e.g., Figure 2The computer readable storage medium may include a magnetic or optical disk storage device, a solid-state storage device such as flash memory, or other non-volatile memory device. The instructions stored on the computer readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 600 may be combined and / or the order of some operations may be changed.
[0052] The electronic system has a camera 260 and obtains (602) image data including a plurality of images (also referred to as keyframes) 404 captured by the camera 260 in a scene. Each image in the plurality of images 404 includes a first landmark 402A. The electronic system generates (604) a plurality of landmark descriptors for the first landmark 402A from the image data and identifies (606) a plurality of camera poses 406 for the plurality of landmark descriptors. Each landmark descriptor is generated (608) from a different image 404 including the first landmark 402A and captured at a different camera pose 406. Based on determining that two camera poses in the plurality of camera poses 406 meet a descriptor elimination criterion (e.g., are substantially close to each other), the electronic system selects (610) a first landmark descriptor corresponding to a first camera pose of the two camera poses in the plurality of camera poses 406 to map the first landmark 402A in the scene.
[0053] In some embodiments, based on determining that two of the plurality of camera poses 406 satisfy the descriptor elimination criteria, the electronic system does not select (i.e., inhibits) (612) a second different landmark descriptor corresponding to a second of the two of the plurality of camera poses 406 to map the first landmark 402A in the scene. Additionally, in some embodiments, the first landmark descriptor is generated (614) from a first image 404A, the first image 404A is captured earlier than a second image 404B from which the second different landmark descriptor is generated, and the second different landmark descriptor is eliminated with a higher priority than the first landmark descriptor based on the descriptor elimination criteria.
[0054] In some embodiments, the electronic system maps the first landmark 402A with a subset of the plurality of landmark descriptors. The subset of the plurality of landmark descriptors corresponds to a subset of different camera poses 406 that captured the subset of the plurality of images 404. The subset of the plurality of landmark descriptors is selected based on a determination that no two camera poses in the subset of different camera poses 406 satisfy a descriptor elimination criterion.
[0055] In some embodiments, the plurality of images includes (616) a target image having a first number of landmarks corresponding to a first number of landmark descriptors. For the target image, based on a descriptor elimination criterion, the electronic system determines (618) that a second number of landmark descriptors corresponding to a subset of landmarks are prohibited from mapping the subset of landmarks. Based on determining that the first number or the second number satisfies the image elimination criterion, the electronic system eliminates (i.e., suspends selection) (620) a first number of landmark descriptors associated with the target image mapping a plurality of landmarks in the scene. Further, in some embodiments, the image elimination criterion requires (622) that a ratio of the second number to the first number exceeds a predetermined threshold. In some embodiments, the target keyframe corresponds to a first camera pose of two camera poses in a plurality of camera poses, and the subset of landmarks is different from the first landmark. By suspending selection of the first landmark descriptor for mapping the first landmark 402A, the first number of landmark descriptors is deselected from mapping the plurality of landmarks. In an example, if 80% or more of the descriptors of the target image are eliminated and prohibited from mapping landmarks in the scene, all descriptors associated with the target image are eliminated. The target image, the corresponding camera pose, and the landmark descriptors are not stored for SLAM.
[0056] In some embodiments, each different camera pose includes a corresponding camera position and a corresponding camera pose. For each landmark descriptor of the first landmark 402A, the electronic system identifies a corresponding image ray 408 connecting the corresponding camera position to the first landmark 402A, identifies a ray angle (e.g., Figure 4B 410 or 412), and determine that the ray angle is less than the ray angle threshold. Based on determining that the ray angle is less than the ray angle threshold, determine that two camera poses in the plurality of camera poses 406 meet the descriptor elimination criteria. In addition, in some embodiments, the ray angle threshold is in the range of [5°-10°].
[0057] In some embodiments, mapping data for the scene is generated (626). The mapping data includes one or more landmark descriptors corresponding to each landmark in the scene and one or more associated camera poses 406. Each landmark descriptor corresponds to a corresponding image and a corresponding camera pose. In addition, in some embodiments, for each landmark in the scene, one or more landmark descriptors are selected based on determining that any two of the one or more associated camera poses 406 do not meet the descriptor elimination criteria.
[0058] refer to Figure 5BIn some embodiments, the electronic system obtains a current frame 504C and extracts a plurality of feature points 506 from the current frame. Each feature point in the plurality of feature points 506 corresponds to an image descriptor determined from the current frame 504C. For each feature point in the plurality of feature points 506, the electronic system compares the image descriptor with the mapping data to identify a matching landmark 402 from the plurality of landmarks 402, and determines a current camera pose for capturing the current frame based on a corresponding camera pose corresponding to the matching landmark.
[0059] In addition, in some embodiments, the plurality of feature points 506 include a first feature point 506A. The electronic device determines that the first image descriptor of the first feature point 506A is a combination of two landmark descriptors matching the landmark 402, and based on the two images (e.g., Figure 5B The two camera poses 406 corresponding to 508 and 510 in the figure are used to determine the current camera pose of the current frame.
[0060] Additionally, in some embodiments, the electronic system extracts a second feature point 506B from the current frame 504C. The second feature point 506B corresponds to a second image descriptor determined from the current frame 504C, and based on determining that the second image descriptor does not match the landmark descriptor in the mapping data, updates the mapping data by associating the second landmark 402B with the second image descriptor and the current camera pose.
[0061] It should be understood that the particular order of operations in FIG. 8 that has been described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways of using descriptors for SLAM and image rendering as described herein. In addition, it should be noted that the above description of FIG. 4 and FIG. 5A to FIG. 5B The details of the other processes described apply in a similar manner to those described above with respect to Figure 6 Method 600. For the sake of brevity, these details are not repeated here.
[0062] The present application relates to a landmark descriptor downsampling method. The camera pose 406 that captures the key frame 404 is connected to the landmark 402 identified in the key frame 404 using an image ray 408. Between the image rays that connect the corresponding landmark 402 to the multiple camera poses 406, a ray angle is formed at each landmark 402. For each small ray angle (e.g., ray angles 410 and 412), one of the two image rays that form the corresponding small ray angle is disconnected, and one of the two camera poses 406 that are connected to form the two image rays 408 is prohibited from providing a landmark descriptor to map the corresponding landmark 402. The other camera pose of the two camera poses 406 that are selected to be connected to form the two image rays provides a landmark descriptor to map the corresponding landmark 402 in the scene. In some embodiments, the scene is further divided into a plurality of map tiles, and the landmark descriptors are further archived according to the plurality of map tiles. For more details on organizing mapping data using a map tile data structure, refer to the international application PCT / CN2021 / 076578, filed on February 10, 2021, entitled "Positioning Method, Electronic Device, and Storage Medium," which is incorporated herein by reference. Therefore, the present application relates to downsampling keyframe related information (e.g., camera pose, descriptors) archived in a map tile mapping data structure.
[0063] In some embodiments, after discarding image rays and descriptors based on descriptor elimination criteria, a subset of camera poses (e.g., Figure 4B The second camera pose 406B in the image ray and descriptor has a small number of indices that point to specific landmarks 402 to use the information of the image rays and descriptors. These camera poses are considered unimportant in improving the 3D depth accuracy of the landmarks 402 and can be replaced by other keyframes. The image elimination criteria are further applied to reduce the subset of camera poses that discard too many image rays due to the descriptor elimination criteria. In these ways, applying the descriptor and image elimination criteria helps to reduce the number of keyframes or the number of landmark descriptors stored to accurately map the scene.
[0064] Alternatively, the number of keyframes or the number of landmark descriptors stored for mapping the scene is downsampled based on the amount of camera motion. The amount of camera motion corresponds to a camera motion threshold. The descriptor elimination criterion requires that if the distance between the two camera poses 406 is less than the camera motion threshold, one of the two camera poses 406 is eliminated. In some embodiments, the camera motion threshold varies with the distance between the corresponding landmark 402 and the two camera poses 406, for example, the shorter the distance of the landmark 402 from the midpoint of the two camera poses 406, the smaller the camera motion threshold.
[0065] The terms used in the description of various embodiments described herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various embodiments described and the appended claims, the singular forms "one", "one", and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should also be understood that when used in this specification, the terms "include" and / or "comprising" indicate the existence of stated features, integers, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. In addition, it should be understood that although the terms "first", "second", etc. can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.
[0066] As used herein, the term “if” is optionally interpreted to mean “when” or “at” or “in response to determining” or “in response to detecting” or “according to a determination of…”, depending on the context. Similarly, the phrase “if it is determined” or “if [stated condition or event] is detected” is optionally interpreted to mean “upon the determination” or “in response to determining” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event]” or “according to a determination of detecting [stated condition or event]”, depending on the context.
[0067] For the purpose of illustration, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the exact forms disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the operating principles and practical applications, thereby enabling those skilled in the art to practice.
[0068] Although the various figures show multiple logical stages in a particular order, stages that are not dependent on the order may be reordered, and other stages may be combined or removed. Although some reordering or other groupings are specifically mentioned, other orderings and groupings will be obvious to those of ordinary skill in the art, and thus the orderings and groupings presented herein are not an exhaustive list of alternatives. In addition, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.
Claims
1. A method implemented in an electronic system having a camera, comprising: obtaining image data, the image data comprising a plurality of images captured by the camera of a scene, each image of the plurality of images comprising a first landmark; generating a plurality of landmark descriptors for the first landmark from the image data; identifying a plurality of camera poses for the plurality of landmark descriptors, wherein each landmark descriptor is generated from a different image including the first landmark and captured at a different camera pose; as well as Based on determining that two camera poses of the plurality of camera poses satisfy a descriptor elimination criterion, a first landmark descriptor corresponding to a first camera pose of the two camera poses of the plurality of camera poses is selected to map the first landmark in the scene.
2. The method according to claim 1, further comprising: Based on determining that the two camera poses in the plurality of camera poses satisfy the descriptor elimination criterion, a second different landmark descriptor corresponding to a second camera pose in the two camera poses in the plurality of camera poses is prohibited from mapping the first landmark in the scene.
3. The method according to claim 2, wherein: The first landmark descriptor is generated from a first image that was captured earlier than a second image from which the second, different landmark descriptor was generated, and the second, different landmark descriptor is eliminated with a higher priority than the first landmark descriptor based on the descriptor elimination criteria.
4. The method according to any one of the preceding claims, further comprising: mapping the first landmark using a subset of the plurality of landmark descriptors; as well as wherein the subset of the plurality of landmark descriptors corresponds to a subset of different camera poses used to capture the subset of the plurality of images; and The subset of the plurality of landmark descriptors is selected based on determining that any two camera poses in the subset of the different camera poses do not satisfy the descriptor elimination criterion.
5. A method according to any one of the preceding claims, wherein: The plurality of images includes a target image having the first number of landmarks corresponding to the first number of landmark descriptors, the method further comprising, for the target image: determining, based on the descriptor elimination criteria, that a second number of landmark descriptors corresponding to the landmark subset are prohibited from mapping the landmark subset; Based on determining that the first number or the second number satisfies an image elimination criterion, eliminating the first number of landmark descriptors associated with the third image mapping the plurality of landmarks in the scene.
6. The method according to claim 5, wherein: The image removal criterion requires that a ratio of the second number to the first number exceeds a predetermined threshold.
7. The method according to claim 6, wherein: The target keyframe corresponds to the first camera pose of the two camera poses of the plurality of camera poses, and the subset of landmarks is different from the first landmark, and wherein deselecting the first number of landmark descriptors mapping the plurality of landmarks further comprises: Selecting the first landmark descriptor to map the first landmark is terminated.
8. The method of any one of the preceding claims, each different camera pose comprising a corresponding camera position and a corresponding camera pose, further comprising: for each landmark descriptor of the first landmark, identifying a corresponding image ray connecting the corresponding camera position to the first landmark; as well as identifying a ray angle between two image rays connecting two camera positions of the plurality of camera poses to the first landmark; as well as Determining that the ray angle is less than a ray angle threshold, wherein determining that the two camera poses of the plurality of camera poses satisfy the descriptor elimination criterion is based on determining that the ray angle is less than the ray angle threshold.
9. The method according to claim 8, wherein: The ray angle threshold is in the range of [5°-10°].
10. The method according to any one of the preceding claims, further comprising: identifying a plurality of landmarks including the first landmark in the scene; as well as Mapping data for the scene is generated, the mapping data comprising one or more landmark descriptors corresponding to each landmark in the scene and one or more associated camera poses, wherein each landmark descriptor corresponds to a corresponding image and a corresponding camera pose.
11. The method according to claim 10, wherein: For each landmark in the scene, the one or more landmark descriptors are selected based on determining that any two of the one or more associated camera poses do not satisfy the descriptor elimination criterion.
12. The method according to claim 10, further comprising Get the current frame; Extract multiple feature points from the current frame, where: Each feature point of the plurality of feature points corresponds to an image descriptor determined from the current frame; as well as for each feature point in the plurality of feature points, comparing the image descriptor to the mapping data to identify a matching landmark from the plurality of landmarks; as well as Based on the respective camera poses corresponding to the matching landmarks, a current camera pose for capturing the current frame is determined.
13. The method according to claim 12, wherein the plurality of feature points include a first feature point, and the method further comprises: determining that a first image descriptor of the first feature point is a combination of two landmark descriptors of the matching landmarks; as well as The current camera pose of the current frame is determined based on two camera poses corresponding to the two images from which the two landmark descriptors are determined.
14. The method according to claim 12, further comprising: extracting a second feature point from the current frame, wherein the second feature point corresponds to a second image descriptor determined from the current frame; and Based on determining that the second image descriptor does not match the landmark descriptor in the mapping data, updating the mapping data by associating a second landmark with the second image descriptor of the second landmark and the current camera pose of the current frame.
15. An electronic system comprising: one or more processors; as well as A memory having instructions stored thereon, which, when executed by the one or more processors, cause the processor to perform the method according to any one of claims 1 to 14.
16. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 14.
