Closed loop gesture detection and mapping in SLAM

Abstract

The present application is directed to disclosure of camera pose determination in synchronous localization and mapping (SLAM). An electronic device obtains a sequence of visual images, where each visual image in the sequence of visual images is captured within a corresponding room. For a first visual image in the visual image sequence, the electronic equipment identifies a current room in a plurality of known rooms; wherein the first visual image is captured in the current room; each known room corresponds to a front image set, and the front image sets map the known rooms; each front image of the front image set is associated with a corresponding front gesture in the known room. In accordance with the identification of the current room in the plurality of known rooms, the electronic device identifies a first set of previous images associated with the current room; selecting a first front image from the first front image set to match the first visual image; and determining a first camera pose associated with the first visual image according to a first front pose associated with the first front image.

Description

Technical Field

[0001] Embodiments of the present application relate to image data processing, including but not limited to methods, systems, and non-volatile computer-readable media for determining a camera pose of a camera device in simultaneous localization and mapping (SLAM). Background Art

[0002] Simultaneous localization and mapping (SLAM) is widely used in virtual reality (VR), augmented reality (AR), autonomous driving, and navigation. In SLAM, feature points of a scene are detected and applied to draw a three-dimensional (3D) virtual space corresponding to the scene. Using optical cameras and inertial measurement units (IMUs) present in many mobile devices, feature points of a scene can be easily and accurately drawn to map into a 3D virtual space. In addition, current work detects historical keyframes that match the current image based on a bag-of-words technique. Bag-of-words descriptors are extracted from each historical keyframe and saved in a mapping database to represent the respective historical keyframe. Descriptor matching is used to compare the current image with each keyframe in a set of historical keyframes and determine the matching historical keyframes, thereby forming a loop closure between the current image and the matching historical keyframes in SLAM. However, as the mapping database is expanded to include more historical keyframes, the search time increases, while the search reliability decreases. In some cases, data is collected from a large space corresponding to similar scenes. Many SLAM systems only consider two-dimensional (2D) descriptors, which lack semantic or structural information, and thus often produce erroneous detection results based on this bag-of-words technique. Even if similar historical keyframes can be detected, image matching may fail in the subsequent stage of comparing feature points. Therefore, it would be beneficial to equip the SLAM system with a fast, efficient, and accurate image matching mechanism. Summary of the invention

[0003] Various embodiments of the present application disclose a hierarchical loop closure detection and matching strategy to achieve accurate SLAM results and provide structural level semantic information. A fast, efficient and accurate SLAM-based image matching mechanism is used to identify historical key frames (also called matched previous images) that match the current image and determine the camera pose of the camera, such as when the camera moves between multiple scenes. The current image and the matched historical key frames form a closed loop in SLAM. Specifically, the mobile device acquires RGB images, wide-angle fisheye images, or depth images, which are captured by cameras integrated in the mobile device or other electronic devices. The camera can be an RGB camera or a wide-angle fisheye camera, which is applied to the SLAM system together with an IMU. The pose of the camera has six degrees of freedom (6DOF) and is determined by the SLAM system using images and inertial data collected by the camera and IMU. A 3D model of the scene is reconstructed using 3D image data and related poses. A room detection thread is implemented to check whether the collected data is sufficient to identify the room. Based on the judgment of determining the room from the collected data, the information of the identified room is stored in the room database, the corresponding image is stored as a historical key frame (also called a previous image), and the corresponding room identification is marked.

[0004] After identifying the room in which the current image was taken, room-level matching is performed to determine whether the room has been visited before. Depending on whether the identified room belongs to one of multiple known rooms that have been visited before, the current image will be associated with a series of historical keyframes stored in the corresponding known room. Region and / or object matching and keypoint matching are performed after room-level matching to identify matching historical keyframes, extract previous poses from them, and use them to determine the camera pose associated with the current image. Therefore, room-level matching is a new level of room localization, which is used to reduce the number of historical keyframes to which the current image is compared, thereby significantly improving the speed, efficiency, and accuracy of image matching and pose determination in SLAM.

[0005] In a first aspect, the present application discloses a method for simultaneous localization and mapping (SLAM), which is implemented by an electronic device. The method includes: acquiring a visual image sequence, wherein each visual image in the visual image sequence is captured in a corresponding room; for a first visual image (e.g., a current image) in the visual image sequence, identifying a current room in a plurality of known rooms; wherein the first visual image is captured in the current room; each of the known rooms corresponds to a front image set, and the front image set maps the known room; each front image in the front image set is associated with a corresponding front pose in the known room. The method further includes: according to the identification of the current room in the plurality of known rooms: identifying a first front image set associated with the current room; selecting a first front image from the first front image set to match the first visual image; and determining a first camera pose associated with the first visual image according to a first front pose associated with the first front image.

[0006] In some embodiments, the method further includes: obtaining a depth image sequence synchronized with the visual image sequence; wherein each of the visual images corresponds to a corresponding depth image in the depth image sequence and is associated with a corresponding camera pose; the visual image sequence has a first visual image subset, the first visual image subset is captured in the current room and includes the first visual image. The method further includes: constructing a current three-dimensional room model of the current room based on the first visual image subset and the corresponding first depth image subset. Further, in some embodiments, each of the plurality of known rooms corresponds to a respective embedding set; the method further includes: extracting a first embedding set of the current three-dimensional room model of the current room; wherein the first embedding set includes semantic and instance information describing the current room; the first embedding set includes a plurality of semantic elements configured to provide semantic information of a plurality of objects or regions located in the current three-dimensional room model. Further, in some embodiments, the method further includes: comparing the first embedding set with the corresponding embedding set of each of the known rooms; and determining, based on the comparison result, that the first embedding set satisfies the room recognition criteria, thereby identifying the current room in the plurality of known rooms.

[0007] On the other hand, the present application discloses an electronic device, comprising: one or more processors; and a memory storing instructions; wherein, when the instructions are executed by the one or more processors, the one or more processors perform any of the above methods.

[0008] On the other hand, the present application discloses a non-transitory computer-readable medium storing instructions; when the instructions are executed by one or more processors, the one or more processors perform any of the above methods.

[0009] These illustrative embodiments and implementations are mentioned not to limit or define the disclosure, but to illustrate and aid in understanding the disclosure. Other embodiments are discussed in the detailed description and further description is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various described embodiments, reference should be made to the following detailed description and to the following drawings, wherein like reference numerals represent corresponding parts throughout the drawings.

[0011] Figure 1 is a diagram of a data processing environment having one or more servers communicatively coupled to one or more client devices according to some embodiments.

[0012] Figure 2 is a schematic block diagram of an electronic system for processing data (eg, image data, motion data) according to some embodiments.

[0013] Figure 3 is a schematic diagram of a data processing environment for training and applying neural network-based data processing models to process visual and / or audio data, according to some embodiments.

[0014] Figure 4A is a schematic diagram of a neural network according to some embodiments, which is applied to a neural network-based data processing model to process content data, Figure 4B is an example node in a neural network according to some embodiments.

[0015] Figure 5 is a flow chart for processing inertial sensor data (also referred to as motion data) and image data of an electronic system (eg, a server, a client device, or a combination of both) using a SLAM module according to some embodiments.

[0016] Figure 6 is a flow chart for determining camera pose based on room matching in accordance with some embodiments.

[0017] Figure 7 is a schematic diagram of multiple visual images captured as a camera moves between multiple rooms, according to some embodiments.

[0018] Fig. 8A is a block diagram showing an example room with projected wall points on a two-dimensional floor plan according to some embodiments, Figure 8B is a block diagram of digitizing an example room into a two-dimensional room diagram in accordance with some embodiments.

[0019] Fig. 9 is a flow chart of identifying a subset of previous images that are substantially similar to a first visual image based on region matching according to some embodiments.

[0020] Fig.10 is a flow chart of matching the region 904 in the first visual image with a corresponding region in one of a plurality of known rooms using key points according to some embodiments.

[0021] Fig.11 is a flow chart of an example method of determining a camera pose associated with a visual image in accordance with some embodiments.

[0022] Reference numerals refer to corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0023] Reference will be made in detail to specific embodiments of the present application, examples of which are illustrated in the accompanying drawings. In the following detailed description, many non-limiting specific details are provided to assist in understanding the subject matter of this document. However, it will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be implemented without these specific details. For example, it will be apparent to those skilled in the art that the subject matter provided herein may be implemented on many electronic devices having digital video capabilities.

[0024] Extended reality includes augmented reality (AR), in which virtual objects are superimposed on the view of the real physical world; virtual reality (VR), which contains only virtual content; and mixed reality (MR), which combines AR and VR and allows users to interact with the real world and digital objects in real time. Key technologies in AR include SLAM and physical world reconstruction. Image data and inertial motion data are collected and processed using SLAM technology, and enable the 6DOF tracking system to identify fixed feature points distributed in the scene in many mobile AR systems. In some embodiments, the image data includes RGBD images, and each RGBD image has a color image and a depth image including red, green, and blue band information. Optionally, the color image used in SLAM can be used to perform monocular depth estimation on the depth image using deep learning technology. SLAM tools are developed to locate the user's position and build a 3D representation of the environment. A computer model of the 3D appearance of the scene is established based on the feature point information in the scene. In the loop closure detection process, the location visited by the camera is detected by establishing a correspondence between the feature points of the current image and each historical key frame.

[0025] The estimation error of the scene 3D map is cumulative and propagable, and the camera pose determined in SLAM has drift. The quality of the reconstructed 3D map and the accuracy of the camera pose generated by SLAM are determined by the loop closure detection and matching technology, that is, the camera must return to a position with no or only a small mismatch with the corresponding camera pose to achieve the desired quality of the reconstructed 3D map and the accuracy of the camera pose. However, many indoor places may look similar to each other. When identifying the corresponding historical key frame matching the current frame from the historical key frame dataset, ambiguity often occurs. This increases the difficulty of loop closure detection and leads to many wrong loop closure results. In some embodiments, a hierarchical loop closure detection strategy is applied based on room matching. Room-level information is generated for the current image and matched with the corresponding room-level information of a set of historical key frames of multiple known rooms. In some embodiments, the shooting room of the current image is identified in multiple known rooms. The known room identified as the shooting room of the current image corresponds to a set of historical key frames. According to object or region matching and key point matching, a historical image is identified from the set of historical key frames to match the current image. Through these methods, this hierarchical loop closure detection can effectively reduce false detection results and improve the positive detection rate and matching speed.

[0026] Figure 1 1 is a schematic diagram of a data processing environment 100 according to some embodiments, the environment having one or more servers 102, which are communicatively coupled to one or more client devices 104. The one or more client devices 104 can be a desktop computer 104A, a tablet computer 104B, a mobile phone 104C, a head-mounted display (HMD) (also known as augmented reality (AR) glasses) 104D, or an intelligent, multi-sensing, networked home device (e.g., a surveillance camera, a smart TV device, a drone). Each client device 104 can collect data or user input, execute a user application, and display output on its user interface. The collected data or user input can be processed locally on the client device 104 or remotely by another client device 104 or server 102. One or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client device 104, and in some embodiments, process data and user input received from the client device 104 when the user application is executed on the client device 104. In some embodiments, data processing environment 100 further includes memory 106 for storing data related to server 102 , client device 104 , and applications executing on client device 104 .

[0027] The one or more servers 102 can communicate real-time data with client devices 104 that are remote from each other or from the one or more servers 102. In addition, in some embodiments, the one or more servers 102 can perform data processing tasks that the client device 104 cannot or preferably does not perform locally. For example, the client device 104 includes a game console (e.g., HMD 104D) that executes an interactive online game application. The game console receives user instructions and sends them to the game server 102 together with user data. The game server 102 generates a video data stream based on the user instructions and user data, and provides the video data stream to the game console and other client devices that are in the same game session with the game console for display.

[0028] The one or more servers 102, the one or more client devices 104, and the memory 106 are communicatively coupled to each other via one or more communication networks 108, which are media for providing communication links between these devices and computers connected together in the data processing environment 100. The one or more communication networks 108 may include wired, wireless communication links, or connections such as fiber optic cables. Examples of the one or more communication networks 108 include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), or a combination thereof. The one or more communication networks 108 may be implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FireWire, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol. The connection to the one or more communication networks 108 can be established directly (e.g., using a 3G / 4G connection to a wireless carrier), can be established through a network interface 110 (e.g., a router, switch, gateway, hub, or intelligent, dedicated whole-house control node), or through any combination thereof. Thus, the one or more communication networks 108 can represent an Internet of global networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite. At the heart of the Internet are high-speed data communication lines between major nodes or hosts, consisting of thousands of commercial, government, educational, and other computer systems, for routing data and messages.

[0029] In some embodiments, deep learning techniques are applied to the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed on the client device 104 to identify information contained in the content data, match the content data with other data, classify the content data, or synthesize related content data. The content data may generally include inertial sensor data captured by an inertial sensor of the client device 104. In these deep learning techniques, data processing models are created based on one or more neural networks to process the content data. Before being applied to process the content data, these data processing models need to be trained with training data. After the model is trained, the mobile phone 104C or HMD 104D obtains the content data (e.g., video data captured by an internal camera) and processes the content data locally using the data processing model.

[0030] In some embodiments, model training and data processing are implemented locally on each individual client device 104 (e.g., mobile phone 104C and HMD 104D). The client device 104 obtains training data from the one or more servers 102 or storage 106, and applies the training data to train the data processing model. Alternatively, in some embodiments, model training and data processing are implemented remotely on a server 102 (e.g., server 102A) associated with a client device 104 (e.g., client device 104A and HMD 104D). Server 102A obtains training data from itself, another server 102, or storage 106, and applies the training data to train the data processing model. The client device 104 obtains content data, sends the content data to server 102A (e.g., in an application) to perform data processing using the trained data processing model, receives data processing results (e.g., recognized gestures) from server 102A, displays the results on a user interface (e.g., associated with an application), presents a virtual object in the field of view based on the gesture, or performs other functions based on the results. Before sending the content data to the server 102A, the client device 104 itself does not perform any or little data processing on the content data. In addition, in some embodiments, data processing is implemented locally on the client device 104 (e.g., the client device 104B and the HMD 104D), and model training is implemented remotely on the server 102 associated with the client device 104 (e.g., the server 102B). The server 102B obtains training data from itself, another server 102, or the memory 106, and applies the training data to train the data processing model. The trained data processing model can be selectively stored in the server 102B or the memory 106. The client device 104 imports the trained data processing model from the server 102B or the memory 106, processes the content data using the data processing model, and generates data processing results for display on the user interface or for locally initiating certain functions (e.g., presenting a virtual object based on the device posture).

[0031] In some embodiments, a pair of AR glasses 104D (also referred to as HMD) are communicatively coupled in the data processing environment 100. The AR glasses 104D include a camera, a microphone, a speaker, one or more inertial sensors (e.g., a gyroscope, an accelerometer), and a display screen. The camera and microphone are used to capture video and audio data from the scene of the AR glasses 104D, and the one or more inertial sensors are used to capture inertial sensor data. In some cases, the camera captures the movements of the user wearing the AR glasses 104D and recognizes gestures locally in real time using a two-level gesture recognition model. In some cases, the microphone records ambient sounds, including the user's voice commands. In some cases, the video or static visual data captured by the camera and the inertial sensor data measured by the one or more inertial sensors are used to determine and predict the posture of the device. The video, static image, audio, or inertial sensor data captured by the AR glasses 104D is processed by the AR glasses 104D, the server 102, or both to recognize the device posture. Optionally, deep learning techniques are applied by the server 102 and the AR glasses 104D to recognize and predict the device posture. The device gesture is used to control the AR glasses 104D itself or to interact with an application (e.g., a game application) executed by the AR glasses 104D. In some embodiments, the display screen of the AR glasses 104D displays a user interface, and the recognized or predicted device gesture is used to present or interact with user-selectable display items (e.g., avatars) on the user interface.

[0032] As described above, in some embodiments, deep learning techniques are applied to the data processing environment 100 to process video data, static image data, or inertial sensor data captured by the AR glasses 104D. Based on the video, static image, and / or inertial sensor data, a first data processing model is used to identify and predict the 2D or 3D device posture. Visual content can be generated using a second data processing model. The training of the first and second data processing models can be implemented by the server 102 or the AR glasses 104D. The inference of the device posture and visual content is implemented independently by the server 102 and the AR glasses 104D, or by the server 102 and the AR glasses 104D.

[0033] Figure 2 1 is a schematic block diagram of an electronic system 200 for processing content data (eg, image data) according to some embodiments. The electronic system 200 includes a server 102, a client device 104 (eg, Figure 1The electronic system 200 typically includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 (also referred to as chipsets) for connecting these components. The electronic system 200 includes one or more input devices 210 for facilitating user input, such as a keyboard, a mouse, a voice command input unit or a microphone, a touch screen display, a touch-sensitive input pad, a gesture capture camera, or other input buttons or controls. In addition, in some embodiments, the client device 104 of the electronic system 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard. In some embodiments, the client device 104 includes one or more optical cameras 260 (e.g., an RGB camera), a scanner, or a photoelectric sensor unit for capturing images, for example, for capturing images of a graphic serial code printed on an electronic device. The electronic system 200 also includes one or more output devices 212 for presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the client device 104 includes a location detection device, such as a GPS (Global Positioning System) or other geolocation receiver, for determining the location of the client device 104 .

[0034] Optionally, client device 104 includes an inertial measurement unit (IMU) 280 that integrates sensor data captured by multi-axis inertial sensors to provide an estimate of the position and orientation in space of client device 104. Examples of one or more inertial sensors of IMU 280 include, but are not limited to, gyroscopes, accelerometers, magnetometers, and inclinometers.

[0035] The memory 206 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory; and optionally includes a non-volatile memory, such as one or more magnetic disk memories, one or more optical disk memories, one or more flash memories, or one or more other non-volatile solid-state memories. Optionally, the memory 206 includes one or more storage devices that are remote from the one or more processing units 202. The memory 206 (or the non-volatile memory in the memory 206) includes a non-volatile computer-readable storage medium. In some embodiments, the memory 206 or the non-volatile computer-readable storage medium in the memory 206 stores the following programs, modules, and data structures or a subset or superset thereof:

[0036] @operating system 214, including programs for handling various basic system services and performing hardware-related tasks;

[0037] @Network communication module 216, used to connect each server 102 or client device 104 to other devices (such as server 102, client device 104, or storage 106) through one or more network interfaces 204 (wired or wireless) and one or more communication networks 108 (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0038] @User interface module 218, which is used to present information (e.g., graphical user interfaces of applications 224, widgets, websites and their web pages and / or games, audio and / or video content, text, etc.) on each client device 104 through one or more output devices 212 (e.g., displays, speakers, etc.);

[0039] @Input processing module 220, used to detect one or more user inputs or interactions from one or more input devices 210, and interpret the detected inputs or interactions;

[0040] A web browser module 222 for browsing, requesting (e.g., via HTTP), and displaying websites and web pages thereof, including a web interface for logging into a user account associated with a client device 104 or other electronic device, controlling a client or electronic device associated with a user account, and editing and viewing settings and data associated with a user account;

[0041] One or more user applications 224 for execution by the electronic system 200; for example, games, social networking applications, smart home applications, and / or other web-based or non-web-based applications; for controlling other electronic devices and viewing data captured by these devices;

[0042] @Model training module 226, for receiving training data and establishing a data processing model to process content data (e.g., video, image, audio, or text data) collected or acquired by the client device 104;

[0043] @data processing module 228, for processing content data using data processing model 250, thereby identifying information contained in the content data, matching the content data with other data, classifying the content data, or synthesizing related content data; wherein, in some embodiments, data processing module 228 is associated with one of user applications 224 to process content data in response to user instructions received from the user application 224;

[0044] @Posture determination and prediction module 230, for determining and predicting the posture of the client device 104 (e.g., AR glasses 104D); wherein, in some embodiments, the posture is jointly determined and predicted by the posture determination and prediction module 230 and the data processing module 228;

[0045] a gesture-based rendering module 238 for rendering virtual objects within the field of view of a camera 260 of the client device 104 or creating mixed, virtual, or augmented reality content using images captured by the camera 260; wherein the virtual objects are rendered and the mixed, virtual, or augmented reality content is created from the perspective of the camera 260 according to a camera pose of the camera 260; and

[0046] @One or more databases 240, used to store data including at least one or more of the following:

[0047] Device settings 242, including common device settings for one or more servers 102 or client devices 104 (e.g., service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0048] User account information 244 for one or more user applications 224, such as user name, security questions, account history data, user preferences, and predefined account settings;

[0049] · Network parameters 246 of one or more communication networks 108, such as IP addresses, subnet masks, default gateways, DNS servers, and host names;

[0050] @ training data 248, used to train one or more data processing models 250;

[0051] @Data processing model 250, for processing content data (such as video, image, audio, or text data) using deep learning technology;

[0052] A gesture data database 252 for storing gesture data of the camera 260;

[0053] content data and results 254 obtained by and output to a client device 104 of the electronic system 200; wherein the content data is processed by the data processing model 250 locally on the client device 104 or remotely by the server 102 to provide relevant results presented on the client device 104, including candidate images; and

[0054] a room database 256 for storing information of 3D room models of a plurality of rooms; wherein the image data is captured by a camera 260; and the information of the 3D room models includes at least one set of respective embeddings,

[0055] It further includes semantic and instance information describing each room.

[0056] In some embodiments, the posture determination and prediction module 230 further includes a SLAM module 232 for mapping the scene in which the client device 104 is located and identifying the posture of the client device 104 in the scene using the image and IMU sensor data. In addition, in some embodiments, the SLAM module 232 includes a multi-level mapping module 234 for matching the current image with historical keyframes at the room level, object and area level, and keypoint level. Based on the previous posture of the client device 104 associated with the historical keyframe, the camera posture of the client device 104 associated with the current image is determined.

[0057] Optionally, the one or more databases 240 are stored in one of the following: the server 102, the client device 104, and the memory 106 of the electronic system 200. Optionally, the one or more databases 240 are distributed in the following combination: the server 102, the client device 104, and the memory 106 of the electronic system 200. In some embodiments, multiple copies of the above data are stored in different devices, for example, two copies of the data processing model 250 are stored in the server 102 and the memory 106 respectively.

[0058] Each of the above-mentioned identified elements can be stored in the above-mentioned one or more memory devices and corresponds to a set of instructions for performing the above-mentioned functions. The above-mentioned determined modules or programs (i.e., instruction sets) do not need to be implemented as separate software programs, processes, modules, or data structures, so various subsets of these modules can be combined or otherwise rearranged in various embodiments. In some embodiments, the memory 206 can store subsets of the above-mentioned modules and data structures. In addition, the memory 206 can store other modules and data structures not described above.

[0059] Figure 3is a schematic diagram of another data processing system 300 for training and applying a neural network (NN-based) data processing model 250 to process content data (e.g., video, image, audio, or text data). The data processing system 300 includes a model training module 226 for establishing the data processing model 250 and a data processing module 228 for processing the content data using the data processing model 250. In some embodiments, the model training module 226 and the data processing module 228 are both located on a client device 104 of the data processing system 300, and different training data sources 304 on the client device 104 provide training data 306 to the client device 104. The training data source 304 can be a server 102 or a memory 106. Alternatively, in some embodiments, the model training module 226 and the data processing module 228 are both located on a server 102 of the data processing system 300. The training data source 304 that provides the training data 306 can be the server 102 itself, another server 102, or the memory 106. Furthermore, in some embodiments, the model training module 226 and the data processing module 228 are located on the server 102 and the client device 104 , respectively, and the server 102 provides the trained data processing model 250 to the client device 104 .

[0060] The model training module 226 includes one or more data preprocessing modules 308, a model training engine 310, and a loss control module 312. The data processing model 250 is trained according to the type of content data to be processed. The training data 306 is consistent with the type of the content data. Therefore, the data preprocessing module 308 is used to process the training data 306, and its type is consistent with the type of the content data. For example, the image preprocessing module 308A is configured to process the image training data 306 into a predefined image format, for example, extracting a region of interest (ROI) in each training image, and cropping each training image to a predefined image size. Alternatively, the audio preprocessing module 308B is configured to process the audio training data 306 into a predefined audio format, for example, using Fourier transform to convert each training sequence into the frequency domain. The model training engine 310 receives the preprocessed training data provided by the data preprocessing module 308, further processes the preprocessed training data using the existing data processing model 250, and generates an output for each training data item. During this process, the loss control module 312 can monitor the loss function and compare the output associated with the respective training data item with the true value of the respective training data item. The model training engine 310 modifies the data processing model 250 to reduce the loss function until the loss function meets the loss criterion (e.g., the comparison result of the loss function is minimized or reduced to below the loss threshold). The modified data processing model 250 is provided to the data processing module 228 to process the content data.

[0061] In some embodiments, the model training module 226 provides supervised learning, where the training data is fully labeled and includes the expected output (also referred to as ground truth in some cases) for each training data item. In contrast, in some embodiments, the model training module 226 provides unsupervised learning, where the training data is unlabeled. The model training module 226 is configured to identify previously undetected patterns in the training data that have no pre-existing labels and have little or no human supervision. In addition, in some embodiments, the model training module 226 provides partially supervised learning, where the training data is partially labeled.

[0062] The data processing module 228 includes a data preprocessing module 314, a model-based processing module 316, and a data post-processing module 318. The data preprocessing module 314 preprocesses the content data according to the type of the content data. The function of the data preprocessing module 314 is consistent with that of the preprocessing module 308, and the content data is converted into a predefined content format acceptable to the input of the model-based processing module 316. Examples of content data include one or more of the following: video, image, audio, text, and other types of data. For example, each image is preprocessed to extract an area of ​​interest or cut to a predefined image size, and the audio clip is preprocessed to be converted to the frequency domain using Fourier transform. In some cases, the content data includes two or more types, such as video data and text data. The model-based processing module 316 applies the trained data processing model 250 provided by the model training module 226 to process the preprocessed content data. The model-based processing module 316 can also monitor an error indicator to determine whether the content data has been correctly processed in the data processing model 250. In some embodiments, the processed content data is further processed by the data post-processing module 318 to present the processed content data in a preferred format or to provide other relevant information that can be derived from the processed content data.

[0063] Figure 4A is a schematic diagram of a neural network (NN) 400 applied to a neural network-based data processing model 250 to process content data according to some embodiments, Figure 4B4 is an example node 420 in a neural network (NN) 400 according to some embodiments. The data processing model 250 is established based on the neural network 400. The corresponding model-based processing module 316 applies the data processing model 250, including the neural network 400, to process the content data converted into a predefined content format. The neural network 400 includes a set of nodes 420 connected by links 412. Each node 420 receives one or more node inputs and applies a propagation function to generate a node output from the one or more node inputs. Since the node output is provided to one or more other nodes 420 through one or more links 412, the weight w associated with each link 412 is applied to the node output. Similarly, according to the propagation function, the one or more node inputs are combined according to corresponding weights w1, w2, w3, and w4. In one example, the propagation function is the product of a nonlinear activation function and a linear weighted combination of the one or more node inputs.

[0064] The collection of nodes 420 is organized into one or more layers in the neural network 400. Optionally, the one or more layers include a single layer that serves as both an input layer and an output layer. Optionally, the one or more layers include an input layer 402 for receiving input, an output layer 406 for providing output, and zero or more hidden layers 404 (e.g., 404A and 404B) between the input layer 402 and the output layer 406. The deep neural network has multiple hidden layers 404 between the input layer 402 and the output layer 406. In the neural network 400, each layer is only connected to its previous layer and / or the next layer. In some embodiments, the layer 402 or 404B is a fully connected layer because each node 420 in the layer 402 or 404B is connected to each node 420 in the next layer. In some embodiments, the one or more hidden layers 404 include two or more nodes that are connected to the same node in the immediately adjacent layer for downsampling or pooling the nodes 420 between the two layers. In particular, max pooling uses the maximum value of two or more nodes in layer 404B to generate nodes in a subsequent layer 406 that are connected to the two or more nodes.

[0065] In some embodiments, a convolutional neural network (CNN) is applied to the data processing model 250 to process content data (particularly video and image data). CNN uses a convolution operation and belongs to a class of deep neural networks 400, i.e., a feedforward neural network that only moves data forward from the input layer 402 through the hidden layer to the output layer 406. One or more hidden layers of a CNN are convolutional layers, which are convolved by multiplication or dot product. Each node in a convolutional layer receives input (e.g., five nodes) from a receiving area associated with the previous layer, which is smaller than the entirety of the previous layer and may vary depending on the position of the convolutional layer in the convolutional neural network. Video or image data is preprocessed into a predefined video / image format corresponding to the CNN input. The preprocessed video or image data is abstracted into respective feature maps by each layer of the CNN. Through these methods, video and image data can be processed by CNN for video and image recognition, classification, analysis, imprinting, or synthesis.

[0066] In addition, in other embodiments, a recurrent neural network (RNN) is applied in the data processing model 250 to process content data (particularly text and audio data). The nodes in the RNN layer follow the time sequence so that the RNN exhibits time dynamic behavior. In one example, each node 420 of the RNN has a real value activation that varies over time. Examples of RNN include but are not limited to long short-term memory (LSTM) networks, complete loop networks, Elman networks, Jordan networks, Hopfield networks, bidirectional associative memory (BAM networks), echo state networks, independent RNNs (IndRNNs), recursive neural networks, and neural history compressors. In some embodiments, RNNs can be used for handwriting or speech recognition. It should be noted that in some embodiments, the data processing module 228 processes two or more types of content data, and applies two or more types of neural networks (such as CNN and RNN) to jointly process the content data.

[0067] The training process is to calibrate all weights w of each layer of the learning model using the training data set provided in the input layer 402 i The training process typically includes two steps, forward propagation and backward propagation, which are repeated multiple times until predefined convergence conditions are met. In forward propagation, sets of weights of different layers are applied to input data and intermediate results from previous layers. In backward propagation, the tolerance of the output error (e.g., loss function) is measured, and the weights are adjusted accordingly to reduce the error. The activation function can be linear, rectified linear unit, S-type, hyperbolic tangent, or other types. In some embodiments, before applying the activation function, the network bias term b is added to the sum of the weighted outputs of the previous layer. The network bias b provides a disturbance that helps NN400 avoid overfitting the training data. The results of the training include the network bias parameter b of each layer.

[0068] Figure 5 According to some embodiments, a SLAM module (e.g. Figure 2 232 in Figure 5 is a flowchart 500 for processing inertial sensor data (also referred to as motion data) and image data of an electronic system (e.g., server 102, client device 104, or a combination of both). The process 500 includes measurement preprocessing 502, initialization 504, local visual inertial odometry (VIO) with relocalization 506, and global pose graph optimization 508. In measurement preprocessing 502, camera 260 captures image data of a scene at an image frame rate (e.g., 30 FPS), and detects and tracks features from the image data (510). Inertial measurement unit (IMU) 280 measures inertial sensor data at a sampling frequency (e.g., 1000 Hz) while camera 260 captures image data, and the inertial sensor data is pre-integrated (512) to provide pose data. In initialization 504, the image data captured by camera 260 and the inertial sensor data measured by IMU 280 are aligned in time (514). A vision-only structure-from-motion (SfM) technique 514 is applied ( 516 ) to couple the image data and the inertial sensor data, estimate the three-dimensional structure, and map the scene of the camera 260 .

[0069] After initialization 504, in relocalization 506, the sliding window 518 and the associated states from loop closure 520 are used to optimize 522 the VIO. When the VIO corresponds 524 to a keyframe for a smooth video transition, and a corresponding loop is detected 526, features are retrieved 528 and used to generate the associated states from the loop closure 520. In global pose graph optimization 508, the multi-DOF pose graph 530 is optimized based on the states of loop closure 520, and the keyframe database 532 is updated with the keyframes associated with the VIO.

[0070] In addition, the features of detection and tracking (510) are used to monitor (534) the motion of the object in the image data and estimate the image-based pose 536, for example, based on the image frame rate. In some embodiments, the pre-integrated (513) inertial sensor data can be propagated (538) based on the motion of the object and used to estimate the inertial-based pose (540), for example, based on the sampling frequency of the IMU 280. The image-based pose 536 and the inertial-based pose 540 are stored in a pose data buffer and used by the SLAM module 232 to estimate and predict poses. Alternatively, in some embodiments, the SLAM module 232 receives the inertial sensor data measured by the IMU 280 and obtains the image-based pose 536 to estimate and predict more poses.

[0071] In SLAM, high-frequency pose estimation is achieved through sensor fusion, which relies on data synchronization between imaging sensors and IMU 280. Imaging sensors (e.g., camera 260, lidar) provide image data required for pose estimation and typically operate at low frequencies (e.g., 30 frames per second) and large delays (e.g., 30 milliseconds). In contrast, IMU 280 can measure inertial sensor data and operate at extremely high frequencies (e.g., 1000 samples per second) with negligible latency (e.g., <0.1 milliseconds). Asynchronous time warping (ATW) is commonly used in AR systems to warp images before they are sent to the display to correct for head movements that occur after the image is rendered. The ATW algorithm can reduce image latency, increase or maintain frame rates, or reduce jitter caused by lost image frames. In SLAM and ATW, related image data and inertial sensor data are stored locally for synchronization and use in pose estimation / prediction. In some embodiments, image and inertial sensor data are stored in one of a plurality of STL containers, such as std::vector, std::queue, std::list, etc., or other custom containers. These containers are generally very convenient to use. Image and inertial sensor data are stored in STL containers along with timestamps, which are used for data searching, data insertion, and data organization.

[0072] In some embodiments, the camera 260 includes a wide-angle fisheye camera or an RGB camera, and the SLAM module 232 uses the images captured by the camera 260 and the IMU data to determine the camera pose of the camera 260. Compared with using only the image data, the camera pose obtained using the image and the IMU data is more accurate. In some cases, the direction of gravity is derived from the IMU data and used to reconstruct the three-dimensional model, in which the normal of the ground plane is parallel to the direction of gravity. The normal of the ground plane helps to distinguish a known room from one or more other similar rooms.

[0073] Figure 6 6 is a flow chart of a process 600 for determining a camera pose based on room matching according to some embodiments. The process 600 is performed by the SLAM module 232 (particularly Figure 2The SLAM module 232 acquires image data 602 and motion data 604. The image data 602 is captured by the camera 260 of the electronic device (e.g., the AR glasses 104D), and the motion data 604 is measured by the IMU 280 of the electronic device. The SLAM module 232 reconstructs a 3D room model 606 based on the image data 602. In some embodiments, a subset of the image data 602 includes a first visual image 602A (e.g., a current image most recently captured by the camera 260). The 3D room model 606 corresponds to the current room in which the subset of the image data 602 is captured. The electronic system 200 detects (608) the current room based on the 3D room model 606 and determines whether the current room is in a plurality of known rooms. If the current room (610) is in a plurality of known rooms, the current room corresponds to a matching known room. Information about the matching known room is extracted from the room database 256 and stored (612) (e.g., room embedding) with information about the current room.

[0074] Each known room corresponds to a set of front images that map the respective known room. The first set of front images is associated with the matching known room. Given that the current room matches the matching known room, the first set of front images is associated with the current room. The first front image is selected from the first set of front images associated with the matching known room to match the first visual image 602A most recently captured by the camera 260. In some embodiments, a subset of the first set of front images is determined to be substantially similar to the first visual image 602A (614). Based on a coarse level of object or area matching and a fine level of key point matching, the first front image is further selected from the subset of the first set of front images (616). The information of the first front image is used to update the 3D room model 606 of the current room. In addition, in some embodiments, the information of the first front image includes a first front pose, and a first camera pose associated with the first visual image is determined based on the first front pose associated with the first front image.

[0075] As camera 260 continuously captures images in image data 602, 3D room model 606 is continuously updated based on information from the latest image (e.g., feature points extracted from the latest image). In particular, when camera 260 initially enters the current room, 3D room model 606 is created based on limited image data 602 and cannot fully represent the current room. As camera 260 begins to collect more and more image data 602 in the current room, 3D room model 606 is gradually completed until 3D room model 606 can fully represent the current room. After 3D room model 606 is completed, image data 602 continues to update 3D room model 606 to improve the accuracy of 3D room model 606.

[0076] Figure 7is a schematic diagram of multiple visual images 700 captured when the camera moves between multiple rooms according to some embodiments. The multiple visual images 700 are captured by the camera 260 (e.g., a wide-angle fisheye camera, an RGB camera), including a first visual image subset 702, a second visual image subset 704, and a third front image subset 706, which is before the first visual image subset 702. Each of the multiple visual images 700 is taken in a respective room (e.g., a first known room 708A, a second known room 708B, ..., an Nth known room 708N). Each image 700 can be a color image or a monochrome image. The first visual image subset 702 is captured in the current room 710, including the first visual image 602A. The third front image subset 706 includes multiple front image sets 706A, 706B, ..., and 706N corresponding to multiple known rooms 708A, 708B, ..., and 708N. Before taking the first visual image subset 702 and the second visual image subset 704, a third front image subset 706 is taken for the known rooms 708. For each known room 708, a respective 3D room model is created from the respective front image 706.

[0077] After capturing the first visual image 602A, a current room 710 is identified in the plurality of known rooms 708. In one example, a first front image set 706A corresponding to the first known room 708A is identified as being associated with the current room 710, for example, based on a respective 3D room model of the first known room 708A. A first front image 712 is selected from the first front image set 706A to match the first visual image 602A. The first front image 712 has a first front pose and the first visual image 602A has a first camera pose. Given that the first visual image 602A matches the first front image 712, the first camera pose is determined based on the first front pose. Therefore, a loop closure occurs in pose estimation between the first visual image 602A of the first visual image subset 702 and the first front image 712A of the first front image set 706A.

[0078] In some embodiments, the sequence of depth images 720 is synchronized with the sequence of visual images 700. Each visual image corresponds to a respective depth image and is associated with a respective camera pose. Based on the first visual image subset 702 and the corresponding first depth image subset 720A, a current 3D room model 732 is constructed for the current room 710. Assume that the first visual image subset 702 starts with an initial image 714, which is captured when the camera 260 initially enters the current room 710. As the camera 260 continuously captures images 702, the current 3D room model 732 is continuously updated based on the information of each newly captured image (e.g., based on feature points extracted from the first visual image 602A). In particular, when the camera 260 initially enters the current room, the current 3D room model 732 is created based on limited image data 602 and cannot fully represent the current room 710. As the camera 260 begins to collect more and more image data 702 in the current room 710, the current 3D room model 732 is gradually completed, for example, before the first visual image 602A is captured. After the 3D room model 732 is complete, the current 3D room model 732 may optionally continue to be updated with newly captured images (eg, the second visual image 602B subsequent to the first visual image 602A).

[0079] In some embodiments, the first visual image subset 702 includes a second visual image 602B that immediately follows the first visual image 602A. A second camera pose associated with the second visual image 602B is determined based on a second front pose associated with a second front image 712B in a first front image set 706A, the first front image set 706A corresponding to a current room identified in a plurality of known rooms 708 (e.g., a first known room 708A). The second visual image 602B immediately follows the first visual image 602A, or is separated from the first visual image by an integer number of visual images. Loop closure occurs in the pose estimation between the second visual image 602B in the first visual image subset 702 and the second front image 712B in the first front image set 706A. Alternatively, in some embodiments, a current 3D room model 732 is established and allows the current room 710 to be associated with the first known room 708A. However, the first or second camera pose of the first or second visual image 602A or 602B is determined based on the current 3D room model 732 and the corresponding first visual image subset 702, rather than based on the first previous image set 706A. By these methods, loop closure occurs locally in the pose estimation between the second visual image 602B and another earlier visual image of the first visual image subset 702.

[0080] In some embodiments, the first visual image subset 702 includes a third visual image 602C that precedes the first visual image 602A. At the time the third visual image 602C is acquired, the current 3D room model 732 is incomplete for the first visual image subset 702. Based on the determination that the current 3D room model 732 is incomplete for the first visual image subset 702, the third visual image 602C is not related to any known room 708 or a corresponding previous image 706 of any known room 708. A third camera pose associated with the third visual image 602C is determined based on a camera pose associated with an earlier visual image in the first visual image subset 702. The third visual image 602C immediately precedes the first visual image 602A or is separated from the first visual image 602A by an integer number of visual images in the first visual image subset 702.

[0081] In other words, in some embodiments, the first visual image subset 702 includes an alternative visual image 716 (e.g., the second visual image 602B, the third visual image 602C). The alternative camera pose associated with the alternative visual image 716 is determined based on the camera pose associated with the earlier visual image in the first visual image subset 702. The alternative visual image 716 is immediately before or after the first visual image, or is separated from the first visual image 602A by an integer number of visual images in the first visual image subset 702.

[0082] After the current 3D room model 723 is completed, a first embedding set 734 is extracted from the current 3D room model 732 of the current room 710. The first embedding set 734 includes semantic and instance information describing the current room 710. The first embedding set 734 includes a plurality of semantic elements configured to provide semantic information of a plurality of objects or regions located in the current 3D room model 732. In addition, in some embodiments, each of the plurality of known rooms 708 corresponds to a respective embedding set 724. The first embedding set is compared with the corresponding embedding set 724 of each known room 708. According to the comparison result, the first embedding set 734 satisfies the room identification criterion 740, thereby identifying the current room 710 in the plurality of known rooms 708, for example, as the first known room 708A.

[0083] In one example, for each known room 708 (e.g., 708A, 708B, ..., 708N), a similarity level between the first embedding set 734 and the respective embedding set 724 of the respective known room is determined. The first embedding set 734 of the current room 710 has a first similarity level 726A with the embedding set corresponding to one of the plurality of known rooms 708 (e.g., 708A). The first embedding set 734 has a plurality of similarity levels 726 with the respective embedding sets 724 corresponding to the plurality of known rooms 708. If the first similarity level 726A is greater than a similarity threshold and is a maximum value among the plurality of first similarity levels 726, the current room 710 is identified as one of the plurality of known rooms 708 (e.g., 708A).

[0084] In some embodiments, the sequence of depth images 720 includes a second depth image subset 720B corresponding to the second visual image subset 704. The second visual image subset 704 is captured in the next room 718. A next 3D room model 736 of the next room 718 is created by the second visual image subset 704 and the corresponding second depth image subset 720B. A second embedding set 738 is extracted from the next 3D room model 736, including semantic and instance information describing the next room 718. For each known room 708, the second embedding set has a similarity level 726' with the corresponding embedding set of the respective known room. The second embedding set has a second similarity level 726A' with the embedding set corresponding to one of the plurality of known rooms 708. The second embedding set 738 has a plurality of second similarity levels 726', where the respective embeddings 724 correspond to the plurality of known rooms 708. If the second similarity level 726A' is the largest among the plurality of second similarity levels 726', but is less than the similarity threshold, the next room is not in the plurality of known rooms, i.e., is not the same as any of the plurality of known rooms. In addition, in some embodiments, the next room 718 is added as a new room to the plurality of known rooms 708. The plurality of known rooms 708 is extended to include the next room 718, and the next room 718 is associated with the second embedding set 738.

[0085] Each known room 708 is associated with a previous 3D room model constructed based on a set of previous images 706, and the current room and the next room are associated with the current 3D room model or the next 3D room model constructed based on the first visual image subset 702 and the second visual image subset 704, respectively. Each previous, current, or next room is associated with a respective embedding set, including semantic and instance information of the respective 3D room model. In some embodiments, for each previous, current, or next room, the information of the respective 3D room model is saved to the room database 256, and the corresponding image is marked with the respective room identifier (e.g., roomID). According to the respective room identifier, the information of each known room is extracted from the room database 256. In some cases, the instance information includes an embedding vector, which organizes a set of corresponding 3D room model embeddings. If two embedding vectors are similar, the corresponding rooms are regarded as the same / similar rooms, that is, matched with each other. The room model is registered in the room database 256, and it is determined whether the corresponding room is the same room based on the embedding vector. In addition, in some embodiments, for two registered room models, if the feature points of the two rooms are close and the semantic labels are the same, the two corresponding rooms can be successfully matched.

[0086] Fig. 8A is a block diagram showing an example room 800 with projected wall points on a two-dimensional floor plan 802 in accordance with some embodiments, Figure 8B is a block diagram of digitizing an example room 800 into a two-dimensional room map 850 according to some embodiments. Figure 7 After the current room model 732 in the 3D image is obtained, a feature point set 804 associated with each wall 806 is determined based on the semantic information of the 3D instance and the respective wall 806. Each wall 806 and the corresponding feature point set 804 are projected onto a 2D plan view 802 having a normal direction aligned with the gravity direction. The projected 2D plan view 802 is vectorized into a 2D digital room map 850.

[0087] In some embodiments, a room segmentation technique is applied to extract the room 800 (e.g., the current room 710). When the camera 260 initially enters the current room 710, the current 3D room model 732 is created using limited image data 602 and cannot fully represent the current room 710. As the camera 260 begins to collect more and more image data 602 in the current room 710, the current 3D room model 732 is gradually completed until the current 3D room model 732 can fully represent the current room 710. Before the 3D room model 732 is completed, the room detection thread adds the newly reconstructed wall points 804 to the 2D floor plan 802 and re-detects the corresponding rooms according to the respective embedding sets 734 until the complete current 3D room model 732 is extracted.

[0088] Fig. 9 9 is a flow chart of an example process 900 for identifying a subset of front images 714 that is substantially similar to a first visual image 602A, the identification being based on region matching, in accordance with some embodiments. The first visual image 602A is captured in a current room 710, and the current room 710 is matched to one of a plurality of known rooms, such as a first known room 708A. In some embodiments, each of the current room and the known rooms 710 and 708 is represented by a respective embedding set, and the current room 710 is identified from the plurality of known rooms 708 (e.g., as the first known room 708A) based on the respective embedding sets 734 and 724. After room matching, the first visual image 602A is associated with the first front image subset 706A captured in the first known room 708A. Based on a coarse level of object or region matching, a similar front image subset 714 to the first front image subset 706A is determined.

[0089] For region or object matching, a first image descriptor 902 represents a first region or object 904 in the first visual image 602A and is extracted using a maximum stable extremal region (MSER) ​​recognition model or a YOLO (You Only LookOnce) object detection model, respectively. For each image in the first front image set 706A, one or more corresponding image descriptors 906 represent a region or object 908 in the front image 706A and are extracted using an MSER recognition model or a YOLO object detection model. For each front image 706A, one or more similarity levels 910 are determined between the first image descriptor 902 and the one or more image descriptors 906 of the respective front image 706A. The first front image set 706A includes a subset of similar front images 714, each of which has at least one region similar to the first region 904 of the first visual image 602A. In other words, the similar image subset 714 is identified in the previous image 706A based on a determination that each similar image 1006 has a respective image descriptor that is substantially close to the first image descriptor of the first visual image 602A. The image descriptors in the similar image subset are very close to each other and correspond to a similarity level subset that is the largest among the plurality of similarities 910 between the first image descriptor 902 and the respective image descriptors 906 of the first previous image set 706A. For example, the previous image subset 714 includes the previous image 706A having the greatest similarity in the first previous image set 706A.

[0090] Fig.101000 is a flow diagram of matching a region 904 in a first visual image 602A to a corresponding region in one of a plurality of known rooms 708 using keypoints, according to some embodiments. After room matching and coarse region matching, a similar image subset 714 is identified from a subset of first front images 706A captured in the first known room 708A. A first front image 714A is matched to the first visual image 602A and further selected from the similar image subset 714 based on a fine level of keypoint matching. In some embodiments, for each image 714 in the similar image subset 714, a corresponding region is associated with at least one image descriptor of the front image 714 and is identified as matching the first region 904 of the first visual image 602A. The keypoints 1002 of the first region 904 of the first visual image 602A are compared to the keypoints 1004 of the corresponding region 9008 of the corresponding front image in the front image subset 714. The keypoint 1004A of the corresponding region 908A of the first front image 712A matches the keypoint 1002 of the first visual image 602A better than the keypoint 1004B of the corresponding region 908B of any other front image 714B in the front image subset 714 .

[0091] For example, the similar front image subset 714 includes a first similar front image 714A and a second similar front image 714B. Region 904 of the first visual image 602A matches region 908A of the first similar front image 714A and region 908B of the second similar front image 714B. Key point 1002 of region 904 is compared with key point 1004A of region 908A and key point 1004B of region 908B. Key point 1004B of region 908B does not match key point 1002 of region 904, while key point 1006A of region 908A matches key point 1002 of region 904. Unmatched key points 1004B in region 908B are reduced, so the coincidence rate of image matching is increased.

[0092] In some embodiments, the first visual image 602A includes a plurality of regions 904, and the previous image subset 714 is determined based on the determination based on the image descriptors 902 and 906 that each region in the plurality of regions 904 matches a corresponding region 908 in each previous image 714. Additionally, in some cases, when the first visual image 602A is compared to each image in the previous image subset 714, feature point matching is performed on each of the plurality of regions 904. Alternatively, in some cases, when the first visual image 602A is compared to each image in the previous image subset 714, feature point matching is performed on a subset of the plurality of regions 904.

[0093] Region matching requires more computing resources than room matching, but less computing resources than feature point matching. Room matching helps reduce the images that match the first visual image 602A, while region or object matching further reduces the images that match the first visual image 602A to match with similar previous image subsets 714. Key point matching only needs to be applied to similar previous image subsets 714. Through these methods, region matching and key point matching do not need to be applied to too many historical images captured before the first visual image 602A (e.g., all previous images 706, a subset of the first visual image subset 702), thereby saving computing resources and speeding up image matching in SLAM.

[0094] Fig.11 1 is a flow chart of an example method 1100 for determining a camera pose associated with a visual image according to some embodiments. For convenience, the method 1100 is described as being implemented by an electronic system (e.g., an electronic system 200 including an HMD 104D, a mobile device 104C, a server 102, or a combination thereof). More specifically, the method 1100 is implemented by the multi-level mapping module 234 of the electronic system 200. Optionally, the method 1100 is controlled by instructions stored in a non-volatile computer-readable storage medium and executed by one or more processors of a computer system. Fig.11 Each operation shown in may correspond to instructions stored in a computer memory or non-transitory computer-readable storage medium (e.g., Figure 2 The computer readable storage medium may include a disk storage device, a solid-state storage device such as a flash memory, or other non-volatile storage devices. The instructions stored on the computer readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 1100 may be combined, and / or the order of some operations may be changed.

[0095] The electronic system 200 acquires (1102) a sequence of visual images 602, each image being captured in a respective room. For a first visual image 602A, the electronic system 200 identifies (1104) a current room 710 in a plurality of known rooms 708, in which the first visual image 602A is captured. Each known room 708 corresponds (1106) to a set of front images 706 that maps the known room 708, and each front image 706 is associated (1108) with a corresponding front pose of the camera 260 in the known room 708. Based on the identification of the current room 710 in the plurality of known rooms 708 (1110), the electronic system 200 identifies (1112) a first set of front images associated with the current room 710, selects (1114) a first front image 712A from the first set of front images to match the first visual image 602A, and determines (1116) a first camera pose associated with the first visual image 602A based on a first front pose associated with the first front image 712A. In some embodiments, each visual image is one of a color image and a monochrome image.

[0096] In some embodiments, the electronic system 200 acquires (1118) a sequence of depth images 720 synchronized with the sequence of visual images 700. Each visual image corresponds to a respective depth image and is associated with a corresponding camera pose. The sequence of visual images has a first subset of visual images 702 that are captured in the current room 710 and include a first visual image 602A. The electronic system 200 constructs (1120) a current 3D room model 732 of the current room 710 based on the first subset of visual images 702 and the corresponding first subset of depth images 720A.

[0097] In addition, in some embodiments, the first visual image subset 702 includes a second visual image 602B immediately following the first visual image 602A. The electronic system 200 determines a second camera pose associated with the second visual image 602B based on a second camera pose corresponding to the second front image 712B in the first front image set 706A of the current room 710. Optionally, the second visual image 602B is immediately after the first visual image 602A, or is separated from the first visual image 602A by an integer number of visual images in the first visual image subset. In some embodiments, the first visual image subset 702 includes a third visual image 602C before the first visual image 602A. The electronic system 200 determines that the current 3D room model 732 is incomplete for the first visual image subset 702 when acquiring the third visual image 602C. Based on the determination that the current 3D room model 732 is incomplete for the first visual image subset 702, the third visual image 602C is not associated with any known room 708 and any corresponding front image 706 of the known room 708. Additionally, in some embodiments, the electronic system 200 determines a third camera pose associated with the third visual image 602C based on a camera pose associated with an earlier visual image 702 in the first subset of visual images 702. In some embodiments, the first subset of visual images 702 includes an alternative visual image 716. The electronic system 200 determines an alternative camera pose associated with the alternative visual image 716 based on a camera pose associated with an earlier visual image in the first subset of visual images 702. Optionally, the alternative visual image 716 is immediately before or after the first visual image, or is separated from the first visual image 602A by an integer number of visual images in the first subset of visual images 702.

[0098] In addition, in some embodiments, each of the plurality of known rooms 708 corresponds to a respective embedding set 724. The electronic system 200 extracts (1122) a first embedding set 734 of the current 3D room model 732 of the current room 710. The first embedding set 734 includes semantic and instance information describing the current room 710. The first embedding set 734 includes a plurality of semantic elements configured to provide semantic information of a plurality of objects or regions located in the current 3D room model 732. In some embodiments, the electronic system 200 compares (1124) the first embedding set 734 with the corresponding embedding set 724 of each known room 708, and determines (1126) that the first embedding set 734 satisfies the room identification criteria 740 based on the comparison result, thereby identifying the current room 710 in the plurality of known rooms 708. For example, for each known room 708, the electronic system 200 determines a similarity level 726 between the first embedding set and the corresponding embedding set 724 of each known room 708. The electronic system 200 further determines that (1) the first similarity level 726A between the first embedding set 734 of the current room 710 and the corresponding embedding set 724 of one of the multiple known rooms 708 is greater than a similarity threshold; and (2) the first similarity level 726A is the maximum value among multiple similarity levels 726 between the first embedding set 734 and the respective embedding sets 724 corresponding to the multiple known rooms 708, thereby identifying the current room 710 as one of the multiple known rooms 708.

[0099] In some embodiments, the electronic system 200 acquires a depth image sequence 720 synchronized with the visual image sequence 700. Each visual image corresponds to a respective depth image and is associated with a respective camera pose. The visual image sequence 700 has a second visual image subset 704 captured in the next room 718. The electronic system 200 creates a next 3D room model 736 of the next room 718 based on the second visual image subset and the corresponding second depth image subset 720B. The electronic system 200 extracts a second embedding set 738 from the next 3D room model 736, and the second embedding set 738 includes semantic and instance information describing the next room 718. For each known room 708, the electronic system 200 determines a similarity level 726' between the second embedding set 738 and the corresponding embedding set 724 of the respective known room 708, and determines that (1) the second similarity level 726A' between the second embedding set 738 and the corresponding embedding set 724 of one of the plurality of known rooms 708 is less than a similarity threshold; and (2) the second similarity level 726A' is a maximum value among a plurality of similarity levels 726A' between the second embedding set 738 and the respective embedding sets 724 corresponding to the plurality of known rooms 708. The electronic system 200 determines that the next room 718 is not in the plurality of known rooms 708. Furthermore, in some embodiments, the electronic system 200 expands the plurality of known rooms 708 to include the next room 718, and the next room 718 is associated with the second embedding set 738.

[0100] In some embodiments, for each front image in the first front image set 706A, the electronic system 200 extracts a corresponding image descriptor 906 that describes an object or region 908 present in the front image 706A. By obtaining a first image descriptor 902 that describes a first region 904 in the first visual image 602A, comparing the first image descriptor 902 with respective image descriptors 906 of the first front image set 706A, and determining, based on the comparison result, that at least one image descriptor 906 of the front image subset 714 satisfies the image selection criteria, the front image subset 714 includes the first front image 712A. For example, for each image in the first front image set 706A, the electronic system 200 determines a respective similarity level 910 for the first image descriptor 902 and the at least one corresponding image descriptor 906, and a subset of the similarity levels 910 between the first image descriptor 902 and the at least one corresponding image descriptor 906 for each image in the front image subset 714 is a maximum of a plurality of similarity levels between the first image descriptor 902 and the respective image descriptors 906 of the first front image set 706A. The front image subset 714 includes the first front image 712A.

[0101] Additionally, in some embodiments, a first image descriptor 902 is extracted from the first visual image 602A using a YOLO object detection model. For each image in the first set of front images 706A, a corresponding image descriptor 906 is extracted from the YOLO object detection model. Alternatively, in some embodiments, the first image descriptor 902 is extracted from the first visual image 602A using a maximum stable extremal region (MSER) ​​recognition model, and for each image in the first set of front images 706A, a corresponding image descriptor 906 is extracted from the MSER recognition model. Additionally, in some embodiments, for each image in the front image subset 714, a region 908 associated with at least one image descriptor 906 corresponding to the front image 706A is identified, and key points of a first region 904 of the first visual image 602A are compared to key points of a region 908 of the front image in the front image subset 714. The corresponding region is matched to the first region 904 of the first visual image 602A. When the feature points 1004 of the corresponding area of ​​the first front image 712A match the feature points 1002 of the first visual image 602A better than the feature points of the corresponding area of ​​any other front image in the front image subset 714, the first front image 712A is selected.

[0102] In some embodiments, the electronic system 200 acquires a plurality of front images 706, which includes a respective set of front images for each known room 708, and creates a respective 3D room model for each known room 708. Based on the respective 3D room model for each known room 708, a current room 710 is identified in the plurality of known rooms 708.

[0103] It should be understood that Fig.11 The specific order of operations described in the description is for example only and does not represent the only order in which the operations are to be performed. A person of ordinary skill in the art will recognize a variety of methods for detecting planes, as described herein. In addition, it should be noted that the above Figure 5-10 The details of the other processes described in the above also apply in a similar manner. Fig.11 For the sake of brevity, these details are not repeated here.

[0104] Method 1100 generates a 3D room model using depth and pose information. In some embodiments, a monocular camera is used to provide image data 602, which is fed simultaneously to a SLAM system and a depth generation neural network to generate camera pose and depth. The accuracy level of the 6DOF pose captured by the monocular camera is lower than the accuracy level captured by a wide-angle fisheye camera. Alternatively, in some embodiments, a camera captures image data 602, and a separate depth sensor is used to measure depth information synchronized with the image data 602.

[0105] Various embodiments of the present application are used to obtain accurate camera postures from SLAM systems and build accurate models in indoor environments with similar scenes. In addition, room matching reduces the search time for identifying loop closure images from collected data. Moreover, data can be collected and fused from different devices, so that the reconstructed 3D room model can provide geometric information and semantic / instance information, thereby improving the success rate of loop closure detection. The reconstructed semantic 3D room model can be applied to user applications for virtual object display and semantic construction model reconstruction. In addition, a hierarchical matching strategy is applied to detect loop closure frames (e.g., the first front image 712A) from a coarse level to a fine level, and from room-level detection / matching, similar image detection in the room, region / object-level matching, to final key point-level matching. Through these methods, the success rate of closed-loop detection and the similarity rate of matching are improved.

[0106] The terms used in the various implementations described herein are only used to describe specific implementations and are not restrictive. When describing the various described embodiments and the appended claims, unless the context clearly indicates, the singular forms "said" and "the" also include plural forms. It should also be understood that the term "and / or" used herein refers to any and all possible combinations of one or more related items listed. In addition, it should also be understood that the terms "including", "having", "containing", and / or "comprising" used in this specification refer to the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. In addition, it should be understood that although the terms "first", "second" and the like may be used here to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish different elements.

[0107] As used herein, the word "if" may be interpreted as "when," "uponce," "in response to a determination," "in response to detection," or "according to a determination," depending on the context. Similarly, phrases such as "if it is determined" or "if [the condition or event] is detected" may be interpreted as "according to a determination," "in response to a determination," "in response to detection of [the condition or event]," or "according to a determination that [the condition or event] has been detected," depending on the context.

[0108] For purposes of explanation, the above description has been made with reference to specific embodiments. However, the above illustrative discussion is not exhaustive and is not intended to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments are chosen and described in order to best explain the principles of operation and practical application, thereby enabling those skilled in the art to implement.

[0109] Although the various figures illustrate several logical stages in a particular order, stages that are not related to the order may be reordered, and other stages may be combined or split. Although certain reordering or other groupings have been specifically mentioned, other groupings will be apparent to one of ordinary skill in the art, and thus the ordering and groupings described herein are not exhaustive. Furthermore, it should be appreciated that the stages may be implemented in hardware, firmware, software, or any combination thereof.

Claims

1. A method for simultaneous localization and mapping (SLAM), implemented by an electronic device, characterized in that: include: Acquire a sequence of visual images, wherein each visual image in the sequence of visual images is captured in a corresponding room; For a first visual image in the sequence of visual images, identifying a current room among a plurality of known rooms; wherein the first visual image is captured within the current room; each of the known rooms corresponds to a set of previous images, the set of previous images mapping the known rooms; each previous image of the set of previous images is associated with a corresponding previous pose in the known room; and According to the identification of the current room among the multiple known rooms: identifying a first set of previous images associated with the current room; selecting a first front image from the first set of front images to match the first visual image; and A first camera pose associated with the first visual image is determined based on a first front pose associated with the first front image.

2. The method according to claim 1, characterized in that Further including: Acquire a depth image sequence synchronized with the visual image sequence; wherein each of the visual images corresponds to a corresponding depth image in the depth image sequence and is associated with a corresponding camera pose; the visual image sequence has a first visual image subset, the first visual image subset is captured in the current room and includes the first visual image; as well as A current three-dimensional room model of the current room is constructed based on the first visual image subset and the corresponding first depth image subset.

3. The method according to claim 2, characterized in that The first visual image subset includes a second visual image immediately following the first visual image; the method further comprising: A second camera pose associated with the second visual image is determined based on a second camera pose associated with a second front image in the first set of front images.

4. The method according to claim 2, characterized in that: The first subset of visual images includes candidate visual images; the method further comprising: An alternative camera pose associated with the alternative visual image is determined based on a camera pose associated with an earlier visual image in the first subset of visual images.

5. The method according to any one of claims 2 to 4, characterized in that The first subset of visual images includes a third visual image preceding the first visual image; the method further comprising: When acquiring the third visual image, it is determined that the current three-dimensional room model is incomplete for the first visual image subset; wherein, based on the determination that the current three-dimensional room model is incomplete for the first visual image subset, the third visual image is unrelated to any of the known rooms and any corresponding previous images of the known rooms.

6. The method according to any one of claims 2 to 4, characterized in that Each of the plurality of known rooms corresponds to a respective embedding set; the method further comprising: Extracting a first embedding set of the current three-dimensional room model of the current room; wherein the first embedding set includes semantic and instance information describing the current room; the first embedding set includes multiple semantic elements, which are configured to provide semantic information of multiple objects or areas located in the current three-dimensional room model.

7. The method according to claim 6, characterized in that The method further comprises: comparing the first embedding set with a corresponding embedding set of each of the known rooms; and According to the comparison result, it is determined that the first embedding set meets the room identification criterion, thereby identifying the current room among the multiple known rooms.

8. The method according to claim 6, characterized in that The method further comprises: For each of the known rooms, determining a similarity level between the first embedding set and a corresponding embedding set of the known room; and Sure: (1) a first similarity level between the first embedding set of the current room and the corresponding embedding set of one of the plurality of known rooms is greater than a similarity threshold; and (2) The first similarity level is a maximum value among a plurality of the similarity levels between the first embedding set and the corresponding embedding sets of the plurality of known rooms, thereby identifying the current room as one of the plurality of known rooms.

9. The method according to claim 1, characterized in that: Further including: Acquire a depth image sequence synchronized with the visual image sequence; wherein each of the visual images corresponds to a corresponding depth image in the depth image sequence and is associated with a corresponding camera pose; the visual image sequence has a second visual image subset captured in a next room; creating a next three-dimensional room model of the next room based on the second visual image subset and the corresponding second depth image subset; Extracting a second embedding set from the next three-dimensional room model; wherein the second embedding set includes semantic and instance information describing the next room; For each of the known rooms, determining a similarity level between the second embedding set and a corresponding embedding set of the known room; Sure: (1) a second similarity level between the second embedding set and the corresponding embedding set of one of the plurality of known rooms is less than a similarity threshold; and (2) the second similarity level is a maximum value among a plurality of the similarity levels between the second embedding set and the corresponding embedding sets of the plurality of known rooms; and It is determined that the next room is not among the plurality of known rooms.

10. The method according to claim 9, characterized in that Further including: The plurality of known rooms are expanded to include the next room; wherein the next room is associated with the second embedding set.

11. The method according to any one of claims 1 to 10, characterized in that Further including: for each front image in the first set of front images, extracting a corresponding image descriptor describing an object or region present in the front image; The selecting the first front image from the first front image set comprises: obtaining a first image descriptor describing a first region in the first visual image; comparing the first image descriptor with each of the image descriptors of the first set of previous images; and According to the comparison result, it is determined that at least one of the image descriptors corresponding to a subset of the first set of front images satisfies an image selection criterion; wherein the subset includes the first front image.

12. The method according to any one of claims 1 to 10, characterized in that Further including: for each front image in the first set of front images, extracting a corresponding image descriptor describing an object or region present in the front image; The selecting the first front image from the first front image set comprises: obtaining a first image descriptor describing a first region in the first visual image; for each previous image in said first set of previous images, determining a level of similarity between said first image descriptor and at least one corresponding said image descriptor; and Determining a subset of the similarity level between the first image descriptor and at least one of the image descriptors corresponding to a subset of the first set of previous images is a maximum value among a plurality of the similarity levels between the first image descriptor and the respective image descriptors of the first set of previous images; wherein the subset includes the first previous image.

13. The method according to claim 11 or 12, characterized in that: The first image descriptor is extracted from the first visual image using a YOLO object detection model; for each front image in the first set of front images, a corresponding image descriptor is extracted from the YOLO object detection model.

14. The method according to claim 11 or 12, characterized in that: The first image descriptor is extracted from the first visual image using a Maximum Stable Extremal Region (MSER) ​​recognition model; for each previous image in the first set of previous images, the corresponding image descriptor is extracted from the MSER recognition model.

15. The method according to claim 13 or 14, characterized in that The selecting the first front image from the first front image set further comprises: For each previous image in the subset: identifying a region associated with at least one of the image descriptors corresponding to each previous image; wherein the region matches the first region of the first visual image; comparing keypoints of the first region of the first visual image to keypoints of the region of each previous image in the subset; and The first front image is selected; wherein the feature points of the area of ​​the first front image match the feature points of the first visual image better than the feature points of the area of ​​any other front image in the subset.

16. The method according to any one of claims 1 to 15, characterized in that The visual image sequence includes a plurality of front images, and the plurality of front images includes a set of front images corresponding to each of the known rooms; the method further includes: Before acquiring the first visual image, a respective three-dimensional room model is created for each of the known rooms; wherein the current room is identified among the plurality of known rooms based on the three-dimensional room model corresponding to each of the known rooms.

17. The method according to any one of claims 1 to 16, characterized in that Each of the visual images is one of a color image and a monochrome image.

18. An electronic device, characterized in that: include: one or more processors; as well as A memory storing instructions; wherein, when the instructions are executed by the one or more processors, the one or more processors perform the method of any one of claims 1 to 17.

19. A non-transitory computer-readable medium, characterized in that Instructions are stored; when the instructions are executed by one or more processors, the one or more processors execute the method according to any one of claims 1 to 17.

Images