Systems and methods for multi-view depth estimation using simultaneous localization and mapping

By integrating SLAM and machine learning models to process camera poses and sparse points from multiple image frames, the method addresses inefficiencies in existing depth estimation techniques, achieving enhanced accuracy and immersion in augmented reality applications.

WO2024232882A1PCT designated stage expired Publication Date: 2024-11-14INNOPEAK TECHNOLOGY INC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/US2023/021721
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing depth estimation methods for monocular moving cameras in augmented reality applications are inefficient and inaccurate due to reliance on single view frames, which fail to provide rich information for creating dense depth maps, and introduce computational burdens and noise, especially in arbitrary camera pose configurations.

Method used

The method employs simultaneous localization and mapping (SLAM) using visual-inertial odometry to determine camera poses and sparse points, which are then processed by a machine learning model to generate accurate depth maps from a temporal sequence of image frames, enhancing the accuracy and vividness of visual effects in augmented reality applications.

Benefits of technology

This approach enables fast, accurate, and efficient depth estimation by leveraging multiple image frames and sparse points, improving the accuracy and immersion of visual effects in augmented reality environments, such as blurring backgrounds and rendering virtual objects in AR environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2023021721_14112024_PF_FP_ABST
    Figure US2023021721_14112024_PF_FP_ABST
Patent Text Reader

Abstract

This application is directed to depth estimation associated with an image frame. An electronic device obtains video data including a plurality of image frames, and the plurality of image frames includes a first image frame. For each of the plurality of image frames, the electronic device determines a respective camera pose of a camera capturing the respective image frame and identifies a plurality of respective sparse points in the respective image frame. A depth map is determined for the first image frame based on the respective camera pose and the plurality of respective sparse points of each of the plurality of image frames. The depth map includes a plurality of depth elements, and each depth element corresponds to one or more respective pixels of the first image frame.
Need to check novelty before this filing date? Find Prior Art

Description

Systems and Methods for Multi- View Depth Estimation using Simultaneous Localization and MappingTECHNICAL FIELD

[0001] This application relates generally to image processing technology including, but not limited to, methods, systems, and non-transitory computer-readable media for determining depth information from image data based on related sparse points that are identified in the image data using simultaneous localization and mapping (SLAM).BACKGROUND

[0002] Depth image estimation from monocular moving cameras is necessary for augmented reality (AR) applications that need to create occlusion and collision effects for virtual objects in a physical environment and generate a dense depth map of the environment. Some solutions use single view frames and fail to obtain sufficiently rich information in the single view frames to create accurate dense depth maps. Some solutions apply polar rectification, stereo matching, and / or a pixel-based triangulation algorithm. These solutions do not work in arbitrary camera pose configurations and could introduce an extra computation burden and noise in depth estimation. It would be beneficial to have a depth estimation mechanism that determines depth information based on image data that are used in an extended reality application (e.g., an AR user application) in a fast, accurate, and efficient manner.SUMMARY

[0003] Various embodiments of this application are directed to methods, systems, devices, non-transitory computer-readable media for determining depth information from image data including a temporal sequence of image frames based on related sparse points that are identified in the image data using SLAM. The depth information is generated from the image data, camera poses, and sparse points, where the camera poses and sparse points are determined by a visual-inertial odometry (VIO) of a SLAM system. Specifically, for each image frame of the image data, the SLAM system determines an associated camera pose of a camera and identifies a set of sparse points in the image frame. The set of sparse points are applied to determine more feature points in the image frame and generate a preliminary depth representation. A machine learning model is further applied to process multiple image framesin the image data jointly based on the associated preliminary depth representations and camera poses and generate a final depth map for one of the multiple image frames of the image data.

[0004] The final depth map can be applied in different user applications to provide advanced visual effects. For example, the visual effect corresponds to a camera portrait mode in which a selfie image is captured with background blurred. A virtual object may be placed in an augmented reality (AR) environment based on the final depth map. Two-dimensional (2D) images are presented in a three-dimensional (3D) or pseudo-3D format including multiple perspectives. Because multiple images and associated sparse points are applied to determine the depth map, an accuracy level of the depth map is enhanced and the resulting visual effects become more vivid.

[0005] In one aspect, a depth estimation method is implemented at an electronic device. The method includes obtaining video data including a plurality of image frames, the plurality of image frames including a first image frame. The method further includes for each of the plurality of image frames, determining a respective camera pose of a camera capturing the respective image frame and identifying a plurality of respective sparse points in the respective image frame. The method further includes determining a depth map corresponding to the first image frame based on the respective camera pose and the plurality of respective sparse points of each of the plurality of image frames. The depth map includes a plurality of depth elements, and each depth element corresponds to one or more respective pixels of the first image frame.

[0006] In some embodiments, determining the depth map corresponding to the first image frame further includes creating a sparse point map for each of the plurality of image frames. The sparse point map has a plurality of sparse point elements and corresponds to the respective image frame. Further, in some embodiments, for the sparse point map of each of the plurality of image frames, a first subset of sparse point elements correspond to the plurality of respective sparse points in the respective image frame. Each of the first subset of sparse point elements is located at a first position in the sparse point map corresponding to a second position of a respective sparse point in the respective image frame, and has a value equal to a depth value of the respective sparse point. A second subset of sparse point elements are complementary to the first subset of sparse point elements, and in some embodiments, set to zero. In some embodiments, determining the depth map corresponding to the first image frame further includes at least one of interpolating values of a first set of sparse point elements in the second subset of sparse point elements based on values of thefirst subset of sparse point elements and applying a pooling operation to determine values of a second set of sparse point elements in the second subset of sparse point elements based on values of the first subset of sparse point elements.

[0007] In another aspect, some implementations include an electronic device that includes one or more processors and memory having instructions stored thereon, which when executed by the one or more processors cause the processors to perform any of the above methods.

[0008] In yet another aspect, some implementations include a non-transitory computer-readable medium, having instructions stored thereon, which when executed by one or more processors cause the processors to perform any of the above methods.

[0009] These illustrative embodiments and implementations are mentioned not to limit or define the disclosure, but to provide examples to aid understanding thereof.Additional embodiments are discussed in the Detailed Description, and further description is provided there.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various described implementations, reference should be made to the Detailed Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.

[0011] Figure 1 is an example data processing environment having one or more servers communicatively coupled to one or more client devices, in accordance with some embodiments.

[0012] Figure 2 is a block diagram illustrating an electronic device configured to process content data (e.g., image data), in accordance with some embodiments.

[0013] Figure 3 is an example data processing environment for training and applying a neural network-based data processing model for processing visual and / or audio data, in accordance with some embodiments.

[0014] Figure 4A is an example neural network applied to process content data in an NN-based data processing model, in accordance with some embodiments, and Figure 4B is an example node in the neural network, in accordance with some embodiments.

[0015] Figure 5 is a flowchart of a process for processing inertial sensor data and image data of an electronic system using a SLAM module, in accordance with some embodiments.

[0016] Figure 6 is a block diagram of an example depth estimation system that estimates a depth map corresponding to an image frame, in accordance with some embodiments.

[0017] Figure 7 is an example image frame including a plurality of sparse points, in accordance with some embodiments.

[0018] Figure 8 is a block diagram of an example preprocessing module for processing a sparse point map, in accordance with some embodiments.

[0019] Figures 9 and 10 are block diagrams of two example data processing modules for estimating a depth map based on image frames, camera poses, and sparse points, in accordance with some embodiments.

[0020] Figure 11 is a block diagram of an example neural network applied by a feature extractor for extracting features from a plurality of image frames, in accordance with some embodiments.

[0021] Figure 12 is a block diagram of an example neural network applied by a feature regressor for estimating a depth map associated with an image frame, in accordance with some embodiments.

[0022] Figure 13 is a flow diagram of an example depth estimation method, in accordance with some embodiments.

[0023] Like reference numerals refer to corresponding parts throughout the several views of the drawings.DETAILED DESCRIPTION

[0024] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to assist in understanding the subject matter presented herein. But it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of claims and the subject matter may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein can be implemented on many types of electronic devices with digital video capabilities.

[0025] Figure 1 is an example data processing environment 100 having one or more servers 102 communicatively coupled to one or more client devices 104, in accordance with some embodiments. The one or more client devices 104 may be, for example, desktop computers 104A, tablet computers 104B, mobile phones 104C, head-mounted display (HMD)(also called augmented reality (AR) glasses) 104D, or intelligent, multi-sensing, network- connected home devices (e.g., a surveillance camera 104E, a smart television device, a drone). Each client device 104 can collect data or user inputs, executes user applications, and present outputs on its user interface. The collected data or user inputs can be processed locally at the client device 104 and / or remotely by the server(s) 102. The one or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client devices 104, and in some embodiments, processes the data and user inputs received from the client device(s) 104 when the user applications are executed on the client devices 104. In some embodiments, the data processing environment 100 further includes a storage 106 for storing data related to the servers 102, client devices 104, and applications executed on the client devices 104.

[0026] The one or more servers 102 are configured to enable real-time data communication with the client devices 104 that are remote from each other or from the one or more servers 102. Further, in some embodiments, the one or more servers 102 are configured to implement data processing tasks that cannot be or are preferably not completed locally by the client devices 104. For example, the client devices 104 include a game console (e.g., the HMD 104D) that executes an interactive online gaming application. The game console receives a user instruction and sends it to a game server 102 with user data. The game server 102 generates a stream of video data based on the user instruction and user data and providing the stream of video data for display on the game console and other client devices that are engaged in the same game session with the game console. In another example, the client devices 104 include a networked surveillance camera 104E and a mobile phone 104C. The networked surveillance camera 104E collects video data and streams the video data to a surveillance camera server 102 in real time. While the video data is optionally pre-processed on the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion or audio events in the video data and share information of these events with the mobile phone 104C, thereby allowing a user of the mobile phone 104 to monitor the events occurring near the networked surveillance camera 104E in the real time and remotely.

[0027] The one or more servers 102, one or more client devices 104, and storage 106 are communicatively coupled to each other via one or more communication networks 108, which are the medium used to provide communications links between these devices and computers connected together within the data processing environment 100. The one or more communication networks 108 may include connections, such as wire, wireless communication links, or fiber optic cables. Examples of the one or more communicationnetworks 108 include local area networks (LAN), wide area networks (WAN) such as the Internet, or a combination thereof. The one or more communication networks 108 are, optionally, implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol. A connection to the one or more communication networks 108 may be established either directly (e.g., using 3G / 4G connectivity to a wireless carrier), or through a network interface 110 (e.g., a router, switch, gateway, hub, or an intelligent, dedicated whole-home control node), or through any combination thereof. As such, the one or more communication networks 108 can represent the Internet of a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, governmental, educational and other computer systems that route data and messages.

[0028] In some embodiments, deep learning techniques are applied in the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed at a client device 104 to identify information contained in the content data, match the content data with other data, categorize the content data, or synthesize related content data. In these deep learning techniques, data processing models are created based on one or more neural networks to process the content data. These data processing models are trained with training data before they are applied to process the content data. Subsequently to model training, the client device 104 obtains the content data (e.g., captures video data via an internal camera) and processes the content data using the data processing models locally.

[0029] In some embodiments, both model training and data processing are implemented locally at each individual client device 104. The client device 104 obtains the training data from the one or more servers 102 or storage 106 and applies the training data to train the data processing models. Alternatively, in some embodiments, both model training and data processing are implemented remotely at a server 102 (e.g., the server 102A) associated with a client device 104. The server 102A obtains the training data from itself, another server 102 or the storage 106 applies the training data to train the data processingmodels. The client device 104 obtains the content data, sends the content data to the server 102A (e.g., in an application) for data processing using the trained data processing models, receives data processing results (e.g., recognized hand gestures) from the server 102A, presents the results on a user interface (e.g., associated with the application), renders virtual objects in a field of view based on the poses, or implements some other functions based on the results. The client device 104 itself implements no or little data processing on the content data prior to sending them to the server 102 A. Additionally, in some embodiments, data processing is implemented locally at a client device 104, while model training is implemented remotely at a server 102 (e.g., the server 102B) associated with the client device 104. The server 102B obtains the training data from itself, another server 102 or the storage 106 and applies the training data to train the data processing models. The trained data processing models are optionally stored in the server 102B or storage 106. The client device 104 imports the trained data processing models from the server 102B or storage 106, processes the content data using the data processing models, and generates data processing results to be presented on a user interface or used to initiate some functions (e.g., rendering virtual objects based on device poses) locally.

[0030] In some embodiments, a pair of AR glasses 104D (also called an HMD) are communicatively coupled in the data processing environment 100. The HMD 104D includes a camera, a microphone, a speaker, one or more inertial sensors (e.g., gyroscope, accelerometer), and a display. The camera and microphone are configured to capture video and audio data from a scene of the HMD 104D, while the one or more inertial sensors are configured to capture inertial sensor data. In some situations, the camera captures hand gestures of a user wearing the HMD 104D, and recognizes the hand gestures locally and in real time using a two-stage hand gesture recognition model. In some situations, the microphone records ambient sound, including user’s voice commands. In some situations, both video or static visual data captured by the camera and the inertial sensor data measured by the one or more inertial sensors are applied to determine and predict device poses. The video, static image, audio, or inertial sensor data captured by the HMD 104D is processed by the HMD 104D, server(s) 102, or both to recognize the device poses. Optionally, deep learning techniques are applied by the server(s) 102 and HMD 104D jointly to recognize and predict the device poses. The device poses are used to control the HMD 104D itself or interact with an application (e.g., a gaming application) executed by the HMD 104D. In some embodiments, the display of the HMD 104D displays a user interface, and the recognized orpredicted device poses are used to render or interact with user selectable display items (e.g., an avatar) on the user interface.

[0031] Figure 2 is a block diagram illustrating an electronic system 200 configured to process content data (e.g., image data), in accordance with some embodiments. The electronic system 200 includes a server 102, a client device 104 (e.g., HMD 104D in Figure 1), a storage 106, or a combination thereof. The electronic system 200 , typically, includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components (sometimes called a chipset). The electronic system 200 includes one or more input devices 210 that facilitate user input, such as a keyboard, a mouse, a voice-command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capturing camera, or other input buttons or controls. Furthermore, in some embodiments, the client device 104 of the electronic system 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard. In some embodiments, the client device 104 includes one or more optical cameras 260 (e.g., an RGB camera), scanners, or photo sensor units for capturing images, for example, of graphic serial codes printed on the electronic devices. The electronic system 200 also includes one or more output devices 212 that enable presentation of user interfaces and display content, including one or more speakers and / or one or more visual displays. Optionally, the client device 104 includes a location detection device, such as a GPS (global positioning system) or other geo-location receiver, for determining the location of the client device 104.

[0032] In some embodiments, the client device 104 includes an inertial measurement unit (IMU) 280 integrating sensor data captured by multi-axes inertial sensors to provide estimation of a location and an orientation of the client device 104 in space. Examples of the one or more inertial sensors of the IMU 280 include, but are not limited to, a gyroscope 280A, an accelerometer 280B, a magnetic sensor 280C, and an inclinometer 280D. The magnetic sensor 280C is configured to sense Earth’s magnetic field so as to identify a predefined direction (e.g., a north direction).

[0033] Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. Memory 206, optionally, includes one or more storage devices remotely located from one or more processing units 202. Memory 206, oralternatively the non-volatile memory within memory 206, includes a non-transitory computer readable storage medium. In some embodiments, memory 206, or the non- transitory computer readable storage medium of memory 206, stores the following programs, modules, and data structures, or a subset or superset thereof:• Operating system 214 including procedures for handling various basic system services and for performing hardware dependent tasks;• Network communication module 216 for connecting each server 102 or client device 104 to other devices (e.g., server 102, client device 104, or storage 106) via one or more network interfaces 204 (wired or wireless) and one or more communication networks 108, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;• User interface module 218 for enabling presentation of information (e.g., a graphical user interface for application(s) 224, widgets, websites and web pages thereof, and / or games, audio and / or video content, text, etc.) at each client device 104 via one or more output devices 212 (e.g., displays, speakers, etc.);• Input processing module 220 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected input or interaction;• Web browser module 222 for navigating, requesting (e.g., via HTTP), and displaying websites and web pages thereof, including a web interface for logging into a user account associated with a client device 104 or another electronic device, controlling the client or electronic device if associated with the user account, and editing and reviewing settings and data that are associated with the user account;• One or more user applications 224 for execution by the electronic system 200 (e.g., games, social network applications, smart home applications, and / or other web or non-web based applications for controlling another electronic device and reviewing data captured by such devices), where in some embodiments, the user application(s) 224 include an extended reality application 224A for interacting with a user and providing extended reality content;• Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104;• Data processing module 228 for processing content data using data processing models 250, thereby identifying information contained in the content data, matching thecontent data with other data, categorizing the content data, or synthesizing related content data, where in some embodiments, the data processing module 228 is associated with one of the user applications 224 to process the content data in response to a user instruction received from the user application 224, and in some embodiments, the data processing module 228 includes a depth estimation module 230 for determining a depth map corresponding to a first image frame based on camera poses and sparse points of a plurality of image frames including the first image frame;• Pose determination and prediction module 232 for determining and predicting a pose of the client device 104 (e.g., HMD 104D), where in some embodiments, the pose is determined and predicted jointly by the pose determination and prediction module 232 and data processing module 228, and the module 232 further includes an SLAM module 234 for mapping a scene where a client device 104 is located and identifying a pose of the client device 104 within the scene using image and IMU sensor data;• Pose-based rendering module 236 for rendering virtual objects on top of a field of view of the camera 260 of the client device 104 or creating extended reality content using images captured by the camera 260 based on information of camera poses determined by the module 232; and• One or more databases 240 for storing at least data including one or more of: o Device settings 242 including common device settings (e.g., service tier, device model, storage capacity, processing capabilities, communication capabilities, etc.) of the one or more servers 102 or client devices 104; o User account information 244 for the one or more user applications 224, e.g., user names, security questions, account history data, user preferences, and predefined account settings; o Network parameters 246 for the one or more communication networks 108, e.g., IP address, subnet mask, default gateway, DNS server and host name; o Training data 248 for training one or more data processing models 250; o Data processing model(s) 250 for processing content data (e.g., video, image, audio, or textual data) using deep learning techniques, where the data processing model 250 includes a machine learning model for mapping a lighting condition of an environment based on one or more input images; and o Content data and results 252 that are obtained by and outputted to the client device 104, respectively, where the content data is processed by the dataprocessing module 228 locally at the client device 104 or remotely at the server 102 to provide the associated results (e.g., depth data).

[0034] Optionally, the one or more databases 240 are stored in one of the server 102, client device 104, and storage 106 of the electronic system 200 . Optionally, the one or more databases 240 are distributed in more than one of the server 102, client device 104, and storage 106 of the electronic system 200 . In some embodiments, more than one copy of the above data is stored at distinct devices, e.g., two copies of the data processing model 250 are stored at the server 102 and storage 106, respectively.

[0035] Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, modules or data structures, and thus various subsets of these modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory 206, optionally, stores a subset of the modules and data structures identified above. Furthermore, memory 206, optionally, stores additional modules and data structures not described above.

[0036] Figure 3 is another example of a data processing system 300 for training and applying a neural network based (NN-based) data processing model 250 for processing content data (e.g., video, image, audio, or textual data), in accordance with some embodiments. The data processing system 300 includes a model training module 226 for establishing the data processing model 250 and a data processing module 228 for processing the content data using the data processing model 250. In some embodiments, both of the model training module 226 and the data processing module 228 are located on a client device 104 of the data processing system 300, while a training data source 304 distinct from the client device 104 provides training data 248 to the client device 104. The training data source 304 is optionally a server 102 or storage 106. Alternatively, in some embodiments, the model training module 226 and the data processing module 228 are both located on a server 102 of the data processing system 300. The training data source 304 providing the training data 248 is optionally the server 102 itself, another server 102, or the storage 106. Additionally, in some embodiments, the model training module 226 and the data processing module 228 are separately located on a server 102 and client device 104, and the server 102 provides the trained data processing model 250 to the client device 104.

[0037] The model training module 226 includes one or more data pre-processing modules 308, a model training engine 310, and a loss control module 312. The dataprocessing model 250 is trained according to the type of content data to be processed. The training data 248 is consistent with the type of the content data, so is a data pre-processing module 308 applied to process the training data 248 consistent with the type of the content data. For example, an image pre-processing module 308A is configured to process image training data 248 to a predefined image format, e.g., extract a region of interest (ROI) in each training image, and crop each training image to a predefined image size. Alternatively, an audio pre-processing module 308B is configured to process audio training data 248 to a predefined audio format, e.g., converting each training sequence to a frequency domain using a Fourier transform. The model training engine 310 receives pre-processed training data provided by the data pre-processing modules 308, further processes the pre-processed training data using an existing data processing model 250, and generates an output from each training data item. During this course, the loss control module 312 can monitor a loss function comparing the output associated with the respective training data item and a ground truth of the respective training data item. The model training engine 310 modifies the data processing model 250 to reduce the loss function, until the loss function satisfies a loss criterion (e.g., a comparison result of the loss function is minimized or reduced below a loss threshold). The modified data processing model 250 is provided to the data processing module 228 to process the content data.

[0038] In some embodiments, the model training module 226 offers supervised learning in which the training data is entirely labelled and includes a desired output for each training data item (also called the ground truth in some situations). Conversely, in some embodiments, the model training module 226 offers unsupervised learning in which the training data are not labelled. The model training module 226 is configured to identify previously undetected patterns in the training data without pre-existing labels and with no or little human supervision. Additionally, in some embodiments, the model training module 226 offers partially supervised learning in which the training data are partially labelled.

[0039] The data processing module 228 includes a data pre-processing module 314, a model-based processing module 316, and a data post-processing module 318. The data preprocessing modules 314 pre-processes the content data based on the type of the content data. Functions of the data pre-processing modules 314 are consistent with those of the preprocessing modules 308 and covert the content data to a predefined content format that is acceptable by inputs of the model-based processing module 316. Examples of the content data include one or more of the following: video, image, audio, textual, and other types of data. For example, each image is pre-processed to extract an ROI or cropped to a predefinedimage size, and an audio clip is pre-processed to convert to a frequency domain using a Fourier transform. In some situations, the content data includes two or more types, e.g., video data and textual data. The model -based processing module 316 applies the trained data processing model 250 provided by the model training module 226 to process the pre- processed content data. The model-based processing module 316 can also monitor an error indicator to determine whether the content data has been properly processed in the data processing module 228. In some embodiments, the processed content data is further processed by the data post-processing module 318 to present the processed content data in a preferred format or to provide other related information that can be derived from the processed content data.

[0040] Figure 4A is an exemplary neural network (NN) 400 applied to process content data in an NN-based data processing model 250, in accordance with some embodiments, and Figure 4B is an example of a node 420 in the neural network (NN) 400, in accordance with some embodiments. The data processing model 250 is established based on the neural network 400. A corresponding model-based processing module 316 applies the data processing model 250 including the neural network 400 to process content data that has been converted to a predefined content format. The neural network 400 includes a collection of nodes 420 that are connected by links 412. Each node 420 receives one or more node inputs and applies a propagation function to generate a node output from the node input(s). As the node output is provided via one or more links 412 to one or more other nodes 420, a weight w associated with each link 412 is applied to the node output. Likewise, the node input(s) can be combined based on corresponding weights wi, W2, ws, and W4 according to the propagation function. For example, the propagation function is a product of a non-linear activation function and a linear weighted combination of the node input(s).

[0041] The collection of nodes 420 is organized into one or more layers in the neural network 400. Optionally, the layer(s) may include a single layer acting as both an input layer and an output layer. Optionally, the layer(s) may include an input layer 402 for receiving inputs, an output layer 406 for providing outputs, and zero or more hidden layers 404 (e.g., 404A and 404B) between the input and output layers 402 and 406. A deep neural network has more than one hidden layer 404 between the input and output layers 402 and 406. In the neural network 400, each layer is only connected with its immediately preceding and / or immediately following layer. In some embodiments, a layer 402 or 404B is a fully connected layer because each node 420 in the layer 402 or 404B is connected to every node 420 in its immediately following layer. In some embodiments, one of the hidden layers 404 includestwo or more nodes that are connected to the same node in its immediately following layer for down sampling or pooling the nodes 420 between these two layers. Particularly, max pooling uses a maximum value of the two or more nodes in the layer 404B for generating the node of the immediately following layer 406 connected to the two or more nodes.

[0042] In some embodiments, a convolutional neural network (CNN) is applied in a data processing model 250 to process content data (particularly, video and image data). The CNN employs convolution operations and belongs to a class of deep neural networks 400, i.e., a feedforward neural network that only moves data forward from the input layer 402 through the hidden layers to the output layer 406. The hidden layer(s) of the CNN can be convolutional layers convolving with multiplication or dot product. Each node in a convolutional layer receives inputs from a receptive area associated with a previous layer (e.g., five nodes), and the receptive area is smaller than the entire previous layer and may vary based on a location of the convolution layer in the convolutional neural network. Video or image data is pre-processed to a predefined video / image format corresponding to the inputs of the CNN. The pre-processed video or image data is abstracted by each layer of the CNN to a respective feature map. By these means, video and image data can be processed by the CNN for video and image recognition, classification, analysis, imprinting, or synthesis.

[0043] Alternatively and additionally, in some embodiments, a recurrent neural network (RNN) is applied in the data processing model 250 to process content data (particularly, textual and audio data). Nodes in successive layers of the RNN follow a temporal sequence, such that the RNN exhibits a temporal dynamic behavior. For example, each node 420 of the RNN has a time-varying real-valued activation. Examples of the RNN include, but are not limited to, a long short-term memory (LSTM) network, a fully recurrent network, an Elman network, a Jordan network, a Hopfield network, a bidirectional associative memory (BAM network), an echo state network, an independently RNN (IndRNN), a recursive neural network, and a neural history compressor. In some embodiments, the RNN can be used for handwriting or speech recognition. It is noted that in some embodiments, two or more types of content data are processed by the data processing module 228, and two or more types of neural networks (e.g., both CNN and RNN) are applied to process the content data jointly.

[0044] The training process is a process for calibrating all of the weights w, for each layer of the learning model using a training data set which is provided in the input layer 402. The training process typically includes two steps, forward propagation and backward propagation, which are repeated multiple times until a predefined convergence condition issatisfied. In the forward propagation, the set of weights for different layers are applied to the input data and intermediate results from the previous layers. In the backward propagation, a margin of error of the output (e.g., a loss function) is measured, and the weights are adjusted accordingly to decrease the error. The activation function is optionally linear, rectified linear unit, sigmoid, hyperbolic tangent, or of other types. In some embodiments, a network bias term b is added to the sum of the weighted outputs from the previous layer before the activation function is applied. The network bias b provides a perturbation that helps the NN 400 avoid over fitting the training data. The result of the training includes the network bias parameter b for each layer.

[0045] Figure 5 is a flowchart of a process 500 for processing inertial sensor data and image data of an electronic system (e.g., a server 102, a client device 104, or a combination of both) using a SLAM module 234, in accordance with some embodiments. The process 500 includes measurement preprocessing 502, initialization 504, local VIO 506 with relocation, and global pose graph optimization 508. In measurement preprocessing 502, an RGB camera 260 captures image data of a scene at an image frame rate (e.g., 30 FPS), and features are detected and tracked (510) from the image data. An IMU 280 measures inertial sensor data at a sampling frequency (e.g., 1000 Hz) concurrently with the RGB camera 260 capturing the image data, and the inertial sensor data are pre-integrated (512) to provide data of a variation of device poses 540. In initialization 504, the image data captured by the RGB camera 260 and the inertial sensor data measured by the IMU 280 are temporally aligned (514). A vision- only structure from motion (SfM) techniques 514 are applied (516) to couple the image data and inertial sensor data, estimate three-dimensional structures, and map the scene of the RGB camera 260.

[0046] After initialization 504 and during relocation of VIO 506, a sliding window 518 and associated states from a loop closure 520 are used to optimize (522) a VIO. When the VIO corresponds (524) to a keyframe of a smooth video transition and a corresponding loop is detected (526), features are retrieved (528) and used to generate the associated states from the loop closure 520. In global pose graph optimization 508, a multi-degree-of-freedom (multi-DOF) pose graph is optimized (530) based on the states from the loop closure 520, and a keyframe database 532 is updated with the keyframe associated with the VIO.

[0047] Additionally, the features that are detected and tracked (510) are used to monitor (534) motion of an object in the image data and estimate image-based poses 536, e.g., according to the image frame rate. In some embodiments, the inertial sensor data that are pre-integrated (512) may be propagated (538) based on the motion of the object and used toestimate inertial-based poses 540, e.g., according to a sampling frequency of the IMU 280. The image-based poses 536 and the inertial-based poses 540 are stored in the database 240 and used by the module 234 to estimate and predict poses that are used by the pose-based rendering module 236. Alternatively, in some embodiments, the SLAM module 234 receives the inertial sensor data measured by the IMU 280 and obtains image-based poses 536 to estimate and predict more poses 540 that are further used by the pose-based rendering module 238.

[0048] In SLAM, high frequency pose estimation is enabled by sensor fusion, which relies on data synchronization between imaging sensors and the IMU 280. The imaging sensors (e.g., the RGB camera 260, a light detection and ranging (LiDAR) scanner) provide image data desirable for pose estimation, and oftentimes operate at a lower frequency (e.g., 30 frames per second) and with a larger latency (e.g., 30 millisecond) than the IMU 280. Conversely, the IMU 280 can measure inertial sensor data and operate at a very high frequency (e.g., 1000 samples per second) and with a negligible latency (e.g., < 0.1 millisecond). Asynchronous time warping (ATW) is often applied in an AR system to warp an image of the virtual objects before it is sent to a display to correct for head movement and pose variation that occurs after the image is rendered. ATW algorithms reduce a latency of the rendered image, increase or maintain a frame rate, or reduce judders caused by missing image frames. In both SLAM and ATW, relevant image data and inertial sensor data are stored locally, such that they can be synchronized and used for the pose estimation / predication. In some embodiments, the image and inertial sensor data are stored in one of multiple STL containers, e.g., std::vector, std::queue, std: :list, etc., or other selfdefined containers. These containers are generally convenient for use. The image and inertial sensor data are stored in the STL containers with their timestamps, and the timestamps are used for data search, data insertion, and data organization.

[0049] Figure 6 is a block diagram of an example depth estimation system 600 that estimates a depth map 604 corresponding to an image 602, in accordance with some embodiments. The depth estimation system 600 includes a SLAM module 234 and a data processing module 228 (specifically, a depth estimation module 230) . The depth estimation system 600 obtains video data 602 including a plurality of image frames 602A, . . . , and 602N, and generates depth data 604 (e.g., depth maps 604A, 604B, . . ., and 604N) corresponding to the plurality of image frames 602 from the video data. In some embodiments, the depth estimation system 600 is implemented on a client device 104, and the video data 602 is captured locally by a camera 260 of the client device 104. Alternatively, in someembodiments, the depth estimation system 600 is implemented on a client device 104 by receiving the video data 602 from a server 102 or a distinct client device 104 via a wired or wireless communication link. For each of the plurality of image frames 602, the SLAM module 234 determines a respective camera pose 606 (e.g., 606A, . . ., 606N) of a camera 260 capturing the respective image frame 602, and identifies a plurality of respective sparse points 608 (e.g., 608A, . . ., 608N) in the respective image frame 602. The plurality of image frames 602 includes a first image frame 602A. The multi-view data processing module 228 determines a depth map 604A corresponding to the first image frame 602A based on the respective camera pose 606 and the plurality of respective sparse points 608 of each of the plurality of image frames 602 (i.e., based on camera poses 606 and sparse points 608 of the plurality of image frames 602 including the first image frame 602A).

[0050] In some embodiments, data of camera poses 606 and sparse points 608 is preprocessed before being used to generate the depth data 604 in the data processing module 228, so is the video data including the plurality of image frames 602. For example, each RGB channel of an image frame 602 is normalized based on a predetermined mean and a standard deviation value of pixel vas follows:Pixel Valuea,<:.B ” Mean^jBPreprocessed Pixel VahieR <;<8—Standard Deviation (1)

[0051] The depth map 604 A includes a plurality of depth elements 610, and each depth element 610 corresponds to one or more respective pixels 612 of the first image frame 602A. In some embodiments, the depth map 604 has the same resolution as the first image frame 602 A, and each depth element 610 corresponds to a single respective pixel 612 of the first image frame 602A. In some embodiments, the depth map 604 has a lower resolution than the first image frame 602 A, and each depth element 610 corresponds to a respective set of pixels of the first image frame 602A. For example, each depth elements 610 corresponds to a respective set of 2^2 or 3x3 pixels.

[0052] In some embodiments, the SLAM module 234 obtains a plurality of sensor data items 614 that are measured by an IMU 280 concurrently with the plurality of image frames 602. For each of the plurality of image frames 602, the respective camera pose 606 and at least a first subset of the plurality of respective sparse points 608 are determined based on the plurality of the image frames 602 and the plurality of sensor data items 614, i.e., by SLAM. In some embodiments, the plurality of sensor data items 614 are measured at a data rate (e.g., 1000 samples per second), and the plurality of image frames 602 are captured at aframe rate (e.g., 30 frames per second (FPS)). The data rate of the sensor data items 614 is greater than the frame rate of the image frames 602.

[0053] Depth data 604 is applied in user applications 224 (e.g., AR applications) that focus on creating immersive user experience. Some prior art solutions obtain depth maps using specialized hardware devices including, but not limited to, stereo cameras, Time-of- Flight (ToF) sensors, LiDAR scanners, and radar. Those hardware devices bear drawbacks of high cost, large power consumption, and bulky sizes, and cannot be integrated in mobile devices (e.g., smartphones 104C and HMDs 104D in Figure 1). Conversely, in some embodiments, depth data 604 is inferred from the plurality of sensor data items 614 that are captured by existing sensors of a client device 104 (e.g., camera 260 and IMU 280 in Figure 2). Further, in some embodiments, the depth data 604 is determined based on a single image frame 602. The single image frame 602 is two-dimensional (2D), and is optionally produced from an infinite number of distinct 3D scenes with the same projection geometry.Alternatively, in some embodiments, the depth data 604 of each image frame 602 is determined based on a plurality of image frame 602 that are captured at different camera poses 606. Particularly, in some embodiments, additional sensor data items 614 distinct from the image frames 602 are processed jointly with the plurality of image frames 602, thereby providing additional information (e.g., concerning camera poses 606 and sparse points 608 in the image frames 602) to facilitate estimation of the depth data 604 of each respective image frame 602.

[0054] Figure 7 is an example image frame 602 including a plurality of sparse points 608, in accordance with some embodiments. Each sparse point 608 corresponds to a respective feature (e.g., a corner of a window frame, an edge of an carpet) identified in the image frame 602. It is noted that in some embodiments, the image frame 602 includes more than a hundred sparse points and that only five of the sparse points 608 are labeled in Figure 7. A point cloud is a discrete set of data points that represents a 3D shape or object in space. Sparse points refer to the data points in the point cloud, and each image frame 602 includes a number of sparse points 608 (e.g., 100-200 sparse points). For each image frame 602, the sparse points 608 are selected from points that represent distinctive features (e.g., corners of objects) in the image frame 602.

[0055] In some embodiments, a SLAM module 234 (Figure 2) is applied to recover camera poses 606 from the plurality of image frames 602 and / or sensor data items 614 (e.g., measured by the IMU 280 in Figure 2). In some embodiments, the SLAM module 234 includes a depth camera or a stereo camera. Alternatively, in some embodiments, the SLAMmodule 234 includes a VIO 506 (Figure 5) that operates based on image frames 602 captured by a monocular camera 260 and sensor data items 614 measured by an IMU 280. The VIO 506 is configured to provide a position, a movement trajectory, and an orientation of an electronic device integrating the SLAM module 234 in real time. Additionally, the VIO 506 of the SLAM module 234 is configured to estimate distances of the sparse points 608 of each image frame 602 from a position of the camera 260 at the time of capturing the respective image frame 602. In some embodiments, an output of the SLAM module 234 includes a camera pose 606 in which the camera 260 captures a corresponding image frame 602, and the camera pose 606 includes a position and an orientation of the camera 260 at a time of capturing the corresponding image frame 602.

[0056] In some embodiments, a trajectory T of the camera 260 is a combination of the position and the orientation of the camera 260, and is tracked at a trajectory rate (e.g., greater than 400 Hz). For example, the trajectory TR is represented as follows:where R is a rotation matrix describing the orientation of the camera 260, and Tis a translation or position vector denoting a distance of the camera 260 relative to an origin point. In some embodiments, the camera 260 has a first trajectory TRi at a first time ti and a second trajectory TR2 at a second time L subsequent to the first time ti. The second trajectory TR2 is determined based on the first trajectory TRi using feature point matching and / or bundle adjustment. Specifically, in some embodiments, the camera 260 captures two image frames 602 at the first time ti and the second time L, and the two image frames 602 have two sets of sparse points 608. If camera perspectives represented by the first and second trajectories TRi and TR2 are different from each other, two sets of sparse points 608 are different, and at least a subset of sparse point 606 exists in only one of the two sets of sparse points 608. In some situations, the two sets of sparse points 608 are compared to determine a relative difference of the first and second trajectories TRi and TR2. Additionally, information of the two sets of sparse points 608 is used to refine 3D coordinates describing a scene geometry, parameters of a relative motion, optical characteristics, and camera poses 606 (e.g., positions, orientations) of the camera 260 employed to acquire the two image frames 602.

[0057] In some embodiments, the number (n) of sparse points 608 is limited in each image frame 602, where n is a positive integer (e.g., 150), and the sparse points 608 are sampled with a minimum distance from each other in order to increase system execution efficiency. For each sparse point 608, the SLAM module 234 is configured to determine adistance to the camera 260. Data of the sparse points 608 in each image frame 602 is organized according to a data structure as follows:where u and v are pixel coordinates of a respective sparse point i in each image frame 602 and z is a depth value of the sparse point i. That said, u and v define a pixel position of a respective sparse point i in each image frame 602, and z defines a depth value measured from the camera 260. The data of the sparse points 608 in each image frame 602 is transferred to the sparse point map 702 where only pixels at [z / / , v;], corresponding to the sparse points 608, have non-zero values and i is equal to 1 to n. An example sparse point map 702 corresponds to n feature points and is represented as follows:

[0058] For each image frame 602, data of the plurality of sparse points 608 form a sparse point map 702, and the sparse point map 702 further includes a plurality of sparse point elements 704 (e.g., 704A, 704B). Each sparse point element 704 corresponds to one or more respective pixels of the image frame 602. In some embodiments, the sparse point map 702 has the same resolution as the image frame 602, and each sparse point element 704 corresponds to a single respective pixel of the image frame 602. In some embodiments, the sparse point map 702 has a lower resolution than the image frame 602, and each sparse point element 704 corresponds to a respective set of pixels of the image frame 602. For example, each sparse point elements 704 corresponds to a respective set of 2^2 or 3x3 pixels.

[0059] In some embodiments, the sparse point map 702 includes a first subset of sparse point elements 704A and a second subset of sparse point elements 704B. The first subset of sparse point elements 704A correspond to the plurality of respective sparse points 608, which is identified by the SLAM module 234 in the respective image frame 602. Each of the first subset of sparse point elements 704A is located at a first position in the sparse point map 702 corresponding to a second position of a respective sparse point in the respective image frame 602, and has a value equal to a depth value of the respective sparse point 608 provided by the SLAM module 234. The second subset of sparse point elements 704B arecomplementary to the first subset of sparse point elements 704A. In some embodiments, each of the second subset of sparse point elements 704B has a depth value set to zero.

[0060] In some embodiments, the second subset of sparse point elements 704B includes a first set of sparse point elements 704B-1 that are generated from the first subset of sparse point elements 704A. A depth value of each of the first set of sparse point elements 704B-1 is interpolated from the depth values of one or more adjacent sparse point elements (e.g., 704A-1, 704A-2, 704A-3) in the first subset of sparse point elements 704A. The sparse point map 702 is updated to a sparse point map 702’ including the depth value of the sparse point element 704B-1. Alternatively, in some embodiments, a pooling layer 802 and a CNN layer 804 (Figure 8) are applied to determine a preprocessed sparse point map 702’ based on the sparse point map 702, and the preprocessed sparse point map 702’ is provided to the data processing module 228 for further processing. Sparse point elements of the preprocessed sparse point map 702’ has more non-zero values than the sparse point elements 704 of the preprocessed sparse point map 702’. For example, one of the second subset of sparse point elements 704B-2 is determined in the preprocessed sparse point map 702’ after the pooling layer 802 and CNN layer 804 are used to process the sparse point map 702. By these means, a resolution of the sparse point map 702 is enhanced by sparse point interpolation or using a neural network in a preprocessing module.

[0061] Figure 8 is a block diagram of an example preprocessing module 800 for processing a sparse point map 702, in accordance with some embodiments. Data of camera poses 606, data of sparse points 608 (e.g., the sparse point map 702), and the video data including the plurality of image frames 602 are preprocessed, before being further processed to generate the depth data 604 in the depth estimation module 230. In some embodiments, the preprocessing module 800 is coupled between the SLAM module 234 and the data processing module 228 (specifically, depth estimation module 230), and configured to receive the data of the sparse points 608 (e.g., the sparse point map 702) and generate preprocessed data of the sparse point 608’ (e.g., the preprocessed sparse point map 702’).

[0062] In some embodiments, the preprocessing module 800 includes one or more pooling layers 802 and one or more CNN blocks 804 coupled to the pooling layers 802. The sparse point map 702 includes the second subset of sparse point elements that have unfilled values (e.g., zeros). The one or more pooling layers 802 are configured to extract the most significant value from each image patch of the sparse point map 702. Examples of the one or more pooling layers 802 include, but are not limited to a max pooling layer, an average pooling, and a min pooling layer. The one or more CNN layers 804 are configured to extractfeatures from the pooling layer output and generate the preprocessed data for the sparse points 608’ (e.g., a preprocessed sparse point map 702A’ and 702B' in Figure 9).

[0063] Alternatively, in some embodiments, 2D interpolation, sparse convolution, or 3D convolution is applied to preprocess the data of sparse points 608 (e.g., the sparse point map 702) and generate the preprocessed data of sparse points 608’ (e.g., the sparse point map 702’). More details on 2D interpolation has been explained above with reference to Figure 7. In some embodiments, the preprocessing module 800 is configured to implement sparse convolution by receiving input data in the form of a sparse input tensor 608 and a convolution kernel, which is a fixed-size array of weights. The preprocessing module 800 applies the convolution kernel to the input tensor including one or more non-zero elements and results in an output tensor 608’. Alternatively, in some embodiments, the preprocessing module 800 is configured to implement 3D convolution based on a 3D kernel. The 3D kernel is moved across a 3D input volume 608 to produce a 3D output volume 608’. In other words, 3D convolution is a technique for processing three-dimensional data using a CNN. A 3D convolution layer of the CNN includes a set of filters, and each filter is a 3D tensor with a spatial extent (e.g., 3x3x3). During training, the filters are learned by adjusting their weights so that they can recognize features in the input volume 608. When the CNN is applied to a new 3D input volume 608, the filters are convolved with the input volume 608, producing a new 3D volume of activations.

[0064] Figures 9 and 10 are block diagrams of two example data processing modules 228 (specifically, depth estimation module 230 in Figure 2) for estimating a depth map 604 based on image frames 602, camera poses 606, and sparse points 608, in accordance with some embodiments. A plurality of image frames 602 (e.g., two image frames 602A and 602B) and associated data of camera poses 606 and sparse points 608 are used to estimate the depth map 604 corresponding to one of the plurality of image frames 602. Referring to both Figures 9 and 10, the data processing module 228 receives a first image frame 602A captured at a first time ti, first data of sparse points 608A associated with the first image frame 602A, and a first camera pose 606A in which a camera 260 is held at the first time ti. The first data of sparse points 608A includes a first sparse point map 702A. The data processing module 228 also receives a second image frame 602B captured at a second time Z2, second data of sparse points 608B associated with the first image frame 602B, and a second camera pose 606B in which the camera 260 is held at the second time t2. The second data of sparse points 608B includes a second sparse point map 702B. The second time t2 is optionally earlier than or subsequent to the first time ti. In some embodiments, the second image frame 602B iscaptured by the camera 260 immediately after the first image frame 602A. Based on all of the image frames 602A and 602B, sparse points 608A and 608B, and camera poses 606A and 606B, the data processing module 228 generates a first depth map 604A corresponding to the first image frame 602A.

[0065] In some embodiments, the data of sparse points 608A and 608B is preprocessed to generate preprocessed data of sparse points 608 A’ and 608B’ before being used to generate the depth data 604 in the data processing module 228, so is the video data including the plurality of image frames 602. For example, each RGB channel of the image frames 602A and 602B is normalized, e.g., based on a predetermined mean and a standard deviation value of pixel values in the respective channel. In some embodiments (Figure 8), a preprocessing module 800 includes one or more pooling layers 802 and one or more CNN blocks 804 coupled to the pooling layers 802. The preprocessing module 800 generates a processed first sparse point map 702A’ from the first sparse point map 702A, and a second processed sparse point map 702B’ from the second sparse point map 702B. Alternatively, in some embodiments, 2D interpolation, sparse convolution, or 3D convolution is applied to preprocess the data of sparse points 608 (e.g., the sparse point map 702).

[0066] The data processing module 228 includes a feature extractor 902 and a feature regressor 904. Referring to Figure 9, in some embodiments, the data of the sparse points 608 is used by the feature extractor 902 to extract features 906. The feature extractor 902 extracts a first feature 906A corresponding to the first time ti based on the first image frame 602A and the first preprocessed data of sparse points 608A’ (e.g., the preprocessed sparse data map 702 A’), and a second feature 906B corresponding to the second time C based on the second image frame 602B and the second preprocessed data of sparse points 608B’ (e.g., the preprocessed sparse data map 702B’). In an example, the feature extractor 902 includes one or more CNN blocks and a feature pyramid. The first camera pose 606A and second camera pose 606B are combined to determine a pose difference 908. The second feature 906B corresponding to the second time C is converted to a warped feature 910 based on the pose difference 908. The first feature 906 A and the warped feature 910 are fused to generate a fused feature 912. The feature regressor 904 receives the fused feature 912 and generates an output depth map 604 (e.g., a first depth map 604 A corresponding to the first image frame 602A).

[0067] Alternatively, in some embodiments not shown, the first feature 906A is warped to a second embedding space associated with the second image frame 602B based on the pose difference 908, thereby generating a warped feature 910 associated with the firstimage frame 602A. The warped feature 910 is fused with the second feature 906B to generate the fused feature 912. The feature regressor 904 receives the fused feature 912 and generates the depth map 604 (e.g., a second depth map corresponding to the first image frame 602B).

[0068] Referring to Figure 10, in some embodiments, the data of the sparse points 608 is used by the feature regressor 904 to generate the depth map 604. The feature extractor 902 extracts a first feature 906A corresponding to the first time ti and a second feature 906B corresponding to the second time t2 from the first image frame 602A and the second image frame 602B, respectively. In an example, the feature extractor 902 includes one or more CNN blocks and a feature pyramid. The first camera pose 606A and second camera pose 606B are combined to determine a pose difference 908. The second feature 906B corresponding to the second time C is converted to a warped feature 910 based on the pose difference 908. The first feature 906 A and the warped feature 910 are fused to generate a fused feature 912. The feature regressor 904 receives the fused feature 912, the first preprocessed data of sparse points 608A (e.g., the preprocessed sparse data map 702A’), and the second preprocessed data of sparse points 608B (e.g., the preprocessed sparse data map 702A’), and generates an output depth map 604 (e.g., a first depth map 604 A corresponding to the first image frame 602A).

[0069] Additionally, in some embodiments not shown, the data processing module 228 used more than two image frames 602 and associated data to generate the depth map 604 corresponding to the first image frame 602A. The more than two image frames 602 includes the first image frame 602 A and two or more second image frames 602B. Each second image frame 602B is processed by the feature extractor 902 to extract a respective second feature 906B. The respective second feature 906B is converted to a respective warped feature 910 based on a pose difference 908 between a second camera pose 606B corresponding to the respective second image frame 602B and the first camera pose 606A corresponding to the first image frame 602A. Stated another way, based on the respective pose difference 908, the respective feature 906B of each second image frame is warped into a first embedding space associated with the first image frame 602A. The first feature 906A and respective warped features 910 of the two or more second features 906B are fused to generate the fused feature 912, which is further processed by the feature regressor 904 to generate the depth map 604. The preprocessed data of sparse points 608A’ or 608B’ (e.g., the preprocessed sparse data maps 702A’ and 702B’) is used by the feature extractor 902 (Figure 9) or by the feature regressor 904 (Figure 10) to generate the depth map 604. The camera poses 606 are convertedbased on the first camera pose 606A, and the depth map 604 corresponds to the first image frame 602A.

[0070] In some embodiments, the data processing module 228 (specifically, the depth estimation module 230 in Figure 2) is configured to combine multi-view data by utilizing a homography-based warping operation (i.e., an image warping operation that is based on a homography matrix). The homography matrix H(d) is determined and applied to convert a pixel location x in a reference view Vi (e.g., corresponding to the first image frame 602A captured at a time ti) to x ’ in a source view Vi (e.g., corresponding to the second image frame 602B captured at the time ti, where i is a positive integer greater than 1). The homography matrix H(d) is represented as follows:where Ki and Ki are scaled camera intrinsics corresponding to the reference view Vi and 17, respectively, / is an identity matrix, n is the normal of a first depth plane associated with the reference view Vi, Ri and Ti represent rotation and translation of the reference view 17, respectively, and Ri and Ti represent rotation and translation of the source view 17, respectively. By these means, each feature 906z corresponding to a time ti is warped to the reference view corresponding to a time ti and generate a respective warped feature 910.Features 906 and 910 from different times (e.g., Zz, Z2, ... Z;, . . . and Zv) are fused together via a cost function (e.g., a L2 norm distance or inter product). Stated another way, a plurality of distances are determined among the plurality of respective features 906 of the plurality of image frames 602 based on the cost function. A plurality of cost functions are enumerated at different depth hypothesis to compose a fused feature 912 including a cost volume tensor, which is further processed by the feature regressor 904.

[0071] Referring to Figures 9 and 10, the data processing module 228 utilizes rough depth estimation of a set of sparse points 608 and camera poses 606 provided by a SLAM module 234. are trained efficiently Data of sparse points 608 enhances efficiency of training neural networks associated with the feature extractor 902 and feature regressor 904. More importantly, additional information (e.g., data of sparse point 608 and camera poses 606) is applied to generate the depth map 604 in addition to the image frames 602 themselves, and the feature extractor 902 and feature regressor 904 has an enhanced generalization ability, compared with existing multi-view approaches that use only image information.

[0072] As sparse points 608 identified in image frames are applied in multi-view depth estimation, feature extraction and regression neural networks applied for feature extraction and regression, image, pose, or sparse point preprocessing methods, and associated hardware configuration can be varied. For example, a SLAM module 234 optionally includes one of a mono-camera VIO module 506 applied with sensor data items 614 measured by the IMU 280, a VIO module 506 applied without sensor data items 614 measured by the IMU 280, a stereo camera system, and a RGBD camera system including a depth camera. In some embodiments, at least a subset of sparse points 608 correspond to comer features. Descriptors are applied in a mapper module of the SLAM module 234. Depth values of additional sparse points 608 (e.g., sparse point element 704B-1 in Figure 7) are estimated using triangulation algorithms and added in the corresponding sparse point map 702 to increase the number of sparse points from a VIO 506. In some embodiments, a cost volume is applied to fuse features 906 extracted from multiple image frames 602. The features 906 are warped to the same perspective via relative camera pose and homography transformation. And a cost function is used to compute a distance between two features 906. For example, an L2 norm distance is applied. In another example, the distance between two features 906 is determined using one of an LI distance, a cosine distance, and a dot-product distance. In some embodiments, a network that is distinct from the network associated with the feature extractor 902 (Figure 11) is applied to extract features 906 from the image frames 602. In some embodiments, a network distinct from the network associated with the feature regressor 904 (Figure 12) is applied to generate the depth map 604 from the fused feature 912.

[0073] Figure 11 is a block diagram of an example neural network applied by a feature extractor 902 for extracting features 906 from a plurality of image frames 602, in accordance with some embodiments. For each image frame 602, the feature extractor 902 generates a feature 906 including a plurality of feature levels 906-1, 906-2, . . ., and 906-N based on the image frame 602 and associated sparse points 608. The feature extractor 902 includes an array of CNN blocks 1102 and a feature pyramid 1104 coupled to the array of CNN blocks 1102. Data of sparse points 608 is successively processed by a first row of CNN blocks 1102A1, 1102A2, . . ., and 1102AN to generate sparse point features 1106-1, 1106-2, . . ., and 1106-N, respectively. In some embodiments, the data of sparse points 608 is preprocessed to generate preprocessed data of sparse points 608’, which is successively processed by the first row of CNN blocks 1102A1-1102AN.

[0074] The image frame 602 is successively processed by a second row of CNN blocks 1102B1, 1102B2, ..., and 1102BN to generate image features 1108-1, 1108-2, ..., and1108-N, respectively. In some embodiments, the image frame 602 is preprocessed before being successively processed by the second row of CNN blocks 1102B1-1102BN. Further, the sparse point features 1106-1, 1106-2, . . ., and 1106-N are fed into the second row of CNN blocks 1102B1, 1102B2, . . ., and 1102BN via concatenation operations. For example, the sparse point feature 1106-1 is concatenated (1110A1) with the image frame 602 (optionally preprocessed), and the sparse point feature 1106-2 is concatenated (1110A2) with the image feature 1108-1.

[0075] The feature pyramid 1104 includes a first feature pyramid block 1104-1, a second feature pyramid block 1104-2, . . ., and an N-th feature pyramid block 1104-N. The feature pyramid takes inputs from the image features 1108 associated with both camera image 602 and sparse point 608 and extracts the feature 906 including the plurality of feature levels 906-1, 906-2, . . ., and 906-N at different scales. Specifically, the image features 1108- 1, 1108-2, . . ., and 1108-N are fed into the feature pyramid blocks 1104-1, 1104-2, . . ., and 1104-N via concatenation operations. For example, the image features 1108-1 is concatenated (1110B1) with the second feature level 906-2, and the image features 1108-2 is concatenated (1110B2) with ta third feature level 906-3. Each feature level 906-z of the feature 906 has a number (Ni) of feature elements. From the first feature level 906-1 to the N-th feature level 906-N, the number (Ni) is scaled down according to a scaling factor (e.g., 4) for every two successive feature levels.

[0076] Figure 12 is a block diagram of an example neural network applied by a feature regressor 904 for estimating a depth map 604 associated with an image frame 602 (e.g., the first image frame 602A in Figures 9 and 10), in accordance with some embodiments. The feature regressor 904 includes a plurality of CNN blocks 1202-1210 arranged according to an encoder-decoder network structure. In some embodiments, the CNN blocks 1202 and 1204 successively encode an input feature 1212 and generate an encoded feature 1214. The encoded feature 1214 is processed by the CNN blocks 1206, and further processed by the CNN blocks 1206 and decoded by the CNN blocks 1208 and 1210 successively. The depth map 604 corresponding to the image frame 602 is thereby generated by the CNN blocks 1202-1210. Further, in some embodiments, the CNN blocks 1202 and 1204 provide two skip connections 1216 and 1218 to the CNN blocks 1210 and 1208, respectively.

[0077] The feature regressor 904 receives a fused feature 912 (Figures 9 and 10) and generates the depth map 604. The depth map 604 corresponds to a first image frame 602 A, and the fused feature 912 is generated based on the first image frame 602A and one or moresecond image frames 602B. In some embodiments, the one or more second image frames 602B precede the first image frame 602 in a sequence of image frames. Alternatively, in some embodiments, the one or more second image frames 602B follows the first image frame 602A in the sequence of image frames. Alternatively and additionally, in some embodiments, the one or more second image frames 602B includes a first subset of image frames following the first image frame 602A and a second subset of image frames followed by the first image frame 602A. In some embodiments, the first image frame 602A and one or more second image frames 602B form a sequence of image frames that are successively captured by a camera 260 according to a frame rate. Alternatively, in some embodiments, the first image frame 602 A is separated from a closest second image frame 602B by one or more unused image frames 602 in a sequence of image frames that are successively captured.

[0078] Stated another way, the plurality of image frames 602 used to generate the depth image 604 includes an ordered sequence of image frames having N consecutive or nonconsecutive image frames. The first image frame 602A is nM-th image frame in the ordered sequence of image frames. Aland N are positive integers, and Al is equal to or less than N. In an example, when N is equal to 3, A / is any of 1, 2, and 3. The first image frame 602 is a first, second, or third one of a sequence of 3 image frames. In another example, when A is equal to 2, Al is 1 or 2.

[0079] Each feature 906 (e.g., 906A and 906B) is extracted from a respective image frame 602, and includes a plurality of feature levels 906-1, 906-2, . . ., and 906-N. Each feature level of the feature 906A and respective feature level(s) of one or more warped features 910 are fused to generate a respective feature level of the fused feature 912. In some embodiments, the fused feature 912 includes a plurality of feature levels, and a subset or all of the plurality of feature levels are processed by the feature regressor 904 to generate the depth map 604. Further, in an example, a single one feature level (e.g., a first feature level, a second feature level) of the fused feature 912 is processed by the feature regressor 904 to generate the depth map 604. In some embodiments, a subset or all of the image frame 602 is provided to the feature regressor 904 and processed jointly with the fused feature 912 to generate the depth map 604. Stated another way, in some embodiments, the depth map 604 is generated from (1) a subset or all of the plurality of feature levels of the fused feature 912 and (2) a subset or all of the image frame 602. For example, a first feature level of the fused feature 912 and a distinct feature level (e.g., a second or N-th feature level) of the feature 906 are combined to generate the input feature 1212 that is converted to the depth map 604.

[0080] Figure 13 is a flow diagram of an example depth estimation method 1300, in accordance with some embodiments. For convenience, the method 1300 is described as being implemented by an electronic device (e.g., a mobile phone 104C, an HMD 104D). Method 1300 is, optionally, governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of the electronic device. Each of the operations shown in Figure 13 may correspond to instructions stored in a computer memory or non-transitory computer readable storage medium (e.g., memory 206 in Figure 2). The computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices. The instructions stored on the computer readable storage medium may include one or more of: source code, assembly language code, object code, or other instruction format that is interpreted by one or more processors. Some operations in method 1300 may be combined and / or the order of some operations may be changed.

[0081] The electronic device obtaining (1302) video data including a plurality of image frames 602, and the plurality of image frames 602 includes a first image frame 602A. For each of the plurality of image frames 602, the electronic device determines (1304) a respective camera pose 606 of a camera capturing the respective image frame 602 and identifies (1306) a plurality of respective sparse points 608 in the respective image frame 602. The electronic device determines (1308) a depth map 604 corresponding to the first image frame 602A based on the respective camera pose 606 and the plurality of respective sparse points 608 of each of the plurality of image frames 602. The depth map 604 includes (1310) a plurality of depth elements 610 (Figure 6), and each depth element 610 corresponds to one or more respective pixels of the first image frame 602 A.

[0082] In some embodiments, the electronic device determines the depth map 604 corresponding to the first image frame 602A by normalizing pixel values of a plurality of pixels of the respective image frame 602 for each of the plurality of image frames 602. For example, each RGB channel of the image frames 602A and 602B (Figure 6) is normalized, e.g., based on a predetermined mean and a standard deviation value

[0083] In some embodiments, the electronic device determines the depth map 604 corresponding to the first image frame 602 A by creating (1312) a sparse point map 702 for each of the plurality of image frames 602. The sparse point map 702 has a plurality of sparse point elements 704 and corresponding to the respective image frame 602. Further, in some embodiments, for the sparse point map 702 of each of the plurality of image frames 602, a first subset of sparse point elements 704A correspond (1314) to the plurality of respectivesparse points in the respective image frame 602. Each of the first subset of sparse point elements 704A is located at a first position in the sparse point map 702 corresponding to a second position of a respective sparse point in the respective image frame 602, and has a value equal to a depth value of the respective sparse point 608. A second subset of sparse point elements 704B are complementary (1316) to the first subset of sparse point elements 704A. In some embodiments, each of the second subset of sparse point elements 704B is set to zero.

[0084] Additionally, in some embodiments, the electronic device determines the depth map 604 corresponding to the first image frame 602A further includes at least one of: interpolating (1318) values of a first set of sparse point elements in the second subset of sparse point elements 704B based on values of the first subset of sparse point elements 704A; and applying (1320) a pooling operation to determine values of a second set of sparse point elements in the second subset of sparse point elements 704B based on values of the first subset of sparse point elements 704A.

[0085] In some embodiments, the electronic device determines the depth map 604 by generating a preprocessed sparse point map from which the depth map is determined by at least one of: interpolating values of a first set of sparse point elements in the second subset of sparse point elements 704B based on values of the first subset of sparse point elements 704A, applying a pooling layer 802 and a convolutional neural network (CNN) layer 804 to process the sparse point map 702, applying a sparse convolution operation on the sparse point map 702, and applying a 3D convolution operation on the sparse point map 702.

[0086] In some embodiments, the electronic device determines the depth map 604 corresponding to the first image frame 602 A by: for each of the plurality of image frames 602, extracting a respective feature 906 from the respective image frame 602 based on the plurality of respective sparse points 608, fusing a plurality of respective features 906 of the plurality of image frames 602 to generate a fused feature 912, and applying a feature regressor 904 to at least a subset of the fused feature, the respective feature of the first image frame 602A, and the first image frame 602A to generate the depth map 604 corresponding to the first image frame 602A. Further, in some embodiments, the plurality of image frames 602 include a subset of second image frames 602B that is distinct from the first image frame 602A. Prior to fusing the plurality of respective features 906 of the plurality of image frames 602, the electronic device determines a respective pose difference 908 of the respective camera pose associated with each second image frame 602B with reference to a first camera pose 902A associated with the first image frame 602A. Based on the respective posedifference 908, the electronic device warps the respective feature 906B of each second image frame 602B into a first embedding space associated with the first image frame 602A.

[0087] Additionally, in some embodiments, the feature regressor network (Figure 12) includes an encoding network and a decoding network. The encoding network includes a plurality of successive encoding stages, and the decoding network includes a plurality of successive decoding stages. In an example, the feature regressor network includes a U-net. In some embodiments, the electronic device fuses the plurality of respective features 906 of the plurality of image frames 602 by determining a plurality of distances among the plurality of respective features 906 of the plurality of image frames 602 based on a cost function.

[0088] In some embodiments, the electronic device obtains (1322) a plurality of sensor data items 614 that are measured by an inertial measurement unit (IMU) 280 concurrently with the plurality of image frames 602. For each of the plurality of image frames 602, the respective camera pose 606 and at least a first subset of the plurality of respective sparse points 608 are determined based on the plurality of the image frames 602 and the plurality of sensor data items 614.

[0089] In some embodiments, the plurality of image frames 602 includes an ordered sequence of image frames 602 having N image frames, and the first image frame is asxM-th image frame in the ordered sequence of image frames, where AT and A are positive integers, and is equal to or less than N. In an example, A is equal to 3, and is equal to 1, 2, or 3. In another example, A is equal to 2, and Mis equal to 1 or 2.

[0090] In some embodiments, for each of the plurality of image frames 602, the respective camera pose 606 includes a position and an orientation of the camera at a time of capturing the respective image frame 602.

[0091] In some embodiments, for the plurality of image frames 602, each of the plurality of respective sparse points 608 has a pixel position (w, v) in a respective image frame 602 and a depth value (z) measured from the camera 260.

[0092] It should be understood that the particular order in which the operations in Figure 13 have been described are merely exemplary and are not intended to indicate that the described order is the only order in which the operations could be performed. One of ordinary skill in the art would recognize various ways to estimate depth information associated with an image based on an associated camera pose and sparse points. Additionally, it should be noted that details of other processes described above with respect to Figures 1-12 are also applicable in an analogous manner to method 1300 described above with respect to Figure 13. For brevity, these details are not repeated here.

[0093] The terminology used in the description of the various described implementations herein is for the purpose of describing particular implementations only and is not intended to be limiting. As used in the description of the various described implementations and the appended claims, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Additionally, it will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0094] As used herein, the term “if’ is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting” or “in accordance with a determination that,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event]” or “in accordance with a determination that [a stated condition or event] is detected,” depending on the context.

[0095] The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain principles of operation and practical applications, to thereby enable others skilled in the art.

[0096] Although various drawings illustrate a number of logical stages in a particular order, stages that are not order dependent may be reordered and other stages may be combined or broken out. While some reordering or other groupings are specifically mentioned, others will be obvious to those of ordinary skill in the art, so the ordering and groupings presented herein are not an exhaustive list of alternatives. Moreover, it should be recognized that the stages can be implemented in hardware, firmware, software or any combination thereof.

Claims

What is claimed is:

1. A method for determining depth information, implemented at an electronic device, comprising: obtaining video data including a plurality of image frames, the plurality of image frames including a first image frame; for each of the plurality of image frames: determining a respective camera pose of a camera capturing the respective image frame; and identifying a plurality of respective sparse points in the respective image frame; and determining a depth map corresponding to the first image frame based on the respective camera pose and the plurality of respective sparse points of each of the plurality of image frames, the depth map including a plurality of depth elements, each depth element corresponding to one or more respective pixels of the first image frame.

2. The method of claim 1, wherein determining the depth map corresponding to the first image frame further comprises for each of the plurality of image frames, normalizing pixel values of a plurality of pixels of the respective image frame.

3. The method of claim 1 or 2, wherein determining the depth map corresponding to the first image frame further comprises: creating a sparse point map for each of the plurality of image frames, the sparse point map having a plurality of sparse point elements and corresponding to the respective image frame.

4. The method of claim 3, wherein, for the sparse point map of each of the plurality of image frames: a first subset of sparse point elements correspond to the plurality of respective sparse points in the respective image frame, wherein each of the first subset of sparse point elements is located at a first position in the sparse point map corresponding to a second position of a respective sparse point in the respective image frame, and has a value equal to a depth value of the respective sparse point; and a second subset of sparse point elements are complementary to the first subset of sparse point elements.

5. The method of claim 4, wherein determining the depth map corresponding to the first image frame further comprises generating a preprocessed sparse point map from which the depth map is determined by at least one of: interpolating values of a first set of sparse point elements in the second subset of sparse point elements based on values of the first subset of sparse point elements; applying a pooling layer and a convolutional neural network (CNN) layer to process the sparse point map ; applying a sparse convolution operation on the sparse point map; and applying a 3D convolution operation on the sparse point map;.

6. The method of any of claims 1-5, wherein determining the depth map corresponding to the first image frame further comprises: for each of the plurality of image frames, extracting a respective feature from the respective image frame based on the plurality of respective sparse points; fusing a plurality of respective features of the plurality of image frames to generate a fused feature; and applying a feature regressor network to at least a subset of the fused feature and the first image frame to generate the depth map corresponding to the first image frame.

7. The method of claim 6, wherein the plurality of image frames include a subset of second image frames that is distinct from the first image frame, and determining the depth map corresponding to the first image frame further comprises, prior to fusing the plurality of respective features of the plurality of image frames: determining a respective pose difference of the respective camera pose associated with each second image frame with reference to a first camera pose associated with the first image frame; and based on the respective pose difference, warping the respective feature of each second image frame into a first embedding space associated with the first image frame.

8. The method of claim 6 or 7, wherein the feature regressor network includes an encoding network and a decoding network, the encoding network including a plurality of successive encoding stages, the decoding network including a plurality of successive decoding stages.

9. The method of any of claims 6-8, wherein fusing the plurality of respective features of the plurality of image frames further comprises: determining a plurality of distances among the plurality of respective features of the plurality of image frames based on a cost function.

10. The method of any of claims 1-9, further comprising: obtaining a plurality of sensor data items that are measured by an inertial measurement unit (IMU) concurrently with the plurality of image frames, wherein for each of the plurality of image frames, the respective camera pose and at least a first subset of the plurality of respective sparse points are determined based on the plurality of the image frames and the plurality of sensor data items.

11. The method of any of claims 1-10, wherein the plurality of image frames includes an ordered sequence of image frames having N image frames, and the first image frame is an M- th image frame in the ordered sequence of image frames, Aland TV being positive integers, M being equal to or less than N.

12. The method of any of claims 1-11, wherein for each of the plurality of image frames, the respective camera pose includes a position and an orientation of the camera at a time of capturing the respective image frame.

13. The method of any of claims 1-12, wherein for the plurality of image frames, each of the plurality of respective sparse points has a pixel position in a respective image frame and a depth value measured from the camera.

14. An electronic device, comprising: one or more processors; and memory having instructions stored thereon, which when executed by the one or more processors cause the processors to perform a method of any of claims 1-13.

15. A non-transitory computer-readable medium, having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method of any of claims 1-13.

Citation Information

Patent Citations

  • Augmented reality display device with deep learning sensors

    US20200334461A1

  • Cross reality system for large scale environment reconstruction

    US20210279949A1

  • Efficient encoding of depth data across devices

    US20220005215A1

  • Obstacle recognition method for autonomous robots

    US20220066456A1

  • Depth maps prediction system and training method for such a system

    US20220262023A1