Method executed by electronic equipment, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a method executed by electronic equipment, the electronic equipment and a storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: obtaining a frame image in a video and IMU data corresponding to the frame image; a camera pose, a sparse map and a dense map corresponding to the frame image are obtained based on the frame image and the IMU data, and when the frame image is a key frame, obtaining of the camera pose, the sparse map and the dense map corresponding to the frame image comprises the steps that the camera pose, the three-dimensional road sign point and the IMU speed and deviation corresponding to the frame image are obtained based on the frame image and the IMU data; performing global optimization based on the camera pose, the three-dimensional road sign point, the IMU speed and the deviation corresponding to at least one key frame including the frame image, wherein the sparse map is generated by the optimized three-dimensional road sign point; and obtaining a dense map based on a global optimization result, the depth map corresponding to the frame image and the frame image through a neural network. Meanwhile, the method executed by the electronic equipment can be executed by using an artificial intelligence model.

Description

Technical Field

[0001] The present application relates to the field of positioning and map building, and in particular to a method related to positioning and map building and executed by an electronic device, an electronic device, and a storage medium. Background Art

[0002] In the field of Augmented Reality (AR) / Virtual Reality (VR), the most commonly used hardware includes image sensors and Inertial Measurement Unit (IMU) sensors. Image sensors can obtain real-world image information in real time, and IMU sensors can obtain high-frequency angular velocity and acceleration information. Based on this information, traditional Simultaneous Localization and Mapping (SLAM) technology usually extracts the feature points of the scene first, locates through data association, and establishes a sparse map, which can quickly adapt to new scenes without pre-training models.

[0003] However, in order to obtain real-time camera poses, SLAM technology can only obtain a sparse map of the scene, which cannot identify the structure and details of the scene, resulting in the inability to achieve virtual-real interaction in specific AR interaction fields. For example, placing a virtual object behind a real object cannot achieve virtual-real occlusion. Summary of the invention

[0004] In order to at least solve the above problems existing in the prior art, the present invention provides a method executed by an electronic device, an electronic device and a storage medium.

[0005] According to a first aspect of an embodiment of the present application, a method performed by an electronic device is provided, comprising: obtaining a frame image in a video and inertial measurement unit IMU data corresponding to the frame image; based on the frame image and the IMU data, obtaining a camera pose, a sparse map and a dense map corresponding to the frame image, wherein, when the frame image is a key frame, the step of obtaining the camera pose, the sparse map and the dense map corresponding to the frame image comprises: based on the frame image and the IMU data, obtaining the camera pose, three-dimensional landmark points and IMU speed and deviation corresponding to the frame image; performing global optimization based on the camera pose, three-dimensional landmark points and IMU speed and deviation corresponding to at least one key frame including the frame image, wherein the sparse map corresponding to the frame image is generated by the optimized three-dimensional landmark points; and obtaining the dense map through a neural network based on the result of the global optimization, the depth map corresponding to the frame image and the frame image.

[0006] Optionally, when the frame image is a non-key frame, the step of obtaining the camera pose, sparse map and dense map corresponding to the frame image includes: obtaining the camera pose of the frame image based on the frame image and the IMU data corresponding to the frame image; determining the sparse map and dense map corresponding to the previous key frame before the frame image as the sparse map and dense map corresponding to the frame image.

[0007] Optionally, based on the camera pose, three-dimensional landmark points, and IMU speed and deviation corresponding to at least one key frame including the frame image, the step of performing global optimization includes: constructing a reprojection error function based on a robust kernel function based on the camera pose and three-dimensional landmark points corresponding to the at least one key frame; constructing an IMU error function based on the camera pose and IMU speed and deviation corresponding to the at least one key frame; and obtaining the optimized camera pose, optimized three-dimensional landmark points, and optimized IMU speed and deviation corresponding to the at least one key frame by minimizing the global optimization objective function constructed by the reprojection error function based on the robust kernel function and the IMU error function.

[0008] Optionally, the robust kernel function includes a Huber kernel function.

[0009] Optionally, when the frame image is a key frame, the step of obtaining a camera pose, a sparse map and a dense map corresponding to the frame image further includes: determining a depth map corresponding to the frame image.

[0010] Optionally, the step of determining the depth map corresponding to the frame image includes: obtaining a binocular disparity map corresponding to the frame image by performing stereo matching on a left-eye image and a right-eye image in the frame image; and obtaining a depth map corresponding to the frame image by transforming the binocular disparity map.

[0011] Optionally, the step of obtaining a binocular disparity map corresponding to the frame image by performing stereo matching on the left eye image and the right eye image in the frame image includes: performing stereo matching based on a dense map corresponding to a previous key frame before the frame image and the left eye image and the right eye image in the frame image to obtain the binocular disparity map.

[0012] Optionally, based on the result of the global optimization, the depth map corresponding to the frame image and the frame image, the step of obtaining a dense map through a neural network includes: training the neural network based on the result of the global optimization, the depth map corresponding to the frame image and the frame image to obtain an implicit dense map representation of the scene corresponding to the binocular video; obtaining a dense map and the confidence of each three-dimensional landmark point in the dense map by inputting the optimized camera pose corresponding to the frame image into the obtained implicit dense map representation.

[0013] Optionally, the neural network is trained based on the result of the global optimization, the depth map corresponding to the frame image and the frame image, and the step of obtaining an implicit dense map representation of the scene includes: obtaining a rendered color image and a rendered depth map based on the optimized camera pose corresponding to the frame image; determining a first loss function based on the rendered color image and the color image of the frame image; determining a second loss function based on the rendered depth map and the depth map; determining a third loss function based on the depth map and the color image of the frame image; and training the neural network based on the weighted sum of the first loss function, the second loss function and the third loss function to obtain an implicit dense map representation of the scene.

[0014] Optionally, when the frame image is a key frame, the step of obtaining the camera pose, sparse map and dense map corresponding to the frame image also includes: updating the obtained sparse map based on the obtained dense map and the confidence of each three-dimensional landmark point in the dense map.

[0015] Optionally, the step of updating the obtained sparse map includes: updating each three-dimensional landmark point in the sparse map by the following operations: for a three-dimensional landmark point in the sparse map, determining a first weight of the one three-dimensional landmark point and a second weight of the three-dimensional landmark point in the dense map corresponding to the one three-dimensional landmark point based on the confidence of the three-dimensional landmark point corresponding to the one three-dimensional landmark point in the dense map; based on the determined first weight and second weight, fusing the one three-dimensional landmark point and the three-dimensional landmark point in the dense map corresponding to the one three-dimensional landmark point to obtain the updated one three-dimensional landmark point.

[0016] According to a second aspect of an embodiment of the present application, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute the method described above.

[0017] According to a third aspect of an embodiment of the present application, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the method as described above.

[0018] The beneficial effects brought about by the technical solutions provided by the embodiments of the present application will be described below in conjunction with specific optional embodiments, or can be learned from the description of the embodiments, or can be known through the implementation of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly and easily illustrate and understand the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in describing the embodiments of the present application.

[0020] Figure 1A is an overall flow chart illustrating a method performed by an electronic device according to an exemplary embodiment of the present application.

[0021] Figure 1B is a detailed flowchart illustrating a method performed by an electronic device according to an exemplary embodiment of the present application.

[0022] Figure 2 is a system diagram illustrating a method performed by an electronic device according to an exemplary embodiment of the present application.

[0023] Figure 3 is a schematic diagram showing a process of learning a neural radiation field according to an exemplary embodiment of the present application.

[0024] Figure 4 It is a schematic diagram showing the structure of an electronic device to which an embodiment of the present invention is applicable. DETAILED DESCRIPTION

[0025] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0026] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined in the appended claims and their equivalents.

[0027] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0028] The term "include" or "may include" refers to the presence of the corresponding disclosed functions, operations or components that can be used in various embodiments of the present disclosure, rather than limiting the presence of one or more additional functions, operations or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components or combinations thereof.

[0029] The term "or" used in various embodiments of the present disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, or may be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0030] Unless defined differently, all terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as understood by those skilled in the art described in the present disclosure. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally unless clearly defined in the present disclosure.

[0031] At least some functions of the device or electronic device provided in the embodiments of the present disclosure can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0032] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0033] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0034] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0035] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0036] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0037] The method provided in the present disclosure may involve one or more technical fields such as speech, language, image, video or data intelligence.

[0038] Optionally, when it comes to the field of speech or language, in a method performed by an electronic device according to the present disclosure, a speech signal as an analog signal can be received via a speech input device (e.g., a microphone), and the speech portion can be converted into computer-readable text using an automatic speech recognition (ASR) model. The user's speech intention can be obtained by interpreting the converted text using a natural language understanding (NLU) model. The ASR model or the NLU model can be an artificial intelligence model. The artificial intelligence model can be processed by an artificial intelligence dedicated processor designed in a hardware structure specified for artificial intelligence model processing. Language understanding is a technology for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0039] Optionally, when it comes to the field of images or videos, in the method performed by the electronic device according to the present disclosure, output data can be obtained by using image data as input data of an artificial intelligence model. The method of the present disclosure may be related to the field of visual understanding of artificial intelligence technology, which is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0040] Optionally, when it comes to the field of intelligent data processing, in the method performed by the electronic device according to the present disclosure, in the inference or prediction stage, an artificial intelligence model can be used to perform predictions by using real-time input data. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an input to the artificial intelligence model. Inference prediction is a technology for logical reasoning and prediction by determining information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning or recommendation.

[0041] In the present application, an artificial intelligence model can be obtained by training. Here, "obtained by training" means obtaining a predefined operating rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm. The artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values.

[0042] As described in the background technology section above, although there are currently some algorithms that can obtain camera poses and dense maps to achieve virtual and real occlusion functions, the camera poses and dense maps obtained by these algorithms have low accuracy and cannot even run in real time on the CPU or terminal.

[0043] For example, the Real-time Monocular Visual SLAM with ORB Features and NeRF-realized Mapping (Orbeez-SLAM) algorithm combines the traditional SLAM algorithm with the Neural Radiance Fields (NeRF) network to output camera poses and dense maps in real time. The algorithm uses the traditional SLAM to output real-time camera poses and sparse maps, takes these data and the corresponding RGB images as input, and uses the NeRF method to learn the multilayer perception (MLP) network structure, so that the implicit representation of the 3D scene can be learned. In this case, according to the implicit representation of the 3D scene, when a viewing angle is given, the NeRF network can output the RGB image under that viewing angle. However, the Differentiable Recurrent Optimization-Inspired Design-SLAM (Orbeez-SLAM) algorithm simply adds the transmission SLAM algorithm and the NeRF network without performing deep fusion. Specifically, the Orbeez-SLAM algorithm uses the sparse map to guide the NeRF network to learn and obtain a dense map, but the Orbeez-SLAM algorithm does not apply the dense map obtained by the NeRF network to the pose calculation in the SLAM algorithm and the sparse map reconstruction.

[0044] For another example, the NeRF-SLAM algorithm unifies the deep learning-based SLAM algorithm Droid-SLAM and the NeRF network into an end-to-end deep framework. The NeRF-SLAM algorithm uses the output of the dense depth map obtained by Dorid-SLAM to supervise the NeRF network. Specifically, according to the input monocular video stream, the NeRF-SLAM algorithm obtains the optical flow and the confidence of the optical flow through the convolution gated recurrent unit (GRU), and then constructs the Hessian matrix and decomposes it through the square root method (i.e., Cholesky) to obtain the relative pose and the confidence of the pose. Then, the above data is input into the NeRF network for training, and the NeRF network uses RGB images and depth for supervision. Although the NeRF-SLAM algorithm can output the camera pose and dense implicit map at the same time, however, since the NeRF-SLAM algorithm unifies the deep learning-based SLAM algorithm Droid-SLAM and the NeRF network into an end-to-end deep framework, it cannot run in real time on the CPU or terminal.

[0045] In order to further improve the accuracy of camera pose, implicit dense map representation, dense map, and / or sparse map obtained by traditional SLAM algorithms, some embodiments of the present application propose a concept of deeply integrating traditional SLAM algorithms, binocular depth estimation and NeRF methods.

[0046] The following describes several optional embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0047] Figure 1A is an overall flow chart illustrating a method performed by an electronic device according to an exemplary embodiment of the present application.

[0048] like Figure 1A As shown in, in step S110, a frame image in a video and inertial measurement unit IMU data corresponding to the frame image (may also be referred to as an IMU data stream, etc.) are obtained, wherein the video may be a binocular video obtained by shooting a scene using a binocular camera, etc. In the present application, the binocular video may also be referred to as a binocular video stream, a stereo video stream, a stereo video, etc. In addition, in the present application, the frame image may include a left-eye image and a right-eye image.

[0049] In step S120, based on the frame image and the IMU data, a camera pose, a sparse map, and a dense map corresponding to the frame image are obtained.

[0050] In an embodiment of the present application, the frame image can be a key frame or a non-key frame (also referred to as an ordinary frame, etc.). In an embodiment of the present application, any known key frame selection method involved in SLAM-related technologies and future key frame selection methods can be used to determine key frames and non-key frames from binocular videos. This application does not make any specific limitations on this.

[0051] The following will refer to Figure 1B A detailed description is given of a process of obtaining a camera pose, a sparse map, and a dense map corresponding to the frame image based on the frame image and IMU data corresponding to the frame image when the frame images are respectively key frames and non-key frames.

[0052] Figure 1B is a detailed flowchart illustrating a method performed by an electronic device according to an exemplary embodiment of the present application. Figure 2 is a system diagram illustrating a method performed by an electronic device according to an exemplary embodiment of the present application.

[0053] Specifically, Figure 1A and Figure 1B The method shown in Figure 2 The system shown in the figure is executed, and the system may mainly include four main modules, namely, a tracking module 210, a sparse map module 220, a binocular depth estimation module 230 and a neural dense map module 240, and these four modules may be executed by different threads.

[0054] First, in step S110, a frame image of a video and IMU data corresponding to the frame image are obtained. Figure 2 As shown in the figure, the frame images of the video and the IMU data are input Figure 2 Since step S110 has been referred to Figure 1A It has been described in detail, so it will not be repeated here.

[0055] In step S121, it is determined whether the frame image is a key frame or a non-key frame. If the frame image is a key frame, steps S122 to S124 are executed. That is, when the frame image is a key frame, Figure 1A The step S120 may include steps S122 to S124.

[0056] First, in step S122, the camera pose, three-dimensional landmark points, and IMU speed and deviation corresponding to the frame image are obtained based on the frame image and the IMU data corresponding to the frame image. In the present application, the three-dimensional landmark points corresponding to the frame image represent the three-dimensional landmark points observable from the frame image. In addition, in the present application, the three-dimensional landmark points may also be referred to as three-dimensional points (Three-Dimensional Point), etc.

[0057] like Figure 2As shown in , the frame image of the video and the IMU data corresponding to the frame image are input to the tracking module 210. The tracking module 210 may include a feature point detection and matching module, an IMU pre-integration module, a feature re-identification module, and a bundle adjustment (BA) module based on a sliding window. Specifically, the feature point detection and matching module performs feature point detection on the frame image in the input video, and then matches the feature points detected from the left eye image and the right eye image of the frame image. The IMU pre-integration module performs a pre-integration operation on the IMU data corresponding to the frame image, and accordingly, the IMU speed and deviation can be obtained. The feature re-identification module identifies whether the newly detected feature point is a point in the global sparse map. If the newly extracted feature point is a point in the sparse map, it is associated with the sparse map. If not, the association operation with the sparse map is not performed. This operation can effectively improve the accuracy of the algorithm. In this application, the sparse map is a globally consistent map, which can be optimized and / or updated in the sparse map module 220 described later. Thereafter, the sliding window-based BA module performs BA optimization on the continuous video frames, three-dimensional landmark points and IMU data within the time domain, thereby obtaining a camera pose consistent with the time domain, wherein these continuous video frames may include key frames and / or non-key frames, and the three-dimensional landmark points may include landmark points in a sparse map and three-dimensional landmark points obtained by triangulating the frame image. The tracking module 210 performs operations on each frame in the binocular video according to the above process to obtain the camera pose of each frame, so that the camera pose of the frame frequency can be obtained for the binocular video. In this case, according to the output result of the IMU pre-integration module and the camera pose of the frame frequency output by the BA module (for example, a 6-degree-of-freedom (DoF) camera pose of 30Hz), the camera pose of the IMU frequency (for example, a 6DoF camera pose of 1KHz) can be obtained. For example, the camera pose of the IMU frequency can be obtained by an interpolation method, however, the present application is not limited to this, and other methods can be used.

[0058] In addition, the tracking module 210 can also obtain (for example, simultaneously obtain) new three-dimensional landmark points (specifically, the three-dimensional coordinates of the landmark points) observed from each frame, and determine key frames based on the number of tracked feature points. For example, if the number of feature points tracked in a frame is less than a predetermined threshold (wherein the threshold is settable), the frame can be determined as a key frame, otherwise the frame can be determined as a non-key frame. That is to say, each frame in the binocular video can be determined (or selected) as a key frame and a non-key frame according to the above process.

[0059] Through the above description, the camera pose of each frame, the observable three-dimensional landmark points, and the IMU speed and deviation can be determined. The tracking module 210 described above can adopt the tracking algorithm in the existing SLAM algorithm, such as Oriented FAST and Rotated BRIEF-SLAM (Oriented FAST and Rotated BRIEF-SLAM, ORB-SLAM), Open Keyframe-based Visual-Inertial SLAM (Open Keyframe-based Visual-Inertial SLAM, OKVIS), etc. For this reason, this application does not make redundant descriptions of the relevant content.

[0060] In step S123, global optimization is performed based on the camera pose, three-dimensional landmark points, and IMU speed and deviation corresponding to at least one key frame including the frame image. Figure 2 As shown in , the global optimization module in the coefficient map module 220 can perform global optimization using the camera pose corresponding to at least one key frame including the frame image, the three-dimensional landmark points (i.e., the three-dimensional landmark points in the current sparse map), and the IMU speed and deviation, thereby obtaining the optimized three-dimensional landmark points corresponding to at least one key frame, the optimized camera pose, and the optimized IMU speed and deviation, wherein the sparse map corresponding to the currently input frame image can be generated based on the optimized three-dimensional landmark points.

[0061] Specifically, the step of performing global optimization based on the camera pose, three-dimensional landmark points, and IMU speed and deviation corresponding to at least one key frame including the frame image includes: constructing a reprojection error function based on a robust kernel function based on the camera pose and three-dimensional landmark points corresponding to the at least one key frame; constructing an IMU error function based on the camera pose and IMU speed and deviation corresponding to the at least one key frame; obtaining the optimized camera pose, optimized three-dimensional landmark points, and optimized IMU speed and deviation corresponding to the at least one key frame by minimizing the global optimization objective function constructed by the reprojection error function based on the robust kernel function and the IMU error function, wherein the sparse map corresponding to the frame image obtained in step S120 is generated based on the optimized three-dimensional landmark points.

[0062] Specifically, as shown in the following formula (1), based on the camera pose and three-dimensional landmark points corresponding to the at least one key frame, a reprojection error function based on a robust kernel function may be constructed:

[0063]

[0064] Among them, E 1 represents the reprojection error function, k m represents the number of the at least one key frame, k 1Indicates the first key frame, V i represents the set of indices of the newly created three-dimensional landmark points in the i-th key frame (for example, the set of indices of the three-dimensional landmark points that can appear in the i-th key frame), E ij represents the reprojection error of the jth 3D landmark point in the i-th key frame, T i represents the camera pose corresponding to the i-th key frame, Ts j Indicates the s j The camera pose corresponding to the key frame, X j Represents the 3D coordinates of the j-th 3D landmark point.

[0065] In an exemplary embodiment of the present application, when constructing a reprojection error function, the present application uses a robust kernel function to improve stability. The constructed reprojection error function can be based on a robust kernel function (for example, a Huber function). The specific calculation process is shown in the following formula (2). Formula (3) is the definition of the Huber function, where b is a hyperparameter, which can be set to a real number greater than 0, for example, it can be set to 1.

[0066]

[0067]

[0068] Among them, z k represents the measurement of the kth keyframe (i.e., the feature points of the kth keyframe), ||·|| ∑ represents the Mahalanobis distance function, and ∑ represents the information matrix. In the embodiment of the present application, the Huber kernel function is used to act on the reprojection errors of all feature points, thereby reducing the influence of outliers on the optimization.

[0069] In addition, an IMU error function may be constructed based on the camera pose corresponding to the at least one key frame and the IMU velocity and deviation as shown in the following formula (4):

[0070]

[0071] Among them, E 2 represents the IMU error function, represents the IMU error between the i-th key frame and the i+1-th key frame, M i and M i+1 They represent the IMU speed and deviation corresponding to the i-th key frame, and the IMU speed and deviation corresponding to the i+1-th key frame, respectively. i and T i+1 They respectively represent the camera pose corresponding to the i-th key frame and the camera pose corresponding to the i+1-th key frame.

[0072] Then, as shown in the following formula (5), the global optimization objective function is constructed by the reprojection error function based on the robust kernel function and the IMU error function:

[0073]

[0074] The optimization amount of the global optimization objective function shown in formula (5) includes the camera pose, IMU speed and deviation, and three-dimensional landmark points corresponding to at least one key frame (including the current frame image). In this case, the solution of the above formula (5) is obtained by minimizing the above global optimization objective function, thereby obtaining the optimized camera pose, optimized three-dimensional landmark points, and optimized IMU speed and deviation corresponding to at least one key frame. In addition, the sparse map corresponding to the current frame image (for example, the global sparse map at this time) can be generated based on the optimized three-dimensional landmark points.

[0075] Referring back to FIG. 1 , in step S124 , based on the result of the global optimization, the depth map corresponding to the frame image, and the frame image, a dense map is obtained through a neural network.

[0076] Before executing step S124, a depth map corresponding to the frame image may also be determined.

[0077] For example, first, stereo matching may be performed on a left-eye image and a right-eye image in a frame image to obtain a binocular disparity map corresponding to the frame image, wherein the binocular disparity map has the same resolution as that of the frame image.

[0078] For any key frame in the binocular video, when determining the depth map corresponding to the key frame, the binocular depth estimation module 230 can obtain the binocular disparity map corresponding to the key frame by directly performing stereo matching on the left eye image and the right eye image in the key frame. For example, the binocular disparity map is obtained by performing stereo matching through a binocular depth estimation network, and then the obtained binocular disparity map is transformed (to be described later) to obtain the depth map corresponding to the frame image.

[0079] However, the present application is not limited thereto, and the step of obtaining a binocular disparity map corresponding to the frame image by performing stereo matching on the left eye image and the right eye image in the frame image may include: performing stereo matching on the left eye image and the right eye image in the frame image according to a dense map corresponding to a previous key frame before the frame image, and obtaining the binocular disparity map. Specifically, Figure 2As shown in , the dense map corresponding to the previous key frame before the frame image can be fed back to the binocular depth estimation module 230, and then the binocular depth estimation module 230 can obtain the binocular disparity map corresponding to the current frame image by performing stereo matching according to the dense map corresponding to the previous key frame and the left eye image and the right eye image in the current frame image, wherein the dense map corresponding to the previous key frame is obtained by inputting the previous key frame into the implicit dense map representation obtained by the neural dense map module 240 to be described later. Then, the binocular disparity map is transformed into a depth map through the transformation operation described later, and the depth map has more accurate scene structure information. Thereafter, the depth map can be input to the neural dense map module 240, so that the dense map obtained by the neural dense map module 240 has more accurate depth information.

[0080] Specifically, the binocular disparity map can be transformed by the following equation (6) to obtain the corresponding depth map:

[0081] z=lf / d (6)

[0082] Wherein, z represents the depth, l represents the distance between the optical centers of the binocular cameras used to obtain the binocular video, d represents the disparity, and f represents the focal length of the binocular cameras.

[0083] In the present application, the execution order of step S123 and the step of obtaining a depth map corresponding to the frame image is not specifically limited, and the execution order is interchangeable.

[0084] The above describes the process of obtaining the depth map corresponding to the frame image. The following describes step S124 in detail.

[0085] Specifically, based on the result of the global optimization, the depth map corresponding to the frame image, and the frame image, the step of obtaining a dense map through a neural network may include: training the neural network based on the result of the global optimization, the depth map corresponding to the frame image, and the frame image to obtain an implicit dense map representation of the scene corresponding to the binocular video; and obtaining a dense map and the confidence of each three-dimensional landmark point in the dense map by inputting the optimized camera pose corresponding to the frame image into the implicit dense map representation.

[0086] like Figure 2As shown, for the currently input frame image, the sparse map module 220 inputs the optimized camera pose of the frame image and the frame image (i.e., the color image of the frame image) into the neural dense map module 240. Correspondingly, the binocular depth estimation module 230 inputs the depth map corresponding to the frame image into the neural dense map module 240. In this case, the neural network (hereinafter referred to as the neural radiance field NeRF) in the neural dense map module 240 can learn the scene corresponding to the binocular video online based on the received information, and use the network to implicitly represent the scene, thereby obtaining an implicit dense map representation of the scene. Figure 3 Describe in detail the process of training the Neural Radiance Field (NeRF).

[0087] Figure 3 is a schematic diagram showing a process of training a neural radiation field according to an exemplary embodiment of the present application.

[0088] like Figure 3 As shown in , first, based on the optimized camera pose corresponding to the currently input frame image, a rendered color image and a rendered depth map are obtained. Specifically, the ray sampler encodes the position information and the viewing angle information in the optimized camera pose corresponding to the frame image, thereby obtaining a hash encoding result corresponding to the position information and a direction encoding result corresponding to the viewing angle information. Thereafter, these encoding results are input into the implicit map network (i.e., the NeRF network) for rendering, thereby obtaining a rendered color image and a rendered depth map.

[0089] Thereafter, based on the rendered color image and the color image of the frame image, a first loss function is determined. Specifically, the first loss function represents the RGB loss L RGB (T, θ 1 ), which may be the L-1 norm between the rendered color image and the color image of the frame image, as shown in the following formula (7):

[0090] L RGB (T,θ 1 )=||II render (T, θ 1 )|| (7)

[0091] Among them, I represents the color image of the frame image, I render (T, θ 1 ) represents the rendered color image, that is, the camera pose T and Θ 1 The corresponding rendered color image, where Θ 1 Represents the parameters of the implicit map network (i.e., the NeRF network).

[0092] Then, based on the rendered depth map and the depth map corresponding to the frame image, a second loss function is determined. Specifically, the second loss function represents the depth loss L Depth (T, θ 1 , Θ 2 ), is the L-1 norm between the rendered depth map and the depth map of the frame image estimated by the binocular depth estimation module 230, as shown in the following formula (8):

[0093] L Depth (T, θ 1 , Θ 2 )=||D Stereo (T, θ 2 )-D render (T, θ 1 )|| (8)

[0094] Among them, D Stereo (T, θ 2 ) represents the depth map corresponding to the frame image estimated by the binocular depth estimation module 230, that is, the camera pose T and θ 2 The corresponding depth map of the frame image, where Θ 2 Denotes the parameters of the depth estimation network in the binocular depth estimation module 230. render (T, θ 1 ) represents the rendered depth map, i.e., the camera pose T and Θ 1 The corresponding rendered depth map.

[0095] Thereafter, based on the depth map corresponding to the frame image and the color image of the frame image, a third loss function is determined. Specifically, the third loss function represents the geometric transformation loss L warp (T, θ 1 , Θ 2 ), which is used to calculate the RGB difference of corresponding points after different image transformations. For example, when the pixel point q on the mth frame image is m =(u, v) is transformed to the nth frame image (where u and v represent the horizontal coordinate and vertical coordinate of the mth frame image, respectively), the pixel point q is calculated according to the depth map estimated by the binocular depth estimation module 230 and the relative pose of the two frame images. m The three-dimensional coordinates of the pixel point are then projected to the nth frame image to obtain the geometrically transformed frame I warp In this application, the third loss function can be calculated according to the following formula (9):

[0096] L warp (T, θ 1 , Θ 2 )=||I warp (T, θ 1 , D stereo(T, θ 2 ))-I(T,Θ 1 )|| (9)

[0097] Then, the neural network is trained (e.g., online learning) based on the weighted sum of the first loss function, the second loss function, and the third loss function to obtain an implicit dense map representation of the scene corresponding to the binocular video. For example, the first loss function, the second loss function, and the third loss function can be weighted summed according to the following formula (10) to obtain a total loss function L for supervising the NeRF network: total (T, θ 1 , Θ 2 ):

[0098] L total (T, θ 1 , Θ 2 )=L RGB (T, θ 1 )+λ D *L Depth (T, θ 1 , Θ 2 )+λ w *L warp (T, θ 1 , Θ 2 ) (10)

[0099] Among them, λ D and λ w is a hyperparameter.

[0100] By minimizing the above-mentioned total loss function, the network parameters of the NeRF network can be obtained, and then the implicit dense map representation of the scene corresponding to the binocular video can be obtained. In addition, when the camera pose of the current frame image is input into the implicit dense map representation, the dense map corresponding to the frame image and the confidence of each three-dimensional landmark point in the dense map can be obtained. In addition, as mentioned above, the dense map corresponding to the frame image can be fed back to the binocular depth estimation module 230, and the dense map can be combined with the left eye image and the right eye image in the next key frame to perform stereo matching, so as to obtain the binocular disparity map corresponding to the next key frame. After being converted into a depth map, the binocular disparity map is input into the neural dense map module 240, which can realize the alternating iterative update of the binocular depth estimation module 230 and the neural dense map module 240 to improve the system and model accuracy. Since this has been described above, it will not be repeated here.

[0101] At this point, the camera pose, sparse map, and dense map corresponding to the frame image when the current frame image is a key frame can be obtained.

[0102] In addition, in an exemplary embodiment of the present application, when the frame image is a key frame, step S120 may further include: updating the obtained sparse map based on the obtained dense map and the confidence of each three-dimensional landmark point in the dense map. For example, when the implicit dense map representation of the above scene is obtained after the neural network optimization converges, the dense map corresponding to the frame image and the confidence of each three-dimensional landmark point in the dense map are obtained according to the current frame image and the implicit dense map representation, the obtained sparse map corresponding to the frame image may be updated, thereby maintaining a globally consistent sparse map.

[0103] Specifically, the step of updating the sparse map may include: updating each three-dimensional landmark point in the sparse map through the following operations: for a three-dimensional landmark point in the sparse map, determining a first weight of the one three-dimensional landmark point and a second weight of the three-dimensional landmark point corresponding to the one three-dimensional landmark point in the dense map according to the confidence of the three-dimensional landmark point corresponding to the one three-dimensional landmark point in the dense map; fusing the one three-dimensional landmark point and the three-dimensional landmark point corresponding to the one three-dimensional landmark point in the dense map based on the determined first weight and second weight to obtain the updated one three-dimensional landmark point.

[0104] For example, the second weight of a 3D landmark point in a dense map corresponding to a 3D landmark point in a sparse map may be proportional to the confidence of the corresponding 3D landmark point. Specifically, when the confidence of the 3D landmark point in the dense map corresponding to the 3D landmark point is high, the second weight of the 3D landmark point in the dense map corresponding to the 3D landmark point may be determined as a larger value, and the first weight of the 3D landmark point in the sparse map may be determined as a lower value. When the confidence of the 3D landmark point in the dense map corresponding to the 3D landmark point is low, the second weight of the 3D landmark point in the dense map corresponding to the 3D landmark point may be determined as a smaller value, and the first weight of the 3D landmark point in the sparse map may be determined as a higher value. For another example, the first weight of a 3D landmark point in a sparse map and the second weight of the 3D landmark point in the dense map corresponding to the 3D landmark point may be determined based on the comparison result between the confidence and a predetermined threshold value, and this application does not specifically limit this. After determining a first weight of a 3D landmark point in a sparse map and a second weight of a 3D landmark point corresponding to the 3D landmark point in a dense map, a weighted sum is performed on a 3D landmark point in the sparse map and a 3D landmark point corresponding to the 3D landmark point in the dense map according to the determined first weight and second weight, and the 3D landmark point in the sparse map is updated to the result of the weighted sum. Thereafter, each 3D landmark point in the sparse map is updated in a similar manner. The updated sparse map can be used as the sparse map finally obtained when the frame image is a key frame.

[0105] The above describes the processing procedure when the frame image is a key frame. When the frame image is a non-key frame (ie, an ordinary frame), steps S125 and S126 may be executed. That is, when the frame image is a non-key frame, step S120 may include steps S125 and S126.

[0106] Specifically, in step S125, based on the frame image and the IMU data corresponding to the frame image, the camera pose corresponding to the frame image is obtained. Figure 2 As shown in , the tracking module 210 can perform feature point detection and matching, IMU pre-integration, feature re-identification, and sliding window-based BA optimization based on the frame image and the inertial measurement unit IMU data corresponding to the frame image, so as to obtain the camera pose corresponding to the non-key frame. Since this process is the same as step S122 performed when the frame image is a key frame, it will not be repeated here.

[0107] In step S126, the sparse map and the dense map corresponding to the previous key frame before the frame image are determined as the sparse map and the dense map corresponding to the frame image.

[0108] Specifically, in an exemplary embodiment of the present application, unlike key frames, non-key frames only perform camera pose estimation in the tracking module 210 and do not participate in the calculation of other modules. Therefore, for a frame image that is a non-key frame, the sparse map and dense map corresponding to the previous key frame before the frame image can be determined as the sparse map and dense map corresponding to the frame image. For example, when the current frame image is the 10th frame and the previous key frame before the frame image is the 8th frame, the sparse map and dense map corresponding to the 8th frame obtained by steps S122 to S124 described above can be determined as the sparse map and dense map corresponding to the frame image.

[0109] The key frames and non-key frames may be processed in the above manner until all frames in the stereoscopic video are processed.

[0110] An embodiment of the present disclosure also provides an electronic device, which includes a processor and, optionally, may also include at least one transceiver and / or at least one memory coupled to the at least one processor, wherein the at least one processor is configured to execute the steps of the method provided in any optional embodiment of the present disclosure.

[0111] Figure 4 A schematic diagram of the structure of an electronic device applicable to an embodiment of the present invention is shown in FIG. Figure 4 As shown, Figure 4 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, each of the processor 4001, the memory 4003 and the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure. Optionally, the electronic device may be a first network node, a second network node or a third network node.

[0112] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0113] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0115] The memory 4003 is used to store computer programs or executable instructions for executing the embodiments of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the above method embodiments.

[0116] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program or instructions stored thereon. When the computer program or instructions are executed by at least one processor, the steps and corresponding contents of the aforementioned method embodiment can be executed or implemented.

[0117] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0118] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that shown or described in the text.

[0119] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0120] The above text and drawings are provided only as examples to help readers understand the present disclosure. They are not intended and should not be interpreted as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present disclosure, and other similar implementation means based on the technical ideas of the present disclosure are adopted, which also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A method performed by an electronic device, comprising: Obtaining a frame image in a video and inertial measurement unit (IMU) data corresponding to the frame image; Based on the frame image and the IMU data, a camera pose, a sparse map and a dense map corresponding to the frame image are obtained, Among them, when the frame image is a key frame, the step of obtaining the camera pose, sparse map and dense map corresponding to the frame image includes: based on the frame image and the IMU data, obtaining the camera pose, three-dimensional landmark points and IMU speed and deviation corresponding to the frame image; performing global optimization based on the camera pose, three-dimensional landmark points and IMU speed and deviation corresponding to at least one key frame including the frame image, wherein the sparse map corresponding to the frame image is generated by the optimized three-dimensional landmark points; based on the result of the global optimization, the depth map corresponding to the frame image and the frame image, obtaining the dense map through a neural network.

2. The method according to claim 1, wherein: When the frame image is a non-key frame, the step of obtaining a camera pose, a sparse map, and a dense map corresponding to the frame image includes: Based on the frame image and the IMU data corresponding to the frame image, obtaining a camera pose of the frame image; The sparse map and the dense map corresponding to the previous key frame before the frame image are determined as the sparse map and the dense map corresponding to the frame image.

3. The method according to claim 1, wherein: The step of performing global optimization based on the camera pose, three-dimensional landmark points, and IMU speed and deviation corresponding to at least one key frame including the frame image comprises: Based on the camera pose and the three-dimensional landmark points corresponding to the at least one key frame, construct a reprojection error function based on a robust kernel function; Constructing an IMU error function based on the camera pose corresponding to the at least one key frame and the IMU velocity and deviation; By minimizing the global optimization objective function constructed by the reprojection error function based on the robust kernel function and the IMU error function, the optimized camera pose, optimized three-dimensional landmark points, and optimized IMU speed and deviation corresponding to the at least one key frame are obtained.

4. The method according to claim 3, wherein: The robust kernel function includes a Huber kernel function.

5. The method according to claim 1, wherein: When the frame image is a key frame, the step of obtaining a camera pose, a sparse map, and a dense map corresponding to the frame image further includes: Determine a depth map corresponding to the frame image.

6. The method according to claim 5, wherein: The step of determining the depth map corresponding to the frame image comprises: Obtaining a binocular disparity map corresponding to the frame image by performing stereo matching on a left-eye image and a right-eye image in the frame image; By transforming the binocular disparity map, a depth map corresponding to the frame image is obtained.

7. The method according to claim 6, wherein: The step of obtaining a binocular disparity map corresponding to the frame image by performing stereo matching on the left eye image and the right eye image in the frame image comprises: Stereo matching is performed according to a dense map corresponding to a previous key frame before the frame image and a left-eye image and a right-eye image in the frame image to obtain the binocular disparity map.

8. The method according to claim 1, wherein: Based on the result of the global optimization, the depth map corresponding to the frame image and the frame image, the step of obtaining a dense map through a neural network includes: Training the neural network based on the result of the global optimization, the depth map corresponding to the frame image, and the frame image to obtain an implicit dense map representation of the scene corresponding to the binocular video; By inputting the optimized camera pose corresponding to the frame image into the obtained implicit dense map representation, a dense map and the confidence of each three-dimensional landmark point in the dense map are obtained.

9. The method according to claim 8, wherein: The step of training the neural network based on the result of the global optimization, the depth map corresponding to the frame image and the frame image to obtain an implicit dense map representation of the scene includes: Based on the optimized camera pose corresponding to the frame image, obtaining a rendered color image and a rendered depth map; Determining a first loss function based on the rendered color image and the color image of the frame image; Determining a second loss function based on the rendered depth map and the depth map; Determining a third loss function based on the depth map and the color image of the frame image; Based on the weighted sum of the first loss function, the second loss function and the third loss function, the neural network is trained to obtain an implicit dense map representation of the scene.

10. The method according to claim 1, wherein: When the frame image is a key frame, the step of obtaining a camera pose, a sparse map, and a dense map corresponding to the frame image further includes: The obtained sparse map is updated based on the obtained dense map and the confidence of each three-dimensional landmark point in the dense map.

11. The method according to claim 10, wherein: The step of updating the obtained sparse map includes: performing an update on each three-dimensional landmark point in the sparse map by the following operations: For a 3D landmark point in the sparse map, determine a first weight of the 3D landmark point and a second weight of the 3D landmark point in the dense map corresponding to the 3D landmark point according to the confidence of the 3D landmark point in the dense map corresponding to the 3D landmark point; Based on the determined first weight and second weight, the one three-dimensional landmark point and the three-dimensional landmark point corresponding to the one three-dimensional landmark point in the dense map are fused to obtain the updated one three-dimensional landmark point.

12. An electronic device comprising: at least one processor; as well as at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 11.

13. A computer-readable storage medium storing instructions, wherein: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the method according to any one of claims 1 to 11.

Images