A positioning and mapping method, device, equipment and storage medium

Abstract

The application relates to the technical field of computer vision, in particular to a positioning and map construction method and device, equipment and a storage medium, which are used for providing real-time high-precision camera poses for 3D GS through visual and inertial coupling, thereby supporting online positioning and map updating based on 3D GS. The method comprises the following steps: acquiring a first image frame; determining a camera pose corresponding to the first image frame according to a camera pose change detected by an IMU between a second image frame and the first image frame and a camera pose corresponding to the second image frame, the second image frame being an image frame acquired before the first image frame; optimizing the camera pose corresponding to the first image frame by taking minimizing a joint error of a photometric error and an IMU pre-integration error corresponding to the second image frame and the first image frame as a target; and updating a 3D GS map according to the first image frame and the camera pose corresponding to the first image frame in the case that the first image frame is a key image frame.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for localization and map construction. Background Technology

[0002] Simultaneous localization and mapping (SLAM) is a core technology that enables robots, vehicles, and other devices to locate themselves in unknown environments in real time and build environmental maps using sensors. Three-dimensional (3D) Gaussian splatting (GS) is a novel explicit scene representation and real-time rendering technique that uses 3D Gaussian ellipsoids as Gaussian primitives. It can explicitly represent a scene using a large number of Gaussian primitives and render scene images with high quality. Currently, the 3D GS map building process mainly adopts an offline construction mode, which includes acquiring multiple image frames from different perspectives using a camera, processing these image frames using the structure from motion (SFM) algorithm to calculate the camera pose corresponding to each image frame, and inferring the 3D coordinates of feature points in the image frame based on the corresponding camera pose, thereby constructing Gaussian primitives in the 3D GS map.

[0003] However, in SLAM scenarios, image frames are typically acquired during motion, which leads to motion blur of feature points within the image frames. This results in inaccurate camera pose estimation based on the image frames, consequently causing inaccurate Gaussian elements constructed in the 3D GS map. Therefore, how to achieve online localization and map updating based on 3D GS is a problem that needs to be addressed. Summary of the Invention

[0004] This application provides a positioning and map building processing method, apparatus, device, and storage medium that provides real-time, high-precision camera pose for 3D GS through visual and inertial coupling, thereby supporting online positioning and map updates based on 3D GS.

[0005] In a first aspect, embodiments of this application provide a localization and map construction method, the method comprising: acquiring a first image frame; determining the camera pose corresponding to the first image frame based on the camera pose change detected by the inertial measurement unit (IMU) between the second image frame and the first image frame, and the camera pose corresponding to the second image frame, wherein the second image frame is an image frame acquired before the first image frame; optimizing the camera pose corresponding to the first image frame with the objective of minimizing the joint error of the photometric error and the IMU pre-integration error between the second image frame and the first image frame; wherein the photometric error is the error between the second image frame projected onto the camera pose corresponding to the first image frame and the photometric error of the first image frame, and the IMU pre-integration error is the error between the camera pose change between the camera pose corresponding to the second image frame and the camera pose change detected by the IMU; and updating a 3D GS map based on the first image frame and the camera pose corresponding to the first image frame if the first image frame is a key image frame.

[0006] The positioning and map building method provided in this application can be executed by an electronic device, or by a component of the electronic device (such as a processor, vehicle domain controller (VDC), or vehicle control unit (VCU), etc.), or by a software module in the electronic device, without limitation. The electronic device can be a vehicle, drone, or robot, etc.

[0007] In this application, the camera pose corresponding to the first image frame is jointly optimized by visual pixel matching (photometric error) and physical inertial constraints (IMU pre-integration error). This not only utilizes the IMU to recover the true physical scale for vision, but also uses vision to continuously correct the long-term drift of the IMU. This can provide real-time high-precision camera pose for 3D GS, and utilize the high-quality rendering of scene images by 3D GS to construct a 3D GS map. This is beneficial to improving positioning accuracy (i.e., camera pose accuracy) and map quality, thereby supporting online positioning and map updates.

[0008] In one possible implementation, before optimizing the camera pose corresponding to the first image frame with the goal of minimizing the joint error of the photometric error and the IMU pre-integration error between the second image frame and the first image frame, the method further includes: optimizing the camera pose corresponding to the first image frame with the goal of minimizing the photometric error between the second image frame and the first image frame.

[0009] Through the above implementation, before optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error, the camera pose of the first image frame can be optimized based on photometric error first. In this way, visual pixel matching (photometric error) can be used as a constraint to correct the error caused by IMU offset to the detected camera pose change. This provides an accurate initial value of camera pose for optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error. This can reduce the probability of getting stuck in a local optimum when optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error, which is beneficial to improving the efficiency and accuracy of optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error.

[0010] In one possible implementation, the first image frame is a key image frame that satisfies at least one of the following conditions: the feature point matching ratio between the first image frame and the previous key image frame is less than or equal to a first matching ratio threshold; the camera pose change of the camera pose corresponding to the first image frame relative to the camera pose corresponding to the previous key image frame is greater than or equal to a camera pose change threshold; and the difference between the acquisition time of the first image frame and the acquisition time of the previous key image frame is greater than or equal to a first duration threshold.

[0011] The above design enables electronic devices to determine whether a captured image frame is a key image frame based on factors such as the feature point matching ratio, camera pose change, and image frame acquisition duration, thus meeting the needs of different scenarios for determining whether a captured image frame is a key image frame.

[0012] In one possible implementation, before updating the 3D GS map based on the first image frame and the camera pose corresponding to the first image frame, the method further includes: determining multiple key image frames, including the first image frame; optimizing the camera pose corresponding to the multiple key image frames with the goal of minimizing the joint total error corresponding to the multiple key image frames, wherein the camera pose corresponding to the multiple key image frames includes the camera pose corresponding to the first image frame; wherein the joint total error is determined based on the photometric error and IMU pre-integration error corresponding to multiple sets of adjacent key image frames in the multiple key image frames.

[0013] By optimizing the camera pose of multiple key image frames together through the above design, the problem of the camera pose being determined based solely on the data collected by the camera and IMU from the second image frame to the first image frame can be improved. This is because the camera pose may have accumulated errors and lacks constraints from global information. This approach is beneficial for improving the accuracy of the camera pose corresponding to the first image frame.

[0014] In one possible implementation, the 3D GS map is updated based on a first image frame and the camera pose corresponding to the first image frame, including: determining the depth range information corresponding to the first image frame, the depth range information including the depth range of multiple feature points in the first image frame; projecting the 3D GS map onto the camera pose corresponding to the first image frame to obtain a first rendered depth map, the first rendered depth map including the rendered depth of the aforementioned multiple feature points; for any feature point among the multiple feature points, if the rendered depth of the feature point is outside the depth range of the feature point, adding Gaussian elements to the 3D GS map based on the position and color information of the feature point in the first image frame, the depth range of the feature point, and the camera pose corresponding to the first image frame.

[0015] The above design reduces the redundancy of 3D GS maps and improves their update efficiency.

[0016] In one possible implementation, before projecting the 3D GS map onto the camera pose corresponding to the first image frame to obtain the first rendered depth map, the method further includes: searching for the depth values ​​of multiple feature points in multiple non-key image frames adjacent to the first image frame; and updating the depth range of multiple feature points based on the depth values ​​of the multiple feature points in the multiple non-key image frames.

[0017] The above design allows depth information from non-critical image frames to be added to critical image frames, which helps improve the accuracy of 3D GS maps.

[0018] In one possible implementation, the method further includes: updating Gaussian elements in the 3D GS map using a backpropagation algorithm based on at least one of photometric loss, depth range loss, or scale consistency loss; wherein the photometric loss is determined based on a rendered color map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the first image frame; the depth range loss is determined based on a second rendered depth map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the depth range information corresponding to the first image frame; the scale consistency loss is determined based on scale factors corresponding to multiple key image frames, including the first image frame.

[0019] The above design allows for the correction of Gaussian elements in 3D GS maps based on photometric, depth constraints, and scale consistency, which helps improve the accuracy of map construction.

[0020] In one possible implementation, before acquiring the first image frame, the method further includes: initializing Gaussian elements in the 3D GS map based on the acquired first key image frame, the camera pose corresponding to the first key image frame, and the depth range information corresponding to the first key image frame.

[0021] Secondly, embodiments of this application provide a positioning and map building apparatus, including a visual-inertial module and a map generation module; the visual-inertial module is used to acquire a first image frame; and to determine the camera pose corresponding to the first image frame based on the camera pose change detected by the IMU between the second image frame and the first image frame, and the camera pose corresponding to the second image frame, wherein the second image frame is an image frame acquired before the first image frame; and to optimize the camera pose corresponding to the first image frame with the goal of minimizing the joint error of the photometric error and the IMU pre-integration error between the second image frame and the first image frame; wherein the photometric error is the error between the second image frame projected onto the camera pose corresponding to the first image frame and the photometric error of the first image frame, and the IMU pre-integration error is the error between the pose change of the camera pose corresponding to the second image frame and the camera pose change detected by the IMU; the map generation module is used to update a 3D GS map based on the first image frame and the camera pose corresponding to the first image frame when the first image frame is a key image frame.

[0022] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the processor is configured to execute a computer program or instructions stored in the memory to perform the method described in the first aspect.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions, which, when executed by a processor, can implement the method described in the first aspect.

[0024] Fifthly, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, can implement the method described in the first aspect.

[0025] In a sixth aspect, embodiments of this application also provide a chip system including a processor, which is coupled to a memory for storing programs or instructions. When the program or instructions are executed by the processor, the method described in the first aspect can be implemented.

[0026] The technical effects achievable by the second to sixth aspects mentioned above are similar to those achievable by the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram illustrating the positioning and map construction process provided in an embodiment of this application; Figure 2 This is a schematic diagram of the front-end tracing process provided in an embodiment of this application; Figure 3 This is a schematic diagram of the backend optimization process provided in the embodiments of this application; Figure 4 This is a schematic diagram of the depth filtering process provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the positioning and mapping system provided in the embodiments of this application; Figure 6 This is a schematic diagram of the visual inertial processing flow provided in the embodiments of this application; Figure 7 This is a schematic diagram of the map generation process provided in the embodiments of this application; Figure 8 This is a schematic diagram of the positioning and mapping device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In this application, "multiple" can refer to two or more. Therefore, "multiple" can also be understood as "at least two." "At least one" can be understood as one or more, such as one, two, or at least three. For example, "at least one" or "at least one item" refers to one, two, or at least three, such as including at least one of A, B, or C. This could include A, B, C, A and B, A and C, B and C, or A, B, and C. "And / or" describes the relationship between related objects. Specifically, there can be three relationships. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0031] The ordinal numbers such as "first" and "second" mentioned in this application are generally used to distinguish different objects, and are not used to limit the order, sequence, priority or importance of multiple objects.

[0032] In this application, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product, or device.

[0033] In this application, the words "exemplarily," "for example," "e.g.," are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as an "example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the term "example" is used to present concepts in a specific manner.

[0034] In this application, "greater than or equal to" and "greater than" are interchangeable. For example, "A is greater than threshold #1" and "A is greater than or equal to threshold #1" are interchangeable. "Less than or equal to" and "less than" are interchangeable. For example, "A is less than threshold #1" and "A is less than or equal to threshold #1" are interchangeable.

[0035] To facilitate understanding by those skilled in the art, some terms used in the embodiments of this application will be explained first.

[0036] 1) Inertial Measurement Unit (IMU).

[0037] An IMU (Integrated Mutual Measure) is a device used to measure the three-axis attitude angles (or angular rates) and acceleration of an object. An IMU may include an accelerometer and a gyroscope. The accelerometer measures the linear acceleration (including gravitational acceleration) of an object in the X, Y, and Z directions of a three-dimensional coordinate system. The gyroscope measures the angular velocity of an object rotating around the X, Y, and Z axes of a three-dimensional coordinate system. By integrating the angular velocities of the object rotating around the X, Y, and Z axes, the changes in the object's attitude angles (pitch angle, roll angle, and yaw angle) can be obtained.

[0038] 2) Camera.

[0039] In the embodiments of this application, a camera can refer to a device used to convert optical images into electronic signals. A camera can also be called an image sensor or a webcam, etc. The camera can be a monocular camera or a multi-view camera, etc., without limitation.

[0040] 3) Camera pose.

[0041] Camera pose describes the camera's position and orientation in three-dimensional space, i.e., "where the camera is and which way it's looking." Camera pose can be composed of information in six degrees of freedom (DoF), specifically including the following two parts: Translation corresponds to three degrees of freedom and can be represented as a three-dimensional vector (x, y, z), usually in meters (m). It means the specific position of the camera in space, that is, the three-dimensional coordinates of the camera in a reference coordinate system (such as a map coordinate system or a world coordinate system).

[0042] Rotation corresponds to three degrees of freedom and can be represented as a rotation matrix or Euler angles (i.e., pitch, roll, and yaw). It means the camera's orientation, that is, which direction the camera's coordinate axes point in the reference coordinate system (such as the map coordinate system or the world coordinate system).

[0043] The positioning and map building method provided in this application can be executed by an electronic device, or by a component of the electronic device (such as a processor, vehicle domain controller (VDC), or vehicle control unit (VCU), etc.), or by a software module in the electronic device, without limitation. The electronic device can be a vehicle, drone, or robot, etc.

[0044] Optionally, the electronic device is equipped with a camera and an IMU. The camera can be used to acquire image frames, and the IMU can be used to acquire IMU data (such as three-axis attitude angles and acceleration).

[0045] Figure 1 This application provides a schematic diagram of a localization and map construction process, which includes: S101: Acquire the first image frame.

[0046] S102: Determine the camera pose corresponding to the first image frame based on the camera pose change detected by the IMU between the second image frame and the first image frame, and the camera pose corresponding to the second image frame.

[0047] The second frame is an image frame acquired before the first frame. That is, the camera acquires the second image frame before the first image frame is acquired, or in other words, the second image frame is acquired earlier than the first image frame.

[0048] In one possible implementation, to improve the accuracy of localization and map building, the electronic device can first perform visual inertial initialization (VII) during localization and map building, calibrating the camera and IMU to address issues such as the camera's inability to directly observe the true scale and direction of gravity. This VII can also be referred to as visual inertial alignment, or camera and IMU calibration.

[0049] During visual-inertial initialization, the electronic device can use joint initialization or loosely coupled initialization methods to solve for the initialization parameters used to calibrate the camera and IMU, based on IMU data from multiple image frames acquired by the camera and multiple image frames acquired by the IMU. These initialization parameters may include at least one of the following: scale factor, gravity vector, velocity, or IMU bias.

[0050] The scale factor, also known as the global scale factor, can be used to represent the proportional relationship between a unit length in the camera coordinate system and a unit length in the real physical world (e.g., a meter).

[0051] The gravity vector can be used to represent the direction and magnitude of gravity in the world coordinate system. The accelerometer in the IMU measures the resultant vector of "carrier acceleration" and "gravitational acceleration". When estimating the camera pose, this gravity vector needs to be subtracted from the IMU measurement to separate the real motion acceleration for accurate velocity and position calculation.

[0052] Velocity can be used to represent the initial three-dimensional velocity vector of the IMU and camera carrier in the world coordinate system.

[0053] IMU bias can include accelerometer bias and gyroscope bias, representing the "systematic error" of the IMU itself, that is, the fixed offset of the IMU measurement value from the true value. Gyroscope bias affects the calculation of angle, while accelerometer bias affects the calculation of velocity and position.

[0054] After acquiring the first image frame via the camera, the electronic device can pre-integrate the IMU data collected by the IMU between the first and second image frames (i.e., the IMU data collected by the IMU between the acquisition time of the second image frame and the acquisition time of the first image frame) to obtain the pre-integration result of the IMU data. This pre-integration result represents the camera pose change detected by the IMU between the second and first image frames. The electronic device can then superimpose this pre-integration result of the IMU data (i.e., superimposed on the camera pose change detected by the IMU between the second and first image frames) onto the camera pose corresponding to the second image frame to obtain the camera pose corresponding to the first image frame.

[0055] S103: Optimize the camera pose corresponding to the first image frame with the goal of minimizing the joint error of the photometric error and the IMU pre-integration error corresponding to the second image frame and the first image frame.

[0056] Among them, the photometric error between the second image frame and the first image frame is the error between the photometric value of the second image frame projected onto the camera pose corresponding to the first image frame and the photometric value of the first image frame. Photometric value can refer to brightness or color intensity. The IMU pre-integration error between the second image frame and the first image frame is the error between the pose change of the camera pose corresponding to the second image frame and the pose change of the camera pose corresponding to the first image frame and the pose change detected by the IMU.

[0057] After the electronic device obtains the camera pose corresponding to the first image frame (such as by obtaining the camera pose corresponding to the first image frame through the above step S102), it can use the camera pose as the initial value of the camera pose corresponding to the first image frame. Based on this, with the goal of minimizing the joint error of the photometric error and the IMU pre-integration error corresponding to the second image frame and the first image frame, it continuously optimizes (or adjusts) the camera pose corresponding to the first image frame to obtain the camera pose with the smallest joint error, which is then used as the optimized camera pose corresponding to the first image frame.

[0058] For example, electronic devices can employ gradient optimization algorithms, such as gradient descent or adaptive moment estimation (ADAM) optimizers, to optimize (or adjust) the camera pose corresponding to the first image frame based on the joint error of photometric error and IMU pre-integration error, so as to obtain the camera pose with the minimum corresponding photometric error.

[0059] It is understood that this application does not limit the method of determining the joint error based on photometric error and IMU pre-integration error. For example, the joint error can be determined by the sum of photometric error and IMU pre-integration error, or by the sum of the product of photometric error and a first weighting coefficient and the product of IMU pre-integration error and a second weighting coefficient. The first weighting coefficient and the second weighting coefficient are values ​​greater than 0 and can be pre-configured in the electronic device.

[0060] In one possible implementation, after step S102 and before step S103, the electronic device may also optimize the camera pose corresponding to the first image frame with the goal of minimizing the photometric error between the second image frame and the first image frame.

[0061] For example: After the electronic device obtains the camera pose corresponding to the first image frame through the above step S102, it can use the camera pose corresponding to the first image frame obtained in step S102 as the initial value of the camera pose corresponding to the first image frame. Based on this, with the goal of minimizing the photometric error between the second image frame projected onto the camera pose corresponding to the first image frame and the first image frame, the camera pose corresponding to the first image frame is continuously optimized (or adjusted) to obtain the camera pose with the minimum photometric error, which is used as the optimized camera pose corresponding to the first image frame.

[0062] Through the above implementation, before optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error, the camera pose of the first image frame is optimized based on photometric error first. Visual pixel matching (photometric error) can be used as a constraint to correct the error caused by IMU offset to the detected camera pose change. This provides an accurate initial value for the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error, which can reduce the probability of getting trapped in local optima when optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error. This is beneficial to improving the efficiency and accuracy of optimizing the camera pose of the first image frame based on the joint error of photometric error and IMU pre-integration error.

[0063] The process of determining the camera pose corresponding to the first image frame described above can also be called front-end tracking. Figure 2This is a schematic diagram of the front-end tracking process provided in an embodiment of this application. The electronic device can predict the camera pose corresponding to a newly arriving image frame (i.e., the first image frame) based on the pre-integration result of the IMU data acquired by the IMU between two image frames (the second image frame and the first image frame) and the camera pose corresponding to the second image frame, thus completing coarse tracking of the camera pose. Then, the camera pose corresponding to the first image frame obtained through coarse tracking is used as the initial value of the camera pose corresponding to the first image frame. Based on this, and combining the camera data acquired by the camera (i.e., the second image frame and the first image frame), with the goal of minimizing the photometric error between the second image frame projected onto the camera pose corresponding to the first image frame and the first image frame, the camera pose corresponding to the first image frame is continuously optimized (or adjusted) to obtain the camera pose with the minimum corresponding photometric error. This optimized camera pose corresponding to the first image frame is then used to complete the refined tracking of the camera pose. Finally, by minimizing the joint error of the photometric error and the IMU pre-integration error corresponding to the second image frame and the first image frame, the final estimation of the camera pose corresponding to the first image frame can be completed, realizing tight-coupled motion tracking of vision (camera data) and inertial (IMU data).

[0064] S104: If the first image frame is a key image frame, update the 3D GS map according to the first image frame and the camera pose corresponding to the first image frame.

[0065] In this embodiment, the camera pose corresponding to the first image frame is jointly optimized by visual pixel matching (photometric error) and physical inertial constraints (IMU pre-integration error). This not only utilizes the IMU to restore the true physical scale for vision, but also uses vision to continuously correct the long-term drift of the IMU. This can provide real-time high-precision camera pose for 3D GS, thereby supporting online positioning and map updates based on 3D GS.

[0066] In one possible implementation, in order to save computing power, the electronic device can update the 3D GS map only when the first image frame is a key image frame, based on the first image frame and the camera pose corresponding to the first image frame.

[0067] The electronic device may determine whether the first image frame is a key image frame using one or more of the following methods.

[0068] Method 1: When the feature point matching ratio between the first image frame and the previous key image frame is less than or equal to the first matching ratio threshold, the electronic device determines the first image frame as the key image frame.

[0069] Feature points, also known as tracking points, are points that are unique (or salient) in a local area of ​​an image frame. For example, feature points can be corner points, edge points, or interest points. Corner points are points in an image whose grayscale values ​​change drastically in at least two directions; edge points are points located on the boundary lines where the grayscale values ​​of an image change sharply and have a large curvature on the boundary; interest points are the center points of regions that are significantly different from the surrounding areas in terms of color, grayscale, or texture.

[0070] The feature point matching ratio between the first image frame and the previous key image frame can be defined as the ratio of the number of successfully matched feature point pairs to the total number of feature points in the first image frame. If the feature point matching ratio between the first image frame and the previous key image frame is less than or equal to a first matching ratio threshold, it indicates that the content of the first image frame and the previous key image frame differs significantly, and the electronic device can identify the first image frame as the key image frame. The first matching ratio threshold can be 60%, 70%, or 80%, etc., and is not limited to any specific threshold.

[0071] Using the above method, it is possible to determine whether the first image frame is a key image frame based on whether the feature point matching ratio with the previous key image frame is less than or equal to the first matching ratio threshold. This can filter out image frames with high content similarity to the previous key image frame, which helps to reduce the number of times the 3D GS map needs to be updated and saves computing power.

[0072] Method 2: When the change in camera pose of the first image frame relative to the camera pose of the previous key image frame is greater than or equal to the camera pose change threshold, the electronic device determines the first image frame as the key image frame.

[0073] Using the above method, it is possible to determine whether the first acquired image frame is a key image frame based on whether the change in camera pose relative to the previous key image frame is greater than or equal to the threshold of camera pose change. This can filter out image frames with small changes in camera pose relative to the previous key image frame, which helps to reduce the number of times the 3D GS map needs to be updated and saves computing power.

[0074] Method 3: If the difference between the acquisition time of the first image frame and the acquisition time of the previous key image frame is greater than or equal to the first duration threshold, the electronic device determines the first image frame as the key image frame.

[0075] The above method allows us to determine whether a first image frame is a key image frame by only considering the interval between the acquisition times of the image frames, without relying on the content of the image frame or the corresponding camera pose. This helps to reduce the overhead of determining whether a first image frame is a key image frame.

[0076] Methods 1 and 2 described above are applicable to determining key image frames in scenarios where electronic devices have sufficient computing power, while method 3 described above is applicable to determining key image frames in scenarios where electronic devices have limited computing power.

[0077] In addition, based on the above method 1 and / or the above method 2, combining the above method 3 for the judgment of key image frames can also solve the problem that when the camera is stationary for a long time or moves slowly, there may be no key image frames for updating the 3D GS map for a long time, and the camera pose drift will continue to accumulate over time. This is beneficial to improving the accuracy of map construction and the utilization rate of computing power of electronic equipment.

[0078] It is understood that the above methods 1-3 are merely examples provided by this application for determining whether the first image frame is a key image frame, and this application does not limit the methods for determining whether the first image frame is a key image frame.

[0079] Since the aforementioned determination of the camera pose corresponding to the first image frame (i.e., front-end tracking) is based solely on data acquired by the camera and IMU between the second and first image frames, the determined camera pose may contain accumulated errors and lacks constraints from global information. To further improve the accuracy of camera pose determination, the electronic device can also determine multiple key image frames, including the first image frame; with the goal of minimizing the joint total error corresponding to these multiple key image frames, the camera pose corresponding to these multiple key image frames is optimized to obtain the optimized camera pose corresponding to the first image frame. The joint total error corresponding to multiple key image frames can be determined based on the photometric error and IMU pre-integration error corresponding to multiple sets of adjacent key image frames within the multiple key image frames.

[0080] Reference Figure 3 The diagram illustrates the backend optimization process. The electronic device can maintain a sliding window containing multiple key image frames. If the first image frame is a key image frame, the electronic device can update the sliding window, for example, by adding the first image frame or deleting the key image frame that has been stored for the longest time. The goal is to minimize the joint total error corresponding to the multiple key image frames in the sliding window, optimizing the camera pose corresponding to these key image frames to obtain the optimized camera pose corresponding to the first image frame. The number of key image frames included in the sliding window can be greater than or equal to 3, such as 5, 6, 7, or 8, etc., without limitation.

[0081] As an example: the sliding window includes 5 key image frames, which correspond to 4 sets of adjacent key image frames. The sum of the photometric errors corresponding to the 4 sets of adjacent key image frames can be used as the total photometric error. The sum of the IMU pre-integration errors corresponding to the 4 sets of adjacent key image frames can be used as the total IMU pre-integration error. The sum of the total photometric error and the total IMU pre-integration error can be the joint total error corresponding to multiple key image frames.

[0082] As another example: the sliding window includes 5 key image frames, which correspond to 4 groups of adjacent key image frames. Weight coefficients can be set for each of the 4 groups of key image frames. The weighted sum of the photometric errors corresponding to the 4 groups of adjacent key image frames can be used as the total photometric error. The weighted sum of the IMU pre-integration errors corresponding to the 4 groups of adjacent key image frames can be used as the total IMU pre-integration error. The sum of the total photometric error and the total IMU pre-integration error can be the joint total error corresponding to multiple key image frames.

[0083] It is understood that the above is merely an example of determining the joint total error of multiple key image frames in a sliding window based on the photometric error and IMU pre-integration error corresponding to multiple sets of adjacent key image frames in multiple key image frames. This application does not limit the method of determining the joint total error of multiple key image frames in a sliding window based on the photometric error and IMU pre-integration error corresponding to multiple sets of adjacent key image frames in multiple key image frames.

[0084] Example: Refer to Figure 3 The diagram of the backend optimization process shown can be used to treat the camera poses corresponding to multiple key image frames as multiple vertices in the graph model, and the photometric error and IMU pre-integration error corresponding to adjacent key image frames as the edges between two vertices corresponding to adjacent key image frames. The graph-based optimization algorithm is used to adjust (or iterate) the parameters of the vertices (i.e., camera poses) in the graph model so as to minimize the joint total error corresponding to multiple key image frames, and obtain the optimized camera poses corresponding to multiple key image frames.

[0085] In some embodiments, in order to improve optimization efficiency and balance optimization accuracy and real-time performance, the number of key image frames in the sliding window is less than or equal to a first quantity threshold, which can be 7, 8, 9 or 10, etc.

[0086] By optimizing the camera pose together with multiple key image frames, the camera pose error between multiple key image frames can be reduced, thereby reducing the accumulation of camera pose error at the first image frame, which is beneficial to improving the accuracy of the camera pose corresponding to the first image frame.

[0087] 3D GS maps can also be called 3D Gaussian maps or simply Gaussian maps without limitation. The following section uses the first image frame as the key image frame as an example to explain how an electronic device updates the 3D GS map based on the first image frame and the corresponding camera pose.

[0088] As an example: when the first image frame is a key image frame, the electronic device can determine the depth range information corresponding to the first image frame, which includes the depth range of multiple feature points in the first image frame. Based on the camera pose corresponding to the first image frame, the depth range information of multiple feature points in the first image frame, the first image frame (e.g., the positions of multiple feature points in the first image frame, and the color information of multiple feature points in the first image frame (e.g., RGB information, i.e., red (R), green (R), and blue (B) information)), Gaussian elements can be added to the 3D GS map.

[0089] The electronic device can employ methods such as adaptive density control (ADC), multi-view geometry splatting (MVG-Splatting), or mask-propagation densification GS (MPD-GS) to add Gaussian primitives to the 3D GS map based on the camera pose corresponding to the first image frame, the depth range information of multiple feature points in the first image frame, and the first image frame itself. This application does not limit the method used to add Gaussian primitives to the 3D GS map.

[0090] Gaussian elements can include attributes such as position, covariance matrix, opacity, and spherical harmonic color coefficient. Position describes the center coordinates of the Gaussian element in three-dimensional space (such as a 3D GS map); covariance matrix describes the shape, size, and orientation of the Gaussian element in three-dimensional space; opacity describes the light transmittance of the Gaussian element, with values ​​typically ranging from 0 to 1; and spherical harmonic color coefficient describes how the color of the Gaussian element changes with the viewing angle.

[0091] As another example: when the first image frame is a key image frame, the electronic device determines the depth range information corresponding to the first image frame. After the depth range information includes the depth range of multiple feature points in the first image frame, the 3D GS map can be projected onto the camera pose corresponding to the first image frame to obtain a first rendered depth map. The first rendered depth map includes the rendered depth of the aforementioned multiple feature points. For any feature point among the multiple feature points, if the rendered depth of the feature point is outside the depth range of the feature point, Gaussian elements are added to the 3D GS map according to the depth range corresponding to the feature point, the camera pose corresponding to the first image frame, and the information of the feature point in the first image frame (such as position and color information).

[0092] Therefore, considering that if the rendering depth of a feature point is within the depth range of the feature point, and the 3D GS map already has the corresponding Gaussian elements for that feature point, not adding Gaussian elements to the 3D GS map for this characteristic can reduce the redundancy of the 3D GS map and improve the update efficiency of the 3D GS map.

[0093] Alternatively, the electronic device can determine the depth range of multiple feature points in the first image frame using methods such as structure from motion (SFM), visual odometry (VO), triangulation, or an AI model for determining the depth range of feature points in an image frame.

[0094] As an example: After an electronic device extracts multiple feature points (such as corner points, edge points, or points of interest) from a first image, for any feature point, it can calculate the depth range of the feature point using triangulation based on the feature point's position in the first image frame, the feature point's position in another image frame captured by the camera (such as the second image frame or the previous key image frame before the first image frame), and the image pose corresponding to the first image frame and the camera pose corresponding to the other image frame.

[0095] In one possible implementation, after the electronic device determines the depth range of multiple feature points in the first image frame (i.e., the depth range information corresponding to the first image frame), it can also search for the depth values ​​of multiple feature points in multiple non-critical image frames adjacent to the first image frame using methods such as epipolar search, optical flow, or multi-view stereo vision; and update the depth range of multiple feature points based on the depth values ​​of multiple feature points in multiple non-critical image frames.

[0096] Through the above implementation, depth information from non-critical image frames can be added to critical image frames for updating 3D GS maps. This reduces the amount of depth information lost in non-critical image frames and helps improve the accuracy of 3D GS maps.

[0097] It is understandable that multiple non-critical image frames adjacent to the first image frame can be multiple non-critical image frames captured by the camera before the first image frame, such as multiple non-critical image frames captured by the camera between the first image frame and the previous critical image frame; they can also be multiple non-critical image frames captured by the camera after the first image frame, such as multiple non-critical image frames captured by the camera between the first image frame and the next critical image frame; of course, they can also be multiple non-critical image frames captured by the camera before and after the first image frame, without limitation.

[0098] The process of determining the depth range information corresponding to the first image frame (i.e., determining the depth range of multiple feature points in the first image frame) described above can also be called depth filtering. Figure 4 This is a schematic diagram of the depth filtering process provided in an embodiment of this application. For an image frame captured by a camera, the electronic device can determine whether the image frame is a key image frame. If the image frame is a key image frame, the electronic device can extract feature points in the image frame and determine the initial depth range of the feature points. For multiple non-key image frames adjacent to the key image frame captured by the camera, the electronic device can use an epipolar search method to search for the depth values ​​of the feature points in the multiple non-key image frames. Then, the initial depth range of the feature points and the depth values ​​obtained from the epipolar search are subjected to Gaussian-uniform fusion filtering to output the depth value range of the feature points. It should be noted that the electronic device can extract multiple feature points.

[0099] pass Figure 4 The depth filtering process shown can add depth information from non-critical image frames to critical image frames for updating the 3D GS map. This reduces the amount of depth information lost in non-critical image frames and helps improve the accuracy of the 3D GS map.

[0100] In some embodiments, to avoid excessive expansion of the 3D GS map, the electronic device can also delete Gaussian elements with a contribution value less than or equal to a contribution threshold, and merge Gaussian elements with similar locations or attributes, based on their contribution to the 3D GS map, thereby improving the compactness and real-time performance of the 3D GS map. The contribution of a Gaussian element can be determined based on its opacity and volume; the more opaque and larger the Gaussian element, the higher its contribution. For example, the contribution can be determined by the product of the Gaussian element's opacity and volume, or by a weighted sum of the Gaussian element's opacity and volume; there is no limitation on this.

[0101] In some implementations, to improve the accuracy of 3D GS map construction, after updating the 3D GS map based on the first image frame, the electronic device can also update the Gaussian elements in the 3D GS map through a backpropagation algorithm based on at least one of photometric loss, depth range loss, or scale consistency loss.

[0102] The photometric loss can be determined based on the rendered color map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the first image frame itself.

[0103] The depth range loss can be determined based on the second rendered depth map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the depth range information corresponding to the first image frame (which includes the depth range of multiple feature points in the first image frame). The scale consistency loss is determined based on the scale factors corresponding to multiple key image frames, including the first image frame. Taking the first image frame as an example, the scale factor corresponding to the first image frame can be determined based on the ratio of the camera displacement detected by the IMU between the second image frame and the first image frame to the camera displacement between the second image frame and the first image frame determined by optical algorithms such as SFM. Optionally, these multiple key image frames can be located within a sliding window.

[0104] For example, an electronic device may employ a differentiable rasterizer to sort the Gaussian pixels of a 3D GS map by depth according to the camera pose corresponding to the first image frame, and perform opacity α blending to generate a rendered color map and a second rendered depth map.

[0105] Optionally, the photometric loss between the first image frame and the rendered color image can be determined using the following formula:

[0106] in, Indicates luminosity loss, This represents the total number of pixels in the first image frame (or rendered color image). Indicates the first The RGB predicted value of each pixel in the rendered color image. Indicates the first The RGB values ​​of each pixel in the first image frame express The L1 norm is used to calculate the sum of the absolute errors of the three RGB channels of a pixel.

[0107] It should be noted that before calculating the photometric loss, it is necessary to... and Normalization is performed to eliminate and The units of measurement. For example, the RGB values ​​of a pixel range from [0, 255]. and Divide each by the maximum possible RGB value (e.g., 255), and... and Mapped to [0, 1], then normalized. This ensures that the formula... , , , All of these are dimensionless parameters.

[0108] Optionally: The depth range information corresponding to the first image frame (which includes the depth range of multiple feature points in the first image frame) and the depth range loss between the second rendered depth map can be determined using the following formula:

[0109] in, Indicates depth range loss. This represents the total number of feature points in the first image frame. Indicates the first image frame The depth of each feature point in the second rendered depth map Indicates the first image frame The minimum value of the depth range corresponding to each feature point. Indicates the first image frame The maximum value of the depth range corresponding to each feature point; when Less than , It is a positive value. A value greater than 0 results in a squared error, which can be represented as... Exceeding the minimum limit The resulting constraints and penalties; when Greater than , It is a positive value. A value greater than 0 results in a squared error, which can be represented as... Exceeding the maximum limit The resulting constraints and penalties.

[0110] It should be noted that before calculating the depth range loss, it is necessary to... , and Normalization is performed to eliminate , and The unit is [0, 100], and the unit is meters. For example, if the depth of a pixel exceeds a certain range, problems such as blurriness and distortion will occur. In this application, the reliable depth range was set to [0, 100] during the experiment, and the unit is meters. , and Divide each by the maximum acceptable depth value (e.g., 100m), and... , and Mapped to [0, 1], then normalized. This ensures that the formula... , , , , All of these are dimensionless parameters.

[0111] Alternatively, the scale consistency loss can be determined using the following formula:

[0112] in, Indicates the loss of scale consistency. This indicates the total number of key image frames within the sliding window. This indicates the number of adjacent keyframe pairs within the sliding window. Indicates the first [number]th ... The scale factor corresponding to each key image frame. Indicates the first [number]th ... The scale factor corresponding to each key image frame. express The square of the L2 norm.

[0113] It should be noted that in the formula for calculating the scale consistency loss above, when calculating the scale consistency loss, it will be done through... right After normalization, a dimensionless value is obtained. The total number of keyframes within the sliding window is represented by a dimensionless parameter, thus allowing us to obtain dimensionless values. .

[0114] The aforementioned photometric loss can be used to characterize the appearance consistency between the first image frame and the rendered color map; the depth range loss can be used to characterize the geometric rationality of the depth range corresponding to the second rendered depth map and the first image frame; the scale consistency loss can constrain the consistency of the global scale factor between different key image frames and avoid scale drift.

[0115] In the embodiments of this application, the Gaussian elements in the 3D GS map can be updated through backpropagation algorithm by at least one of photometric loss, depth range loss, or scale consistency loss, thereby improving the accuracy of 3G GS map construction.

[0116] For example, the Gaussian elements in a 3D GS map can be updated using the backpropagation algorithm based on the combined losses of photometric loss, depth range loss, and scale consistency loss, thereby improving the accuracy of 3G GS map construction.

[0117] Alternatively, the combined loss of photometric loss, depth range loss, and scale uniformity loss can be determined according to the following formula:

[0118] in, Indicates joint loss, Indicates photometric loss, Indicates depth range loss, Indicates scale consistency loss, The weighting coefficients corresponding to photometric loss The weighting coefficients corresponding to the depth range loss are represented by... The weight coefficients represent the scale consistency loss, where , , Greater than 0. In this formula , , , , , , It is a dimensionless parameter.

[0119] In some embodiments, before updating the 3D GS map, the electronic device may also initialize the 3D GS map based on the first (i.e., the first) key image frame acquired, i.e., initialize the Gaussian elements in the 3D GS map.

[0120] The first key image frame is the first image frame used for 3G GS map construction. This image frame can be the i-th image frame acquired by the camera after the electronic device initiates 3G GS map construction, where i is an integer greater than or equal to 1.

[0121] For example, an electronic device can add Gaussian primitives to a 3D GS map based on the acquired first key image frame, the camera pose corresponding to the first key image frame, and the depth range information corresponding to the first key image frame, which includes the depth range of multiple feature points in the first key image frame. The specific implementation of adding Gaussian primitives to the 3D GS map can refer to the above implementation of adding Gaussian primitives to the 3D GS map based on the first image frame, the camera pose corresponding to the first image frame, and the depth range information corresponding to the first image frame, and will not be repeated here.

[0122] The camera pose corresponding to the first key image frame can be predefined by the electronic device. For example, the camera pose corresponding to the first key image frame can be the origin in the three-dimensional coordinate system corresponding to the 3D GS map, that is, no displacement (the three degrees of freedom corresponding to displacement are all 0) and no rotation (the three degrees of freedom corresponding to rotation are all 0). As for how to determine the depth range information corresponding to the first key image frame, you can refer to the above implementation of determining the depth range information corresponding to the first image frame, and will not be repeated here.

[0123] Figure 5 This is a schematic diagram of a positioning and mapping system provided in an embodiment of this application. The system may include a visual-inertial module and a map generation module. The visual-inertial module takes image frames acquired by a camera and IMU data acquired by an IMU as input. By performing visual-inertial processing on the image frames and IMU data, it outputs key image frames, the camera pose corresponding to the key image frames, and depth range information corresponding to the key image frames. The depth range information includes the depth range of multiple feature points in the key image frames. The map generation module can construct a 3D GS map based on the above information output by the visual-inertial module.

[0124] The visual-inertial module may include an initialization unit, a front-end tracking unit, a depth filtering unit, and a back-end optimization unit. (See reference...) Figure 6 The diagram illustrates the visual-inertial processing flow. The initialization unit can be used to perform visual-inertial initialization operations, calibrating the camera and IMU. The specific implementation of the visual-inertial initialization operation can be referred to the description in step S102 above and will not be repeated here. The front-end tracking unit is used to determine the camera pose corresponding to the image frame (such as the first image frame) based on the pre-integration results of the IMU data between image frames after the initialization unit performs visual-inertial initialization. The specific implementation can be referred to the above. Figure 2 The implementation of the front-end tracking process will not be elaborated further; the depth filtering unit can be used to determine whether an image frame (such as the first image frame) is a key image frame. If the image frame is a key image frame, the depth range information corresponding to the key image frame is determined. The specific implementation can be referred to the above. Figure 4 The implementation of the depth filtering process will not be elaborated further. The back-end optimization unit can add key image frames to a sliding window, optimize multiple key image frames in the sliding window together, and output the key image frames and the camera poses corresponding to the optimized key image frames. The specific implementation can be found above. Figure 3 The implementation of the backend optimization process will not be elaborated further.

[0125] The map generation module may include a map initialization unit, a map update unit, and a joint optimization unit. (See reference...) Figure 7The diagram illustrates the map generation process. If the key image frame is the first key image frame, the map initialization unit can initialize the 3D GS map (i.e., add Gaussian elements to the 3D GS map) based on the first key image frame, the depth range information corresponding to the first key image frame, and the camera pose corresponding to the first key image frame. If the key image frame is not the first key image frame, the map update unit can update the 3D GS map based on the key image frame, the depth range information corresponding to the key image frame, and the camera pose corresponding to the key image frame (e.g., add Gaussian elements corresponding to non-first key image frames to the 3D GS map). The joint optimization module can use a differentiable rasterizer to sort the Gaussian elements of the 3D GS map by depth based on the camera pose corresponding to non-first key image frames, perform opacity α blending, generate a rendered color map and a rendered depth map, and calculate the joint loss of photometric loss, depth range loss, etc., based on the non-first key image frames and the depth information corresponding to non-first key image frames, and optimize the 3D GS map based on the joint loss.

[0126] The above mainly introduces the solution provided by this application from the perspective of method and process. The following will elaborate on the technical solution of the embodiments of this application from the perspective of hardware or logical module division. It is understood that, in order to achieve the above functions, the positioning and map building device may include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily realize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] Figure 8 The diagram illustrates a possible exemplary block diagram of a positioning and mapping device 800 according to an embodiment of this application. This positioning and mapping device 800 may exist as a software module or a hardware module. The positioning and mapping device 800 may include a visual inertial module 810 and a map generation module 820.

[0128] The visual inertial module 810 is used to acquire a first image frame; and to determine the camera pose corresponding to the first image frame based on the camera pose change detected by the IMU between the second image frame and the first image frame, and the camera pose corresponding to the second image frame, wherein the second image frame is an image frame acquired before the first image frame; and to optimize the camera pose corresponding to the first image frame with the goal of minimizing the joint error of the photometric error and the IMU pre-integration error between the second image frame and the first image frame; wherein the photometric error is the error between the second image frame projected onto the camera pose corresponding to the first image frame and the photometric error of the first image frame, and the IMU pre-integration error is the error between the camera pose change between the camera pose corresponding to the second image frame and the camera pose change detected by the IMU; The map generation module 820 is used to update the 3D GS map based on the first image frame and the camera pose corresponding to the first image frame when the first image frame is a key image frame.

[0129] In one possible design, the visual inertial module 810 is further configured to optimize the camera pose of the first image frame with the goal of minimizing the photometric error between the second image frame and the first image frame before optimizing the camera pose of the first image frame with the goal of minimizing the joint error of the photometric error and the IMU pre-integration error between the second image frame and the first image frame.

[0130] In one possible design, the first image frame is a key image frame that satisfies at least one of the following conditions: the feature point matching ratio between the first image frame and the previous key image frame is less than or equal to a first matching ratio threshold; the camera pose change of the camera pose corresponding to the first image frame relative to the camera pose corresponding to the previous key image frame is greater than or equal to a camera pose change threshold; and the difference between the acquisition time of the first image frame and the acquisition time of the previous key image frame is greater than or equal to a first duration threshold.

[0131] In one possible design, the visual inertial module 810 is further configured to determine multiple key image frames, including the first image frame, before updating the 3D GS map based on the first image frame and the camera pose corresponding to the first image frame; optimize the camera pose corresponding to the multiple key image frames, including the camera pose corresponding to the first image frame, with the goal of minimizing the joint total error corresponding to the multiple key image frames; wherein the joint total error is determined based on the photometric error and IMU pre-integration error corresponding to multiple sets of adjacent key image frames in the multiple key image frames.

[0132] In one possible design, the visual inertial module 810 is further used to determine the depth range information corresponding to the first image frame, the depth range information including the depth range of multiple feature points in the first image frame; when the map generation module 820 updates the 3D GS map according to the first image frame and the camera pose corresponding to the first image frame, it is specifically used to project the 3D GS map onto the camera pose corresponding to the first image frame to obtain a first rendered depth map, the first rendered depth map including the rendering depth of multiple feature points; for any feature point among the multiple feature points, if the rendering depth of the feature point is outside the depth range of the feature point, Gaussian elements are added to the 3D GS map according to the position and color information of the feature point in the first image frame, the depth range of the feature point, and the camera pose corresponding to the first image frame.

[0133] In one possible design, the visual inertial module 810 is further configured to search for depth values ​​of multiple feature points in multiple non-critical image frames based on multiple non-critical image frames adjacent to the first image frame; and update the depth range of multiple feature points based on the depth values ​​of multiple feature points in multiple non-critical image frames.

[0134] In one possible design, the map generation module 820 is further configured to update Gaussian elements in the 3D GS map using a backpropagation algorithm based on at least one of photometric loss, depth range loss, or scale consistency loss; wherein the photometric loss is determined based on a rendered color map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the first image frame; the depth range loss is determined based on a second rendered depth map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the depth range information corresponding to the first image frame; the scale consistency loss is determined based on the scale factors corresponding to multiple key image frames, including the first image frame.

[0135] In one possible design, the map generation module 820 is also used to initialize Gaussian elements in the 3D GS map based on the first key image frame acquired, the camera pose corresponding to the first key image frame, and the depth range information corresponding to the first key image frame.

[0136] This application also provides an electronic device in its embodiments. For example... Figure 9As shown, the electronic device 900 includes a memory 910 and a processor 920, which are interconnected. Optionally, the memory 910 and the processor 920 can be interconnected via a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can also be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0137] The memory 910 stores computer programs or instructions; the processor 920 can execute the computer programs or instructions stored in the memory 910 to perform the positioning and map construction in the above method embodiments.

[0138] This application also provides a computer-readable storage medium storing a program or instructions thereon, which, when executed by a processor, can implement the positioning and map construction method in the above method embodiments.

[0139] This application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the positioning and map building method described in the above method embodiments can be implemented.

[0140] This application also provides a chip system including a processor, which is coupled to a memory for storing programs or instructions. When the program or instructions are executed by the processor, the positioning and map building method described in the above method embodiments can be implemented.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0146] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for positioning and map construction, characterized in that, The method includes: Obtain the first image frame; Based on the camera pose change detected by the inertial measurement unit (IMU) between the second image frame and the first image frame, and the camera pose corresponding to the second image frame, the camera pose corresponding to the first image frame is determined, and the second image frame is an image frame acquired before the first image frame. With the goal of minimizing the joint error of the photometric error and the IMU pre-integration error between the second image frame and the first image frame, the camera pose corresponding to the first image frame is optimized; wherein, the photometric error is the error between the photometric error of the second image frame projected onto the camera pose corresponding to the first image frame and the photometric error of the first image frame, and the IMU pre-integration error is the error between the camera pose change of the second image frame and the camera pose change detected by the IMU; If the first image frame is a key image frame, the 3D Gaussian sputtering 3D GS map is updated according to the first image frame and the camera pose corresponding to the first image frame. The step of updating the 3D GS map based on the first image frame and the camera pose corresponding to the first image frame includes: Determine the depth range information corresponding to the first image frame, wherein the depth range information includes the depth range of multiple feature points in the first image frame; Based on multiple non-critical image frames adjacent to the first image frame, search for the depth values ​​of the multiple feature points in the multiple non-critical image frames respectively; For any one of the plurality of feature points, the depth range of the feature point and the depth value of the feature point in the plurality of non-critical image frames are subjected to Gaussian-uniform fusion filtering to update the depth range of the feature point; The 3D GS map is projected onto the camera pose corresponding to the first image frame to obtain a first rendering depth map, which includes the rendering depth of the plurality of feature points. For any of the plurality of feature points, if the rendering depth of the feature point is outside the depth range of the feature point, Gaussian elements are added to the 3D GS map based on the position and color information of the feature point in the first image frame, the depth range of the feature point, and the camera pose corresponding to the first image frame.

2. The method as described in claim 1, characterized in that, Before optimizing the camera pose corresponding to the first image frame with the objective of minimizing the joint error of the photometric error and the IMU pre-integration error corresponding to the second image frame and the first image frame, the method further includes: With the goal of minimizing the photometric error between the second image frame and the first image frame, the camera pose corresponding to the first image frame is optimized.

3. The method as described in claim 1 or 2, characterized in that, The first image frame is a key image frame that satisfies at least one of the following conditions: The feature point matching ratio between the first image frame and the previous key image frame is less than or equal to the first matching ratio threshold. The change in camera pose corresponding to the first image frame relative to the camera pose corresponding to the previous key image frame is greater than or equal to the camera pose change threshold. The difference between the acquisition time of the first image frame and the acquisition time of the previous key image frame is greater than or equal to the first duration threshold.

4. The method as described in claim 3, characterized in that, Before updating the 3D GS map based on the first image frame and the camera pose corresponding to the first image frame, the method further includes: Identify multiple key image frames, including the first image frame. With the goal of minimizing the joint total error corresponding to the multiple key image frames, the camera pose corresponding to the multiple key image frames is optimized, including the camera pose corresponding to the first image frame. The joint total error is determined based on the photometric error and IMU pre-integration error corresponding to multiple groups of adjacent key image frames in the multiple key image frames.

5. The method as described in claim 1 or 2, characterized in that, The method further includes: Based on at least one of photometric loss, depth range loss, or scale consistency loss, the Gaussian elements in the 3D GS map are updated using a backpropagation algorithm. The photometric loss is determined based on the rendered color map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the first image frame. The depth range loss is determined based on the second rendered depth map generated by projecting the 3D GS map onto the camera pose corresponding to the first image frame, and the depth range information corresponding to the first image frame. The scale consistency loss is determined based on the scale factors corresponding to multiple key image frames, including the first image frame.

6. A positioning and mapping device, characterized in that, Includes a visual inertial module and a map generation module; The visual inertial module is used to acquire a first image frame; and to determine the camera pose corresponding to the first image frame based on the camera pose change detected by the inertial measurement unit (IMU) between the second image frame and the first image frame, and the camera pose corresponding to the second image frame, wherein the second image frame is an image frame acquired before the first image frame. With the goal of minimizing the joint error of photometric error and IMU pre-integration error between the second image frame and the first image frame, the camera pose corresponding to the first image frame is optimized; wherein, the photometric error is the error between the photometric error of the second image frame projected onto the camera pose corresponding to the first image frame and the photometric error of the first image frame, and the IMU pre-integration error is the error between the pose change of the camera pose corresponding to the second image frame and the camera pose change detected by the IMU; The map generation module is used to update the 3D Gaussian sputtering 3D GS map based on the first image frame and the camera pose corresponding to the first image frame when the first image frame is a key image frame. The visual inertial module is further configured to determine the depth range information corresponding to the first image frame, the depth range information including the depth range of multiple feature points in the first image frame; search for the depth values ​​of the multiple feature points in the multiple non-critical image frames adjacent to the first image frame; and update the depth range of the feature point by performing Gaussian-uniform fusion filtering on the depth range of the feature point and the depth value of the feature point in the multiple non-critical image frames for any feature point among the multiple feature points. When updating the 3D GS map based on the first image frame and the camera pose corresponding to the first image frame, the map generation module specifically projects the 3D GS map onto the camera pose corresponding to the first image frame to obtain a first rendering depth map. The first rendering depth map includes the rendering depth of the plurality of feature points. For any feature point among the plurality of feature points, if the rendering depth of the feature point is outside the depth range of the feature point, Gaussian elements are added to the 3D GS map based on the position and color information of the feature point in the first image frame, the depth range of the feature point, and the camera pose corresponding to the first image frame.

7. An electronic device, characterized in that, Including processor and memory; The processor is configured to execute a computer program or instructions stored in the memory, such that the processor performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a processor, cause the method as described in any one of claims 1-5 to be implemented.

Images