A neural-radiance-field-based positioning mapping method and device, and a storage medium

Abstract

The application discloses a positioning mapping method and device based on a neural radiation field and a storage medium, wherein the method comprises the following steps: acquiring image data; screening the image data to obtain key frames; performing SFM pose optimization on all the key frames to obtain an initial pose of each key frame; performing triangulation processing on feature points according to the initial pose, and storing the processed feature points as global map points; for each key frame, constructing a visual constraint by using the global map points in its continuous co-view key frames, performing Ceres back-end optimization according to the visual constraint, and outputting a pose; performing coordinate system correction on the output pose, inputting the corrected pose and the image data into a trained neural radiation field, and reconstructing a corresponding three-dimensional model. The application proposes a single-purpose visual front-end tracking scheme, combines dense mapping, solves the problem that sparse tracking cannot realize dense three-dimensional reconstruction in a traditional rule-based SLAM algorithm, and can be widely applied to the field of positioning and map construction.

Description

Technical Field

[0001] This invention relates to the field of positioning and mapping, and in particular to a positioning and mapping method, apparatus and storage medium based on neural radiation fields. Background Technology

[0002] For RGB images, existing neural radiation field methods fail to converge and reconstruct effectively, while for grayscale images, they cannot reconstruct the grayscale scene. In existing solutions, neural radiation field SLAM methods for reconstructing continuous scenes mostly derive the camera pose directly from the radiation field. However, this method requires the continuity of spatial feature encoding, thus consuming significant computational resources and GPU memory. Furthermore, using photometric alignment of the entire image pixel also leads to a substantial increase in the computational cost of pose derivation. Summary of the Invention

[0003] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a localization mapping method, device and storage medium based on neural radiation field.

[0004] The technical solution adopted in this invention is:

[0005] A localization mapping method based on neural radiation fields includes the following steps:

[0006] Acquire image data, which is RGB image data or grayscale image data;

[0007] Based on the image data, construct an image pyramid for the current image and perform forward and reverse optical flow alignment to obtain the feature points of the current image;

[0008] Image data is filtered according to preset conditions to obtain keyframes;

[0009] After detecting and determining that the number of keyframes has accumulated to a preset number, SFM pose optimization is performed on all keyframes to obtain the initial pose of each keyframe.

[0010] Based on the initial pose, the feature points are triangulated, and the processed feature points are stored as global map points.

[0011] For each keyframe, construct visual constraints using global map points in its consecutive co-view keyframes, and perform Ceres backend optimization based on the visual constraints.

[0012] After optimizing the Ceres backend, the image and pose are added to the bag-of-words model to determine whether to perform loop closure detection on the current image. If no loop closure is performed, the pose is directly output.

[0013] The output pose is corrected in coordinate system, and the corrected pose and image data are input into the trained neural radiation field to reconstruct the corresponding 3D model.

[0014] Furthermore, the image data is acquired by a monocular camera.

[0015] Furthermore, the localization and mapping method also includes the step of fusing inertial data:

[0016] Acquire inertial data, which includes acceleration and angular acceleration collected by inertial sensors;

[0017] Estimate the uncertainty of the current inertial sensor measurement state, and determine whether to perform data fusion based on the uncertainty;

[0018] If data fusion is determined, pose optimization is performed by combining inertial data and keyframes to obtain the initial pose of each keyframe;

[0019] The Ceres backend optimization based on visual constraints includes:

[0020] For each keyframe, an inertial pre-integration constraint is constructed, and the covariance matrix is ​​used to measure the weight of each constraint.

[0021] Furthermore, the estimation of the uncertainty of the current inertial sensor measurement state and the optimization to reduce measurement noise include:

[0022] For inherent frequency noise, use a Butterworth filter for frequency domain filtering;

[0023] For unknown measurement noise, the measurement covariance of the inertial sensor is compensated using relative acceleration, as expressed below:

[0024]

[0025]

[0026] in, F is the covariance matrix of the inertial sensor at time t+δt, consisting of two parts: one is the accumulated error at time t, and the other is the measurement error at the current time; t G t δt is the first derivative of the IMU error function at time t; δt is the time difference between two frames of IMU data; n represents the measurement noise; P(a t ) represents the correction value, λ is the compensation base of the inertial sensor, and a t t is the measured acceleration, and g is the gravitational acceleration.

[0027] Furthermore, the neural radiation field reconstructs the three-dimensional model in the following manner:

[0028] Acquire the image and its corresponding pose, and construct a ray centered on the image pose; wherein, for each pixel, sample multiple 3D points along the ray;

[0029] During the sampling of 3D points, an occupancy grid is maintained to identify the occupancy of 3D points in each region. When light passes through the grid, the transmittance of the light is calculated. If it is less than a threshold, the light stops sampling.

[0030] The three-dimensional space is segmented at multiple scales to obtain the voxels to which the sampled three-dimensional points belong. The voxel sequence numbers to which the three-dimensional points belong are mapped to a hash feature vector table to obtain the corresponding feature vectors. The feature vectors of the three-dimensional points are obtained by concatenating the vectors mapped at multiple scales.

[0031] Input the feature vector of a 3D point into an MLP network to obtain the color value and occupancy of the 3D point.

[0032] The entire scene's 3D model is reconstructed by rendering based on the color values ​​and occupancy of the obtained 3D points.

[0033] Furthermore, when constructing dense maps, the color rendering formula used by the differentiable renderer of the neural radiation field is as follows:

[0034]

[0035]

[0036] In the formula, E(c) represents the predicted color of the pixel. c represents the transparency of light rays emitted from the optical center of the camera at point i. i It is the color prediction of 3D points by an MLP network, σ i It is the occupancy prediction of three-dimensional points by the MLP network, σ i It is the occupancy prediction of the 3D point at location i by the MLP network; δ i This represents the step size from the 3D point at position i+1 to the 3D point at position i.

[0037] The loss function of an MLP network is defined as follows:

[0038] For images that exist in the RGB color space:

[0039]

[0040] For images that exist only in grayscale space:

[0041]

[0042] Where T is the pose of the image, θ is the parameter of the MLP network and the feature vector of the multi-resolution hash encoding, and I(c) represents the observed pixel color of the image.

[0043] Furthermore, during the training process of the neural radiation field, the L2 loss function is used to calculate the loss.

[0044] Furthermore, the step of adding the image and pose to the bag-of-words model and determining whether to perform loop closure detection on the current image includes:

[0045] The image and pose are added to the bag-of-words model, and the average rotation angle within the sliding window is detected. The number of frames between the image frame corresponding to the last successful loop closure and the current image frame is calculated. Based on the number of frames, it is determined whether to perform loop closure detection on the current image.

[0046] Another technical solution adopted in this invention is:

[0047] A localization and mapping device based on neural radiation fields, comprising:

[0048] At least one processor;

[0049] At least one memory for storing at least one program;

[0050] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0051] Another technical solution adopted in this invention is:

[0052] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the method described above.

[0053] The beneficial effects of this invention are: This invention proposes a visual front-end tracking scheme that supports monocular vision, combined with dense mapping, to solve the problem that sparse tracking in traditional rule-based SLAM algorithms cannot achieve dense 3D reconstruction. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of sparse tracking in an embodiment of the present invention;

[0056] Figure 2 This is a flowchart of the training process for the neural radiation field in an embodiment of the present invention;

[0057] Figure 3 This is a flowchart of loop closure detection in an embodiment of the present invention;

[0058] Figure 4 This is the reconstruction result of a large grayscale scene in an embodiment of the present invention. Detailed Implementation

[0059] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0061] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0062] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0063] Traditional rule-based SLAM methods can infer the current camera pose with only a few feature points. Therefore, this paper proposes a method that combines traditional rule-based SLAM with learning-based neural radiation field reconstruction.

[0064] like Figure 1As shown, this embodiment provides a localization mapping method based on neural radiation fields, including the following steps:

[0065] S1. Acquire image data, wherein the image data is RGB image data or grayscale image data.

[0066] S2. Based on the image data, construct an image pyramid for the current image and align the forward and reverse optical flows to obtain the feature points of the current image.

[0067] S3. Filter the image data according to preset conditions to obtain keyframes. As an optional implementation, by detecting how far the current image has moved from the previous image, images whose moving distance exceeds a threshold are selected as keyframes.

[0068] S4. After detecting and determining that the number of keyframes has accumulated to a preset number, perform SFM pose optimization on all keyframes to obtain the initial pose of each keyframe.

[0069] After accumulating a certain number of keyframes, perform SFM (Structure From Motion) pose optimization on all keyframes to obtain the initial pose estimate for each keyframe.

[0070] As an optional implementation, pose optimization can also be performed by fusing IMU data (i.e., inertial data). If IMU data is available, frequency domain filtering is performed using a Butterworth filter for the body's inherent frequency noise and vibration. For unknown noise, covariance propagation is performed using relative acceleration to compensate for covariance, estimating the uncertainty of the current IMU measurement state. IMU constraints are jointly initialized with the reprojection constraints of keyframes, while aligning the gravity vector and optimizing its scale. In this embodiment, IMU data is acquired through an inertial sensor (IMU).

[0071] S5. Based on the initial pose, perform triangulation of the feature points and store the processed feature points as global map points.

[0072] S6. For each keyframe, construct visual constraints using global map points in its consecutive co-view keyframes, and perform Ceres backend optimization based on the visual constraints.

[0073] For each keyframe, we construct visual constraints using global map points in its successive co-view keyframes. If IMU data exists, we also construct IMU pre-integration constraints between each keyframe, using a covariance matrix to measure the weight of each constraint.

[0074] S7. After optimization in the Ceres backend, add the image and pose to the bag-of-words model and determine whether to perform loop closure detection on the current image. If not, output the pose directly.

[0075] After optimization of the Ceres backend, the image and pose are added to the bag-of-words model and the average rotation angle within the window is detected using a sliding window. The number of frames that have passed since the last successful loop closure is detected is used to determine whether to perform loop closure detection on the current image. If loop closure is not performed, the pose is directly output.

[0076] S8. Correct the coordinate system of the output pose, and input the corrected pose and image data into the trained neural radiation field to reconstruct the corresponding three-dimensional model.

[0077] As an optional implementation, see [link to implementation details]. Figure 2 The aforementioned neural radiation field is obtained through training in the following manner:

[0078] A1. After obtaining the estimated pose and image, construct rays centered on the image pose. The expression is as follows:

[0079] r(t) = o + td

[0080] Where o is the optical center of the camera, usually represented by the translation in pose. d is the ray direction; for each pixel, its ray direction is equal to the direction vector from the optical center to the pixel on the unit plane. t is our sampling step size. Therefore, for each pixel, we can sample multiple 3D points (i.e., three-dimensional points) along the ray.

[0081] A2. During the sampling of 3D points, we maintain an occupancy grid, which identifies the occupancy of 3D points in each region. When light passes through this grid, the transmittance of the light is calculated. If it is less than a threshold, the light stops sampling.

[0082] A3. Perform multi-scale segmentation of the 3D space, calculate which voxels each sampled 3D point belongs to, and map the voxel sequence number of the 3D point to a hash feature vector table to obtain the corresponding feature vector. Concatenate the multi-scale mapped vectors to obtain the feature vector of the 3D point.

[0083] A4. By using the feature vectors of 3D points as input to the MLP network, we obtain the color value and occupancy of each 3D point. Through integration along the optical path, we obtain the pixel values ​​of the scene observed from that viewpoint. The input RGB image serves as the ground truth to supervise the training of the neural radiation field. Our optimization variables include not only the MLP parameters and feature vector table but also the given estimated pose. Thus, we can densely reconstruct a 3D model of the entire scene.

[0084] As can be seen from the above, the method of this embodiment can perform fast tracking and reconstruction using only a monocular camera, and also supports fast tracking and dense reconstruction using a monocular camera and inertial measurement tightly coupled.

[0085] For monocular cameras, we extract feature points from grayscale images and use pyramid optical flow to map these feature points during tracking. To maintain the number of feature points, we re-extract features for new images until the specified number is reached. If IMU data is available, we wait for joint IMU initialization; otherwise, we perform pure visual initialization using SFM.

[0086] Regarding the IMU noise problem, analysis reveals that it originates from both fixed vibration noise from the IMU itself and random noise during the measurement process. Therefore, this embodiment proposes a noise processing method: for inherent frequency noise, a Butterworth filter is used for frequency domain filtering. For unknown measurement noise, previous methods always assumed the error velocity conformed to a Wiener process with Gaussian white noise. However, under instantaneous high acceleration motion, the error velocity deviates from this assumption, leading to larger random walks and ultimately, positioning failure. Therefore, this embodiment considers using relative acceleration to compensate for the IMU's measurement covariance. This is expressed as:

[0087]

[0088]

[0089] in, This is the covariance matrix of IMU at time t+δt, which consists of two parts: the accumulated error at time t and the measurement error at the current time. We use a Gaussian probability model for covariance propagation, where F... t G t It is the first derivative of the IMU error function at time t. Meanwhile, to correct the assumption that the bias velocity is a Gaussian white noise model, we added P(a t The correction factor is λ, where λ is the IMU's compensation base. This base varies for each IMU, but generally, for λ ∈ [10, 100], the L2 norm of the relative acceleration relative to gravitational acceleration at the current moment is used as the exponent of the compensation base. The larger the relative instantaneous acceleration, the greater the correction bias we compensate for, resulting in a larger covariance matrix at the current moment, indicating less reliable measurement data. This allows us to reduce the weight of the IMU error term in backend optimization.

[0090] As an optional implementation, global pose correction for loop closure detection in step S7 is a relatively time-consuming operation. To synchronize the neural radiation field mapping and tracking processes and accelerate the overall processing speed, we adopt a sliding window rotation angle detection loop closure strategy. (See the step diagram.) Figure 3 As shown.

[0091] As an optional implementation, when constructing dense maps, we refer to the color rendering formula used by a typical differentiable renderer for neural radiation fields. The discretization calculation formula can be expressed as:

[0092]

[0093]

[0094] Where E(c) represents the predicted color of the pixel. c represents the transparency of light rays emitted from the optical center of the camera at point i. i It is the color prediction of 3D points by an MLP network, σ i This refers to the MLP network's prediction of occupancy rates for 3D points. Therefore, the network's loss function is defined as follows:

[0095] For images that exist in the RGB color space:

[0096]

[0097] For images that exist only in grayscale space:

[0098] linear(gray) = f(gray)

[0099]

[0100] T represents the pose of the image, and θ represents the network parameters of the MLP and the feature vector of the multi-resolution hash encoding. By updating T and θ, we can reconstruct the neural radiation field of the scene. For grayscale images, we map the grayscale space to a linear color space, and then calculate the L2 loss function based on the color in the linear space and the expected value of the current pixel. For the feature encoding of spatial points, we use multi-resolution hash encoding.

[0101] Due to the instability of manual presampling of rays, it cannot accurately sample the 3D surface of objects in different unknown environments, and it is prone to the problem of failing to sample feature points in open spaces. Furthermore, we maintain the transmission matrix of rays and calculate the step probability of the current ray in the trained neural radiation field. We sample 3D points based on ray transmittance; when the transmittance is less than a certain threshold, we consider the current ray to be invisible. This avoids wasting computational resources on sampling and calculation in open spaces, allowing us to focus more on training MLP parameters on the surface of 3D objects, thus accelerating scene reconstruction.

[0102] We fuse traditional rule-based SLAM pose estimation with neural radiation fields, selecting keyframes with large disparities from each keyframe group as inputs for the multi-resolution hash-encoded neural radiation field. For pure grayscale images, our reconstruction results are as follows. Compared to RGB image reconstruction, because the error function has only one dimension, some blurring occurs during the reconstruction process, such as... Figure 4 As shown.

[0103] Experiments show that the method in this embodiment achieves state-of-the-art (SOTA) performance in both inference speed and reconstruction quality compared to domestic and international research methods. Furthermore, by introducing IMU pre-integration to align the world coordinate axes and gravitational acceleration, we can calculate the absolute scale of the reconstructed scene. In contrast, previous neural radiation field SLAM reconstruction methods could not determine the ratio of the reconstructed scene to the real scene.

[0104] In summary, the method of the present invention has at least the following advantages and beneficial effects compared with the prior art:

[0105] (1) This invention proposes a method that supports monocular or monocular-inertial tightly coupled pose estimation and tracking, which combines computer graphics and neural radiation fields to enable rapid localization and dense mapping, thus solving the problem that sparse tracking in traditional rule-based SLAM algorithms cannot achieve dense 3D reconstruction.

[0106] (2) This invention minimizes IMU measurement noise by combining Butterworth filter and dynamic noise propagation matrix, making the robot system more robust in high-speed and low-computational-resource conditions.

[0107] (3) This invention proposes a loop detection module based on rotation angle detection to provide the most accurate global pose for the neural radiation field quickly.

[0108] This embodiment also provides a localization and mapping device based on neural radiation fields, including:

[0109] At least one processor;

[0110] At least one memory for storing at least one program;

[0111] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.

[0112] This embodiment of the localization and mapping device based on neural radiation fields can execute the localization and mapping method based on neural radiation fields provided in the method embodiment of the present invention. It can execute any combination of the implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0113] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0114] This embodiment also provides a storage medium storing instructions or programs that can execute the localization and mapping method based on neural radiation fields provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0115] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0116] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0117] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0119] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0120] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0122] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0123] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for localization mapping based on neural radiance fields, characterized in that, The method comprises the following steps: obtaining image data, wherein the image data is RGB image data or grayscale image data; constructing an image pyramid for a current image according to the image data, and performing positive and negative optical flow alignment to obtain feature points of the current image; screening the image data according to a preset condition to obtain key frames; detecting and determining whether the key frames are accumulated to a preset number, and performing SFM pose optimization on all the key frames to obtain an initial pose of each key frame; performing triangulation processing on the feature points according to the optimized key frame pose, and storing the processed feature points as global map points; for each key frame, constructing a visual constraint using the global map points in its continuous co-visible key frames, and performing Ceres back-end optimization according to the visual constraint; after the Ceres back-end optimization, adding the image and the pose into a bag-of-words model, and judging whether to perform loop detection on the current image, and if not, directly outputting the pose; performing coordinate system correction on the output pose, inputting the corrected pose and the image data into a trained neural radiance field, and reconstructing a corresponding three-dimensional model; the neural radiance field reconstructs the three-dimensional model by the following method: obtaining an image and a pose corresponding to the image, and constructing a light ray with the image pose as the center; wherein for each pixel, a plurality of three-dimensional points are sampled along the light ray; in the process of sampling the three-dimensional points, a occupancy grid is maintained, and the occupancy grid is used to identify the occupancy of the three-dimensional points in each region; when the light ray passes through the grid, the transmittance of the light ray is calculated, and if the transmittance is less than a threshold, the light ray stops step sampling; performing multi-scale segmentation on the three-dimensional space to obtain voxels to which the three-dimensional points belong, mapping the voxel serial numbers to which the three-dimensional points belong to a hash feature vector table to obtain corresponding feature vectors, and splicing the multi-scale mapped vectors to obtain the feature vector of the three-dimensional point; inputting the feature vector of the three-dimensional point into an MLP network to obtain the color value and the occupancy of the three-dimensional point; rendering according to the obtained color value and the occupancy of the three-dimensional point to reconstruct a three-dimensional model of the entire scene.

2. The neural radiance fields based positioning mapping method of claim 1, wherein, The image data is obtained by a monocular camera.

3. The neural-radiance-field-based positioning mapping method according to claim 1 or 2, characterized in that, The positioning and mapping method further comprises the step of fusing inertial data: obtaining inertial data, wherein the inertial data comprises acceleration and angular acceleration collected by an inertial sensor; estimating the uncertainty of the current inertial sensor measurement state, and judging whether to perform data fusion according to the uncertainty; if it is determined to perform data fusion, jointly optimizing the pose by using the inertial data and the key frames to obtain an initial pose of each key frame. The Ceres back-end optimization according to the visual constraint comprises: constructing an inertial pre-integration constraint between each key frame, and using a covariance matrix to measure the weight of each constraint.

4. The neural radiance fields based positioning mapping method of claim 1, wherein, In the training process of the neural radiance field, an L2 loss function is used to calculate the loss.

5. The neural radiance fields based positioning mapping method of claim 1, wherein, The image and the pose are added to the bag-of-words model, and the average rotation angle in the detection window is used to calculate the number of frames between the image frame corresponding to the last successful loop and the current image frame, and whether to perform loop detection on the current image is judged according to the number of frames. It comprises:

6. A neural-radiance-field-based positioning mapping device, characterized by ​ at least one processor; at least one memory for storing at least one program; the at least one program, when executed by the at least one processor, causing the at least one processor to carry out the method according to any one of claims 1-5.

7. A computer readable storage medium having stored therein a program that is executable by a processor, characterized in that, a program, executable by the processor for performing the method according to any one of claims 1-5 when executed by the processor.

Images