Real-time mapping method and system based on fine texture map of neural radiation field

Abstract

The application discloses a real-time mapping method and system of a fine texture map based on a neural radiation field, and the method comprises the following steps: firstly, inputting scene depth information through a laser radar, inputting scene texture information through a camera, and obtaining device state quantities to construct an input data set; inputting the input data set into a SLAM algorithm after point cloud de-distortion; obtaining target data after processing the input data set; then, inserting key frames into the target data through an optical flow tracking judgment method, and storing part of light rays sampled by sparse depth pictures into a key frame database; finally, performing targeted refinement on color through an external parameter refinement and inverse affine texture reconstruction strategy, and performing implicit neural mapping to obtain a map with fine texture, so that the texture definition of long-time large-scale measurement based on external parameter optimization is improved, the influence of light change on real-time reconstruction of a target object is avoided as much as possible, and the application can be widely applied to the technical field of map construction.

Description

Technical Field

[0001] This invention relates to the field of map construction technology, and in particular to a real-time mapping method and system for fine-texture maps based on neural radiation fields. Background Technology

[0002] Real-time construction of detailed maps has applications in robotics, digital twins, and AR / VR. Dense mapping refers to using sensors (such as cameras and LiDAR) to acquire a large amount of observational data of a scene. By fusing and registering this data, a high-resolution, continuous 3D map can be created to represent the scene's geometry and surface features. In contrast to sparse mapping, dense mapping aims to acquire richer and more detailed geometric and textural information about the scene. Therefore, dense mapping is commonly used in tasks such as scene exploration, virtual reality, and robotics. Classical methods utilize explicit scene representation methods, such as occupancy grids, symbolic distance fields, and meshes, to construct a global map. These methods have been proven to be applicable in real-time in industrial environments, such as using drones to explore factories and build maps. However, explicit scene representation still has two major problems: first, it is difficult to balance memory consumption and mapping accuracy, requiring redundant data structures to obtain detailed maps; second, it is difficult to handle unseen areas, where maps are almost impossible to form when sensors cannot obtain spatial observation data. Intuitively, this leads to increased labor consumption (humans will take longer to model scenes of the same size by hand) and wasted energy (robots will take longer to complete tasks).

[0003] In recent years, with the emergence of neural radiation fields, implicit representation methods have become increasingly popular. These methods use implicit features to describe the geometry and appearance of a scene, utilizing neural networks to decode density fields, symbolic distance fields, occupancy fields, and color information, and then synthesizing images through rendering for self-supervision. This approach leverages the powerful fitting and predictive capabilities of neural networks to overcome the problems inherent in explicit representations. These methods using neural implicit representations can generate high-fidelity reconstructed maps with less memory, predict unseen regions through neural networks, and synthesize new views.

[0004] Using implicit neural expressions for dense mapping is a forward-looking and practical research direction. Its background and significance involve not only the fields of robotics and bionics, but also multiple fields such as disaster response, rescue, exploration and military, and it has important research value and application prospects.

[0005] The existing real-time implicit graphing methods mainly have the following problems:

[0006] Most existing methods lack robust pose estimation, resulting in high requirements for data acquisition or only achieving good results in simulation environments.

[0007] Existing methods have not been able to simultaneously incorporate radar and camera measurements to introduce texture reconstruction features for implicit mapping of large outdoor scenes.

[0008] Existing real-time reconstruction methods rarely take into account lighting changes, and in implicit mapping, existing exposure modeling methods are difficult to constrain feature vertex colors, meaning they can only model exposure in the rendered image. Summary of the Invention

[0009] In view of this, embodiments of the present invention provide a real-time mapping method and system for fine texture maps based on neural radiation fields, which can improve the texture clarity of long-term, large-scale measurements.

[0010] One aspect of this invention provides a real-time mapping method for fine-texture maps based on neural radiation fields, comprising:

[0011] The input dataset is constructed by inputting scene depth information through LiDAR, scene texture information through camera, and acquiring device state data. The input dataset is then processed by point cloud distortion correction and input into the SLAM algorithm. After processing, the input dataset yields target data, which includes images, point cloud data, and pose extrinsic parameter data.

[0012] The target data is inserted using an optical flow tracing method, and the light rays sampled from some sparse depth images are stored in the keyframe database.

[0013] After targeted refinement of colors using extrinsic parameter refinement and inverse affine texture reconstruction strategies, implicit neural mapping is performed to obtain a map with fine texture.

[0014] Optionally, the inverse affine texture reconstruction strategy is specifically as follows:

[0015] An inverse affine transformation is performed on the observed ground truth image to obtain an image free from the effects of illumination.

[0016] Transform the prior point cloud to the current camera coordinate system and project it into a sparse map. Use the location where the point cloud is projected as the ground truth after removing the influence of illumination.

[0017] After inverse affine texture reconstruction, exposure compensation is performed during the optimization process. The luminance loss between the affine transformed color and the real color is adjusted, and the exposure module is optimized and adjusted to complete the entire exposure optimization.

[0018] Optionally, the expression for the process of performing an inverse affine transformation on the observed ground truth image to obtain an image free from the influence of illumination is:

[0019]

[0020] in, This is the image observed by the camera in the current frame. The image represents the image after removing the effects of lighting, and (A,b) is the decoded affine transformation.

[0021] Optionally, the expression for the process of transforming the prior point cloud to the current camera coordinate system and projecting it into a sparse map, using the location where the point cloud is projected as the ground truth after removing the influence of illumination, is as follows:

[0022]

[0023] in, It is the inverse affine photometric loss, used to constrain exposure features and the exposure decoder; It is a sparse RGB image projected from the prior point cloud; P represents the prior point cloud projected onto the current camera frame.

[0024] Optionally, the implicit neural mapping process specifically includes:

[0025] Coordinate encoding, multi-resolution hash grid encoding, and multilayer perceptual encoding are constructed and input into a geometric decoder to obtain high-frequency features; the high-frequency features are used to describe the geometric details and high-fidelity appearance of the scene.

[0026] The coordinate encoding, multi-resolution hash grid encoding, and multilayer perceptual encoding are combined with the high-frequency features to obtain the final SDF value, LiDAR intensity, and feature vector, which are then input into the color decoder to obtain color information.

[0027] The color information is subjected to an affine transformation to learn how to map image-independent radiation fields to image-related radiation fields in each image, thus completing the implicit neural mapping.

[0028] Optionally, the method further includes a process of optimizing the constructed map, which includes:

[0029] Construct a global voxel counter and use an exponential function to adaptively compute learnable parameters;

[0030] Volume density is used to render the color and depth of each ray, thus completing the conversion of the directional distance field into volume density.

[0031] Optionally, the method further includes a step of constructing a total loss function, which includes:

[0032] The first loss function, which includes free space loss and SDF loss, is applied to the supervised harnesses through point-by-point supervision.

[0033] Construct a rendering loss function, which includes RGB loss and depth loss;

[0034] Construct the intensity loss function;

[0035] The total loss function is constructed based on the first loss function, the rendering loss function, and the intensity loss function.

[0036] Another aspect of this invention provides a real-time mapping system for fine-texture maps based on neural radiation fields, comprising:

[0037] The first module is used to construct an input dataset by inputting scene depth information through LiDAR, scene texture information through a camera, and acquiring device state parameters; the input dataset is then processed by point cloud distortion correction and input into the SLAM algorithm; the input dataset is processed to obtain target data, which includes images, point cloud data, and pose extrinsic parameter data;

[0038] The second module is used to insert keyframes into the target data using an optical flow tracing method, and to store the light rays sampled from some sparse depth images into the keyframe database.

[0039] The third module is used to refine the colors in a targeted manner through extrinsic refinement and inverse affine texture reconstruction strategies, and then perform implicit neural mapping to obtain a map with fine texture.

[0040] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0041] The memory is used to store programs;

[0042] The processor executes the program to implement the method described above.

[0043] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0044] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0045] The embodiments of the present invention first construct an input dataset by inputting scene depth information through LiDAR, scene texture information through a camera, and acquiring device state data. The input dataset is then processed by point cloud distortion correction and input into a SLAM algorithm. After processing, target data is obtained. Then, keyframes are inserted into the target data using an optical flow tracing method, and light rays sampled from some sparse depth images are stored in a keyframe database. Finally, after targeted color refinement using extrinsic refinement and inverse affine texture reconstruction strategies, implicit neural mapping is performed to obtain a map with fine texture. This invention can improve the texture clarity of long-term, large-scale measurements based on extrinsic optimization and minimize the impact of illumination changes in real-time reconstruction of target objects. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating the overall implementation steps of this invention.

[0048] Figure 2 This is a schematic diagram of a neural network structure provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] To address the problems existing in the prior art, one aspect of the present invention provides a real-time mapping method for fine texture maps based on neural radiation fields, comprising:

[0051] The input dataset is constructed by inputting scene depth information through LiDAR, scene texture information through camera, and acquiring device state data. The input dataset is then processed by point cloud distortion correction and input into the SLAM algorithm. After processing, the input dataset yields target data, which includes images, point cloud data, and pose extrinsic parameter data.

[0052] The target data is inserted using an optical flow tracing method, and the light rays sampled from some sparse depth images are stored in the keyframe database.

[0053] After targeted refinement of colors using extrinsic parameter refinement and inverse affine texture reconstruction strategies, implicit neural mapping is performed to obtain a map with fine texture.

[0054] Optionally, the inverse affine texture reconstruction strategy is specifically as follows:

[0055] An inverse affine transformation is performed on the observed ground truth image to obtain an image free from the effects of illumination.

[0056] Transform the prior point cloud to the current camera coordinate system and project it into a sparse map. Use the location where the point cloud is projected as the ground truth after removing the influence of illumination.

[0057] After inverse affine texture reconstruction, exposure compensation is performed during the optimization process. The luminance loss between the affine transformed color and the real color is adjusted, and the exposure module is optimized and adjusted to complete the entire exposure optimization.

[0058] Optionally, the expression for the process of performing an inverse affine transformation on the observed ground truth image to obtain an image free from the influence of illumination is:

[0059]

[0060] in, This is the image observed by the camera in the current frame. The image represents the image after removing the effects of lighting, and (A,b) is the decoded affine transformation.

[0061] Optionally, the expression for the process of transforming the prior point cloud to the current camera coordinate system and projecting it into a sparse map, using the location where the point cloud is projected as the ground truth after removing the influence of illumination, is as follows:

[0062]

[0063] in, It is the inverse affine photometric loss, used to constrain exposure features and the exposure decoder; It is a sparse RGB image projected from the prior point cloud; P represents the prior point cloud projected onto the current camera frame.

[0064] Optionally, the implicit neural mapping process specifically includes:

[0065] Coordinate encoding, multi-resolution hash grid encoding, and multilayer perceptual encoding are constructed and input into a geometric decoder to obtain high-frequency features; the high-frequency features are used to describe the geometric details and high-fidelity appearance of the scene.

[0066] The coordinate encoding, multi-resolution hash grid encoding, and multilayer perceptual encoding are combined with the high-frequency features to obtain the final SDF value, LiDAR intensity, and feature vector, which are then input into the color decoder to obtain color information.

[0067] The color information is subjected to an affine transformation to learn how to map image-independent radiation fields to image-related radiation fields in each image, thus completing the implicit neural mapping.

[0068] Optionally, the method further includes a process of optimizing the constructed map, which includes:

[0069] Construct a global voxel counter and use an exponential function to adaptively compute learnable parameters;

[0070] Volume density is used to render the color and depth of each ray, thus completing the conversion of the directional distance field into volume density.

[0071] Optionally, the method further includes a step of constructing a total loss function, which includes:

[0072] The first loss function, which includes free space loss and SDF loss, is applied to the supervised harnesses through point-by-point supervision.

[0073] Construct a rendering loss function, which includes RGB loss and depth loss;

[0074] Construct the intensity loss function;

[0075] The total loss function is constructed based on the first loss function, the rendering loss function, and the intensity loss function.

[0076] Another aspect of this invention provides a real-time mapping system for fine-texture maps based on neural radiation fields, comprising:

[0077] The first module is used to construct an input dataset by inputting scene depth information through LiDAR, scene texture information through a camera, and acquiring device state parameters; the input dataset is then processed by point cloud distortion correction and input into the SLAM algorithm; the input dataset is processed to obtain target data, which includes images, point cloud data, and pose extrinsic parameter data;

[0078] The second module is used to insert keyframes into the target data using an optical flow tracing method, and to store the light rays sampled from some sparse depth images into the keyframe database.

[0079] The third module is used to refine the colors in a targeted manner through extrinsic refinement and inverse affine texture reconstruction strategies, and then perform implicit neural mapping to obtain a map with fine texture.

[0080] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0081] The memory is used to store programs;

[0082] The processor executes the program to implement the method described above.

[0083] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0084] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0085] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0086] like Figure 1 As shown, the overall framework of the real-time mapping method for fine texture maps based on neural radiation fields of the present invention begins with an input module. In this embodiment, mapping can be performed in different indoor and outdoor scenes. First, scene depth information is input via LiDAR, scene texture information is input via camera, and device state data is captured by IMU. The point cloud data is then distortion-corrected and input into a traditional SLAM algorithm. The processed data includes images, point cloud data, and pose data. External reference These data are used to determine keyframe insertion through optical flow tracing. Light samples from some sparse depth images are stored in the keyframe database. After targeted color refinement through extrinsic parameter thinning and inverse affine texture reconstruction strategies, implicit neural mapping is finally performed to obtain a map with fine texture.

[0087] like Figure 2 As shown, this embodiment employs a multi-layer hash sparse encoder to memorize the geometric and color information of the scene, ensuring both speed and predictability. To achieve smoother results, an MLP (Multi-Level Processing) is used as a low-frequency encoder, and its result is summed with the geometric decoding result to decode the color. Coordinate encoding is added to enhance the network's perception of geometric location and improve predictability. Here, the signed distance field decoded from the geometric model is used to represent explicit geometry, and is used for volume rendering with intensity and the color c after affine transformation. Supervised training is performed by subtracting the distance field from the ground truth image and the point cloud depth. The exposure compensation part utilizes the exposure features to decode a 12-dimensional affine transformation matrix as a model for image color transformation. The meanings of the vectors in the diagram are labeled in the upper right corner.

[0088] The details of each implementation step of the present invention are described below:

[0089] 1. Data Input:

[0090] This embodiment primarily uses a camera, radar, and IMU as inputs. It also utilizes IMU data for point cloud motion distortion correction before inputting it into a traditional radar / visual SLAM algorithm. The processed data includes images, point cloud data, and pose data. External reference These data. Then we use extrinsic parameters to transform the data from the radar coordinate system to the camera coordinate system, projecting it into sparse depth maps and intensity maps as supervisory data. We can also use device pose data to transform point cloud and other data into the global coordinate system for manipulation.

[0091] 2. Light sampling:

[0092] This embodiment employs a keyframe strategy using light rays as storage units to avoid the forgetting problem inherent in implicit neural representations and hash sparse coding. It also enhances and refines the color and geometric information of historical observation areas. This storage format is more space-efficient and better aligns with the principles of implicit representation optimization. We sample 2% of the image based on the projected depth map and store the observations according to the light ray data format for subsequent optimization. This includes {o,d,c} gt D gt} represent the starting point of the light ray, the direction of the light ray, the observed true color, and the observed true depth, respectively.

[0093] Meanwhile, unlike existing methods that use data at a fixed frequency as keyframes, we utilize optical flow tracing results as the criterion for determining whether keyframes need to be inserted, thus obtaining a more reasonable keyframe storage strategy. As for the strategy of selecting points on rays for rendering, we employ a combination of uniform sampling and depth-guided sampling.

[0094] 3. Latent neural expression:

[0095] 1) Combined coding:

[0096] To ensure predictive power and real-time performance when constructing highly detailed maps, this embodiment employs a joint encoding method using coordinate encoding, multi-resolution hash grids, and MLP. The implicit neural representation structure of this invention is as follows: Figure 2 As shown. First, coordinate encoding encodes the coordinates of the sampling points into γ(x) to enhance prediction capabilities and obtain a more complete map. Simultaneously, this embodiment utilizes multi-resolution hash grid encoding to promote convergence and memory efficiency, and obtains geometric features V(x) at multiple levels. Then, this embodiment inputs these features into the geometric decoder to obtain high-frequency information:

[0097] (s h i h h h ) = Dg (γ(x), V(x); θ)

[0098] Among them, s h It is the SDF value, i h It is the LiDAR intensity, h h These are geometric features. These high-frequency geometric features describe the geometric details and high-fidelity appearance of the scene. However, due to sparse point clouds and measurement noise, they often exhibit local surface roughness during reconstruction. Low-frequency features obtained directly from MLP encoding are often over-smoothed; therefore, we combine them with high-frequency features to obtain the final SDF value s, LiDAR intensity i, and feature vector h:

[0099] (s, i, h) = (s h i h h h +D l (x;φ))

[0100] The final geometric features are input into the color decoder to obtain the color:

[0101] c′=D c (h;ω)

[0102] 2) Exposure compensation:

[0103] In outdoor environments, lighting conditions can introduce photometric errors when capturing images using automatic white balance and automatic exposure. Intuitively, this can lead to texture loss because the ground color is close to white during exposure. To address this issue, this embodiment uses exposure compensation, which learns to map the image-independent radiation field to the image-dependent radiation field in each image. This embodiment performs an affine transformation on the color predicted by c′, where the affine transformation is derived from the embedding encoding ε of each image. k ∈R F →R 12x1 Decode a 12-dimensional feature vector.

[0104] 4. External parameter refinement:

[0105] Inaccurate extrinsic parameter estimation can lead to misalignment between color and geometry. Since the input in this embodiment is a sequence, this can cause texture blurring during continuous optimization, especially with rich textures or large scales. This embodiment first optimizes the extrinsic parameters while keeping the network parameters constant, and then performs implicit neural mapping. To mitigate the performance degradation caused by long convergence times, this embodiment uses only photometric loss during optimization. The details of this loss will be discussed later.

[0106] 5. Inverse affine texture reconstruction:

[0107] like Figure 1 As shown, this embodiment models the impact of factors such as lighting on the camera through affine transformation of RGB. The color information output by the color decoder needs to undergo exposure affine transformation and then be color trained by photometric loss with the colors of the real image (c′→c). Using this structure, we can constrain the neural radiation field to be independent of map lighting. However, the map texture is still directly obtained by decoding the high-dimensional features of the hash grid vertices.

[0108] However, the above lighting modeling not only needs to ensure that c is approximately the image captured by the camera, but also needs to ensure that c′ is free from the influence of lighting. Learning exposure features through just one MLP is challenging and cannot be iterated many times. Moreover, since real-time mapping is a sequential input, it lacks sufficient prior information. Therefore, this embodiment proposes inverse affine texture reconstruction to obtain and utilize prior information.

[0109] First, this embodiment acquires a colored prior point cloud during the algorithm initialization phase. This color style is then used as guiding information for texture reconstruction and supplemented when the visible prior point cloud is limited. This embodiment utilizes observed ground truth images... Figure 2 The inverse affine transformation in the image is performed to obtain an image free from the effects of lighting.

[0110]

[0111] in, This is the image observed by the camera in the current frame. This represents the image after removing the effects of illumination, and (A,b) is the affine transformation amount decoded by the exposure module. Now, the prior point cloud is directly transformed to the current camera coordinate system and projected into a sparse map. The location where the point cloud is projected is taken as the ground truth after removing the effects of illumination.

[0112]

[0113] in, It is the inverse affine photometric loss, which is used to constrain the exposure characteristics and exposure decoder, thereby ensuring that c′ removes the influence of illumination. It is a sparse RGB projection of the prior point cloud.

[0114] After the inverse affine texture reconstruction, this embodiment will perform exposure compensation during the optimization process, that is, to reduce the luminance of the color c after the affine transformation and the real color. At the same time, the exposure module will be optimized and adjusted during this process to complete the entire exposure optimization.

[0115] 6. Optimization process:

[0116] 1) SDF-based volume rendering:

[0117] To perform volume rendering using SDF values, most NeRF graphing methods directly convert the SDF values ​​s into weights w:

[0118]

[0119] Here, σ is the sigmoid function, and tr is the cutoff distance. However, methods that directly convert SDF to weights lack occlusion awareness. When there are two surfaces on a ray, the weights of both surfaces are the same. Therefore, these methods simply mask the weights of the regions behind the surfaces, leading to a decrease in prediction accuracy.

[0120] Some methods propose converting SDF (Signed Distance Field) into volume density and then using NeRF rendering formulas. The method of this invention also directly converts SDF into volume density:

[0121]

[0122] Here, β is a learnable parameter used to control the sharpness of surface boundaries. Given the large scale of the scene, using a single β to describe the entire scene is likely to lead to suboptimal results. Therefore, this invention employs adaptive calculation of β. Specifically, this embodiment maintains a global voxel counter and adaptively calculates β using an exponential function. The volume density is then used to render the color and depth of each ray:

[0123]

[0124]

[0125] Where N is the number of samples taken along each ray.

[0126] 2) Loss function:

[0127] To obtain accurate geometry, this embodiment uses a point-by-point supervised loss function for the supervised wire harness R, including free space loss. and SDF loss

[0128]

[0129]

[0130] in, It is a set of points on ray r located between the surface cutoff region measured by the camera and the depth sensor. It is a set of points within the truncated region. s p It is the predicted SDF. This is the difference between the distance to point p on ray r and the measured depth of that ray. This embodiment uses the same loss for all points in the truncated region, but it distinguishes the middle of the truncated region. With the tail of the truncated area The importance of points closer to the surface. Formally, this embodiment will... Defined as The set of points. And define Then define:

[0131]

[0132] This embodiment also incorporates a rendering loss function, including RGB loss. and depth loss

[0133] Here Indicates the depth of rendering and input. These represent the rendered and input colors, respectively.

[0134] The intensity of a lidar signal reflects the intensity of the pulse echo at the measurement point, and similar surface materials have similar intensities. Therefore, this invention proposes an intensity loss function.

[0135]

[0136] Where, ζ r and These represent the rendering intensity and the input intensity, respectively. The final loss function is a weighted sum of these loss functions.

[0137] The weighted sum of the loss functions. The weights are determined by α. sdf-m ,α sdf-t ,α fs ,α d ,α rgb and α intensity Decide.

[0138]

[0139] In summary, the present invention has the following advantages:

[0140] 1. Wide applicability: The system uses traditional SLAM as the positioning module, plus radar and camera sensor configuration, and has inverse affine exposure constraints to counteract exposure changes, which makes our data requirements lower and enables relatively robust positioning and mapping both indoors and outdoors.

[0141] 2. High speed and low computation: It utilizes a multi-layer hash grid and traditional SLAM, achieving high speed compared to existing methods.

[0142] 3. Clearer map texture: By using extrinsic parameter optimization and exposure modeling, we can ensure the map has fine texture as much as possible during real-time mapping.

[0143] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0144] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0145] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0147] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0148] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0149] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0150] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0151] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A real-time mapping method for fine-texture maps based on neural radiation fields, characterized in that, include: The input dataset is constructed by inputting scene depth information through LiDAR, scene texture information through camera, and acquiring device state parameters. The input dataset is processed by point cloud distortion correction and then input into the SLAM algorithm; the input dataset is processed to obtain target data, which includes images, point cloud data, and pose extrinsic data. The target data is inserted using an optical flow tracing method, and the light rays sampled from some sparse depth images are stored in the keyframe database. After the colors are refined in a targeted manner using extrinsic parameter refinement and inverse affine texture reconstruction strategies, implicit neural mapping is performed to obtain a map with fine texture. The inverse affine texture reconstruction strategy is as follows: An inverse affine transformation is performed on the observed ground truth image to obtain an image free from the effects of illumination. Transform the prior point cloud to the current camera coordinate system and project it into a sparse map. Use the location where the point cloud is projected as the ground truth after removing the influence of illumination. After inverse affine texture reconstruction, exposure compensation is performed during the optimization process. The luminance loss between the affine transformed color and the real color is adjusted, and the exposure module is optimized and adjusted to complete the entire exposure optimization. The implicit neural mapping process specifically includes: Coordinate encoding, multi-resolution hash grid encoding, and multilayer perceptual encoding are constructed and input into a geometric decoder to obtain high-frequency features; the high-frequency features are used to describe the geometric details and high-fidelity appearance of the scene. The coordinate encoding, multi-resolution hash grid encoding, and multilayer perceptual encoding are combined with the high-frequency features to obtain the final SDF value, LiDAR intensity, and feature vector, which are then input into the color decoder to obtain color information. The color information is subjected to an affine transformation to learn how to map image-independent radiation fields to image-related radiation fields in each image, thus completing the implicit neural mapping.

2. The real-time mapping method for fine-texture maps based on neural radiation fields according to claim 1, characterized in that, The expression for the process of performing an inverse affine transformation on the observed ground truth image to obtain an image free from the effects of illumination is as follows: in, This is the image observed by the camera in the current frame. This represents an image that has had the effects of lighting removed. This is the decoded affine transformation quantity.

3. The real-time mapping method for fine-texture maps based on neural radiation fields according to claim 2, characterized in that, The expression for the process of transforming the prior point cloud to the current camera coordinate system, projecting it into a sparse map, and using the location where the point cloud is projected as the ground truth after removing the influence of illumination is as follows: in, It is the inverse affine photometric loss, used to constrain exposure features and the exposure decoder; It is a sparse RGB projection of the prior point cloud; This represents the prior point cloud projected onto the current camera frame.

4. The real-time mapping method for fine-texture maps based on neural radiation fields according to claim 1, characterized in that, The method also includes a process for optimizing the constructed map, which includes: Construct a global voxel counter and use an exponential function to adaptively compute learnable parameters; Volume density is used to render the color and depth of each ray, thus completing the conversion of the directional distance field into volume density.

5. The real-time mapping method for fine-texture maps based on neural radiation fields according to claim 4, characterized in that, The method further includes a step of constructing a total loss function, which includes: The first loss function, which includes free space loss and SDF loss, is applied to the supervised harnesses through point-by-point supervision. Construct a rendering loss function, which includes RGB loss and depth loss; Construct the intensity loss function; The total loss function is constructed based on the first loss function, the rendering loss function, and the intensity loss function.

6. A system for implementing a real-time mapping method for fine-texture maps based on neural radiation fields as described in any one of claims 1-5, characterized in that, include: The first module is used to construct the input dataset by inputting scene depth information through LiDAR, scene texture information through camera, and acquiring device state parameters. The input dataset is processed by point cloud distortion correction and then input into the SLAM algorithm; the input dataset is processed to obtain target data, which includes images, point cloud data, and pose extrinsic data. The second module is used to insert keyframes into the target data using an optical flow tracing method, and to store the light rays sampled from some sparse depth images into the keyframe database. The third module is used to refine the colors in a targeted manner through extrinsic refinement and inverse affine texture reconstruction strategies, and then perform implicit neural mapping to obtain a map with fine texture.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images