Generalized radiation field characterization method based on index optimization in mine weak light environment

By adopting an index-optimized generalized radiation field characterization method in the low-light environment of the mine, a feature grid is generated using the data of multiple low-light scenes to construct normal light and dark fields, which solves the problem of traditional three-dimensional reconstruction in the low-light environment. Poor generalization and data acquisition safety hazards have achieved high-precision three-dimensional reconstruction and robot perception positioning.

CN119006704AInactive Publication Date: 2024-11-22CHINA UNIV OF MINING & TECH

Patent Information

Application Number
CN202411022863.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In low-light environments such as mines, traditional three-dimensional reconstruction methods are difficult to achieve ideal results. The NeRF method based on deep learning has problems such as reconstruction error, loss of details and poor generalization under low-light conditions, and there are safety hazards and signals in data acquisition. Stability, it is difficult to generalize to unknown scenarios.

Method used

Using the generalizable radiation field characterization method based on index optimization, by collecting RGB images and lidar data from multiple low-light scenes, geometric features and appearance feature grids are generated, normal light fields and dark fields are constructed, and feature grids are used for Training, optimize loss function to improve generalization and accuracy.

Benefits of technology

It realizes three-dimensional reconstruction with high robustness and high precision in low-light environments of mines, which can provide accurate information for the perception and positioning of mine robots, and improves its application capabilities in unknown scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006704A_ABST
    Figure CN119006704A_ABST
Patent Text Reader

Abstract

The invention discloses a generalizable radiation field characterization method based on index optimization in a mine weak light environment. The method comprises the following steps: acquiring and preprocessing required data, performing feature extraction on an image by using a two-dimensional CNN, obtaining a cost body through a distortion operation, and obtaining a feature grid for storing feature information through a three-dimensional CNN; a feature grid optimization module is introduced, a sparse point cloud is projected into a two-dimensional depth map, a dense depth map is obtained through depth completion, points with uncertain depth are obtained through indexing, features of the points in a feature grid are extracted and input into an optimization CNN to obtain optimized features, and the feature grid is updated to obtain an optimized feature grid. And constructing a normal illumination field model and a dark field model, sampling features in the feature grid, and inputting the features into a radiation field for training. And after training is completed, inputting data of a weak light scene in the mine for reasoning, and finally obtaining scene representation in the weak light scene. Accurate information is provided for sensing, positioning and navigation of the underground robot.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a generalizable radiation field characterization method based on index optimization in a weak-light environment of a mine, and belongs to the technical field of intelligent positioning of mines in a weak-light environment. Background Art

[0003] In recent years, with the rapid development of artificial intelligence and computer vision technology, the application of 3D reconstruction technology in various fields has made significant progress. As an emerging 3D reconstruction technology, NeRF (Neural Radiance Fields) has shown a wide range of application prospects by using deep learning models to generate high-precision 3D models from 2D images. However, 3D reconstruction in low-light environments such as mines faces many challenges.

[0004] The mine environment is characterized by insufficient light, narrow space, and complex structure. Traditional 3D reconstruction methods are difficult to achieve ideal results in this environment. The deep learning-based NeRF method is prone to reconstruction errors and detail loss under low-light conditions due to its high sensitivity to light and materials. In addition, the function expression learned by NeRF is only for a single scene, and there are problems such as poor generalization, slow training and inference speed. In addition, considering the safety hazards when collecting data in mines, as well as unstable signals and limited sensor performance, it is unrealistic to collect data from multiple low-light scenes in mines. Therefore, how to generalize high-robustness and high-precision 3D reconstruction to the low-light environment of mines has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a generalizable radiation field characterization method based on index optimization in a low-light environment in a mine. By modeling a generalizable radiation field, the features of multiple different low-light scenes are extracted and stored through a feature grid, and the features in the sampling grid are input into normal illumination fields and dark fields for training, so that it can be applied to unknown low-light scenes in mines.

[0006] In order to achieve the above object, the present invention provides a generalizable radiation field characterization method based on index optimization in a weak light environment of a mine, comprising the following steps:

[0007] S1: Collect RGB images of multiple indoor and outdoor low-light scenes and a low-light scene in a mine, and generate sparse point cloud data describing geometric feature information through LiDAR scanning data;

[0008] S2: preprocess the RGB image collected in step S1 to obtain the image pose;

[0009] S3: Geometric and appearance feature mesh generation;

[0010] S4: Geometry and appearance feature mesh optimization;

[0011] S5: construct normal illumination field and dark field;

[0012] S6: construct loss function;

[0013] S7: Normal illumination field reasoning.

[0014] In step S2, the collected RGB image and lidar scanning data are preprocessed. The specific steps are as follows:

[0015] The SfM algorithm is used to obtain the image pose. First, the correspondence between images is determined by matching feature points. These correspondences are used to estimate the motion trajectory of the camera, including the rotation and translation of the camera. Then, the iterative optimization method is used to continuously adjust the camera pose to minimize the error between the projection model and the actual feature point position. Finally, the camera pose of each image in the global coordinate system is obtained, that is, its three-dimensional spatial position coordinates and shooting direction.

[0016] The method for generating geometric features and appearance features meshes in step S3 includes the following specific steps:

[0017] S3.1: Extract 2D image features from a single input view using deep 2D CNNL;

[0018] S3.2: Convert the image features under the input view into a warped feature map through a uniform warping operation;

[0019] S3.3: construct the cost volume based on the distorted feature map;

[0020] S3.4: Train a deep 3D CNNM to transform the constructed image feature cost volume into a new C-channel feature grid S.

[0021] The geometric feature and appearance feature mesh optimization method proposed in step S4 has the following specific steps:

[0022] S4.1: Project the sparse point cloud obtained by the laser radar scanning onto a two-dimensional plane to obtain a depth map;

[0023] S4.2: Perform depth completion on the depth map to obtain a dense depth map;

[0024] S4.3: Find points with uncertain depth by indexing the dense depth map;

[0025] S4.4: By extracting the features corresponding to these depth-uncertain points in the feature grid S, input them into the two-dimensional optimized CNN

[0026] The optimized features are obtained, and the feature grid S is updated to obtain a more accurate feature grid S after optimization.

[0027] The method for constructing the normal illumination field and the dark field proposed in step S5 has the following specific steps:

[0028] S5.1: Construct normal light field and dark field;

[0029] S5.2: The position coordinate X of the sampling point is encoded to obtain the position feature, and the multi-layer perceptron is used to further encode and map the position feature. Then, the convolution layer is used to convolve it to simulate the uneven darkness. Finally, the adjustment weights of the light field and the dark field are obtained through the Sigmoid activation function.

[0030] The loss function of step S6 is constructed, and the loss function includes: a new MSE loss Metric loss Contrast loss Loss of color consistency Density Field Regularization Loss Dark Field Regularization Loss Feature grid optimization loss

[0031] Where: Metric loss Adjust the enhancement level;

[0032] Contrast loss Calculate the contrast loss between adjacent sampled pixels of the predicted normal light image and the true dark image;

[0033] Loss of color consistency adjusting the color of the predicted normal light image;

[0034] To improve the sparsity of tensor factor parameters, regularization losses are used for density field and dark field respectively. and

[0035] Introducing feature grid optimization loss

[0036] The normal illumination field inference method proposed in step S7 has the following specific steps:

[0037] The RGB images and lidar scanning data of a low-light scene in a mine collected for testing are input into the trained feature grid, and their accurate features are extracted and input into the trained normal light field and dark field to obtain the normal light representation and dark representation as well as the adjustment weights between them. The normal light image of the scene can be obtained by removing the influence of the dark field and the adjustment weights and rendering using the volume density and color values ​​of normal light.

[0038] Compared with the prior art, the present invention collects RGB image data and laser radar scanning data in multiple indoor and outdoor low-light scenes, and pre-processes the collected data to obtain the required information. Use deep two-dimensional CNNL to extract features from the image, obtain the cost body through distortion operation, and then obtain the feature grid that stores feature information through deep three-dimensional CNNM. Introduce a feature grid optimization module, first project the sparse point cloud into a two-dimensional depth map, obtain a dense depth map through depth completion, and then index to obtain points with uncertain depth. By extracting the features corresponding to these points in the feature grid, input them into the optimized CNNNN to obtain optimized features, and update the feature grid to obtain a more accurate feature grid after optimization. Construct a normal illumination and dark radiation field model, sample the features in the feature grid, and then input them into the radiation field for training. After the training is completed, input the data of the low-light scene in the mine for reasoning, and finally obtain the scene representation in the low-light scene. The present invention can provide accurate information for the perception, positioning and navigation of the robot in the mine. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0040] Figure 2 A schematic diagram of a feature grid generation and optimization process in an embodiment of the present invention;

[0041] Figure 3 This is a normal illumination field model diagram in an embodiment of the present invention;

[0042] Figure 4 This is a dark field model diagram in an embodiment of the present invention;

[0043] Figure 5 It is a volume density field model diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings.

[0045] like Figure 1 and Figure 2 As shown, a generalizable radiation field characterization method based on index optimization in a weak light environment of a mine includes the following steps:

[0046] S1: Collect RGB images of multiple indoor and outdoor low-light scenes and a low-light scene in a mine, and generate sparse point cloud data describing geometric feature information through LiDAR scanning data;

[0047] The data collection vehicle is run to collect videos V and laser Lidar scanning data L of multiple different low-light scenes indoors and outdoors. In addition, data of a low-light scene in a mine is collected in the same way. The video V of each scene is divided into 100 RGB images with a resolution of 1080×1920 at a certain frame rate; the laser Lidar signal scans the scene to generate sparse point cloud data describing geometric feature information. In addition, in order to ensure data consistency, the RGB image signal and the laser Lidar data signal need to be time synchronized. The specific time synchronization method is: Since GPS allows nanosecond-level time synchronization progress, before collecting data, the GPS sends a signal pulse to activate the time signals of the camera and the laser radar to align the GPS time signal together.

[0048] S2: preprocess the RGB image collected in step S1 to obtain the image pose;

[0049] The purpose of preprocessing in the present invention is to generate data that can be directly used later from the collected raw data. This embodiment uses the Structure from Motion (SfM) method to obtain the rough approximate pose to be optimized corresponding to each frame of the image, including the initial camera three-dimensional coordinates and orientation. This method extracts key point information from each frame of the image, matches between frames, and estimates the pose using the projection relationship constraints between multi-view continuous frames, and finally obtains the camera pose of each image in the global coordinate system, that is, its three-dimensional spatial position coordinates and shooting direction.

[0050] S3: Geometric and appearance feature mesh generation;

[0051] S3.1: Extract image features: Use deep two-dimensional CNNL to extract the two-dimensional appearance features of the input image, and convert the input image into Convert to 2D feature map

[0052] F i =L(I i )

[0053] Where: H i and W i is the height and width of the image;

[0054] C is the number of image feature channels.

[0055] S3.2: Warp feature map: Given the camera’s intrinsic parameters [K] and extrinsic parameters [R,t], the image features at the input view are transformed to the reference view through a uniform warp operation:

[0056]

[0057] in: is the matrix warp from view i to the reference view at depth z;

[0058] K is the internal parameter matrix;

[0059] R and t are the rotation and translation of the camera;

[0060] Each feature map F i You can warp to a reference view by:

[0061]

[0062] Among them: F i,z is the distorted feature map at depth z;

[0063] (u,v) represents a pixel position in the reference view, and (u,v,z) is parameterized using the Normalized Device Coordinates (NDC) on the reference view.

[0064] S3.3: Construct the cost volume P based on the warped feature map and calculate the cost using a variance-based metric. In particular, for each voxel centered at (u, v, z) in the cost volume P, its cost feature vector is calculated as:

[0065] P(u,v,z)=Var(F i,z (u,v)

[0066] Among them: Var calculates the variance between M views.

[0067] S3.4: Feature grid: Achieving high-quality rendering requires inferring more appearance-aware information from the cost volume P. Therefore, a deep 3D CNNM is trained to transform the constructed image feature cost volume into a new C-channel feature grid S, where the feature space is learned and discovered by the network itself. This process can be expressed as:

[0068] S=M(P)

[0069] 3DCNNM is a 3D UNet with downsampling and upsampling convolutional layers and skip connections that can efficiently infer and propagate scene appearance information to obtain a feature grid S that is predicted in an unsupervised manner and inferred in end-to-end training using volume rendering.

[0070] S4: Geometry and appearance feature mesh optimization;

[0071] The sparse point cloud obtained by the laser radar scanning is projected onto a two-dimensional plane to obtain a depth map. Then, the depth map is completed to obtain a dense depth map. By indexing the dense depth map, points with uncertain depth are found, and the features corresponding to these points with uncertain depth are input into the two-dimensional optimized CNNNN to update the feature grid, thereby obtaining a feature grid S with more accurate features and stronger generalization.

[0072] S5: construct normal illumination field and dark field;

[0073] S5.1: Construct normal illumination field: Figure 3 As shown, the volume density σ and color c grids are vectorized and matrix decomposed to efficiently represent the normal lighting scene, and the geometric grid This breaks down into:

[0074]

[0075] in: represents the density feature vector on the X-axis in the current vector-plane basis, Represents the density feature plane on the YZ axis in the current vector-plane basis;

[0076] Represents the density feature vector on the Y axis in the current vector-plane basis, Represents the density feature plane on the XZ axis in the current vector-plane basis;

[0077] Represents the density feature vector on the Z axis in the current vector-plane basis, Represents the density feature plane on the XY axis in the current vector-plane basis;

[0078] R σ Represents the number of decomposed plane basis combinations, where each set of plane basis contains three components, and each component is two mutually perpendicular characteristic planes;

[0079] r represents the serial number of the current plane basis combination;

[0080] Appearance Grid is broken down into:

[0081]

[0082] in: Represents the color feature vector on the X-axis in the current vector-plane basis, Represents the color feature plane on the YZ axis in the current vector-plane basis;

[0083] Represents the color feature vector on the Y axis in the current vector-plane basis, Represents the color feature plane on the XZ axis in the current vector-plane basis;

[0084] Represents the color feature vector on the Z axis in the current vector-plane basis, Represents the color feature plane on the XY axis in the current vector-plane basis;

[0085] S5.2: Constructing a dark field: Figure 4 As shown, for the dark field, a single-channel grid g is used Ω Directly represents the darkness without any other mapping function; the continuous grid-based low-light radiation field can be written as:

[0086]

[0087] in: and represents the trilinear interpolation features from three grids at position x;

[0088] The above three grids are vectorized and matrix decomposed to efficiently represent and separate low-light scenes, such as Figure 5 As shown, the geometric mesh This breaks down into:

[0089]

[0090] Similar dark grid This breaks down into:

[0091]

[0092] Appearance Grid is broken down into:

[0093]

[0094] like Figure 5 As shown, the separation representation of the weak light field is obtained, and the attributes of the sampling point of each light (including volume density, normal light color, and darkness) need to be regressed; for volume density, since it is independent of direction, it is obtained in the following way:

[0095]

[0096] ReLU is the name of the activation function. The darkness of the sampling point is considered to be independent of direction when modeling.

[0097]

[0098] The color of the sampling point is related to different viewing angles, so the following formula is used:

[0099]

[0100] Among them: Sigmoid represents the activation function;

[0101] E represents a multi-layer perceptron network;

[0102] PE(d) represents the direction encoding operation;

[0103] In this way, the sampling points can obtain their corresponding attribute values ​​from the separated weak light radiation field.

[0104] S5.3: Dark field and normal light field adjustment;

[0105] The dark field and the normal light field have been obtained, and then the adjustment weights of the simulated dark field and the normal light field are obtained from the spatial position. The position coordinate X of the sampling point is encoded, and the multi-layer perceptron is used to further encode and map the features. Since the dark lengths of the scene are uneven, the convolution layer is used to convolve the features of the local light to simulate the uneven darkness. Finally, the adjustment weight is obtained through the Sigmoid activation function. The formula is as follows:

[0106]

[0107] in To adjust the weight, Conv is the convolution layer, S represents the multi-layer perceptron, and PE(X) represents the position encoding.

[0108] S6: construct loss function;

[0109] Directly applying NeRF's MSE loss function in low-light scenes will lead to an imbalance in pixel weights. The contribution of dark pixels with smaller values ​​is minimal, while the contribution of bright pixels with larger values ​​is dominant. To correct this imbalance and ensure effective NeRF training under non-standard lighting conditions, an inverse curve adjustment is used:

[0110]

[0111] The new MSE loss is defined as:

[0112]

[0113] In order to adjust the enhancement level, a metric loss is introduced and applied to the predicted normal light image C nor (r). The metric loss involves a hyperparameter called e, which represents an inherent degree of enhancement. To calculate the metric loss, for each training batch of normal light images C nor (r) Perform a global average pooling operation and pool the normal light image C nor(r) Compare with the specified e:

[0114]

[0115] Then, the predicted normal light image C is calculated nor and the ground-truth dark image C low Contrast loss between adjacent sampled pixels:

[0116]

[0117] The contrast loss will maintain the normal light image C nor and the ground-truth dark image C low The structural similarity between them is also controlled by the parameter η. nor When the enhancement level is fixed, a higher η represents a normal light image C nor High contrast in the medium.

[0118] Photos taken in low-light conditions can easily lose their color information, and directly removing and adding dark fields may lead to color imbalance. In order to adjust the predicted normal-light image C nor Here, we assume that the normal light image C nor Following the gray world hypothesis:

[0119]

[0120] Where: (p,q)∈{(R,G),(G,B),(B,R)} represents any pair of color channels;

[0121] To improve the sparsity of tensor factor parameters, the standard L 1 Regularization. The following losses are used for the density field and the dark field respectively:

[0122]

[0123] During the feature mesh optimization process, noise pixels are obtained by rendering the light The noise pixel is represented by the corresponding true pixel color b gt Supervision:

[0124]

[0125] In summary, the total loss function consists of 7 parts, and the total loss of low-light scene rendering for training can be expressed as:

[0126]

[0127] Where: 1 to λ 5are 5 non-negative hyperparameters that balance the total loss weights.

[0128] S7: Normal illumination field reasoning.

[0129] The RGB images and lidar scanning data of a low-light scene in a mine collected for testing are input into the trained feature grid after the same preprocessing, and their accurate features are extracted and input into the trained normal light field and dark field to obtain the normal light representation and dark representation as well as the adjustment weights between them. The normal light image of the scene can be obtained by removing the influence of the dark field and the adjustment weights and rendering using the volume density and color values ​​of normal light.

Claims

1. A generalizable radiation field characterization method based on index optimization in a weak light environment of a mine, characterized in that: The following steps are involved: S1: Collect RGB images of multiple indoor and outdoor low-light scenes and a low-light scene in a mine, and generate sparse point cloud data describing geometric feature information through LiDAR scanning data; S2: preprocess the RGB image collected in step S1 to obtain the image pose; S3: Geometric and appearance feature mesh generation; S4: Geometry and appearance feature mesh optimization; S5: construct normal illumination field and dark field; S6: construct loss function; S7: Normal illumination field reasoning.

2. According to claim 1, a generalizable radiation field characterization method based on index optimization in a mine weak light environment is characterized in that: In step S2, the collected RGB image and lidar scanning data are preprocessed, specifically: The SfM algorithm is used to obtain image pose. First, the correspondence between images is determined by feature point matching. These correspondences are used to estimate the camera's motion trajectory, including the camera's rotation and translation. Then, the camera pose is continuously adjusted using an iterative optimization method to minimize the error between the projection model and the actual feature point position. Finally, the camera pose of each image in the global coordinate system is obtained, that is, its three-dimensional spatial position coordinates and shooting direction.

3. According to claim 2, a generalizable radiation field characterization method based on index optimization in a mine weak light environment is characterized in that: The method for generating geometric features and appearance features meshes in step S3 includes the following specific steps: S3.1: Extract 2D image features from a single input view using deep 2D CNNL; S3.2: converting the features of the two-dimensional image features under the input view into a warped feature map through a uniform warping operation; S3.3: construct the cost volume based on the distorted feature map; S3.4: Train a deep 3D CNNM to transform the constructed image feature cost volume into a new C-channel feature grid S.

4. According to claim 3, a generalizable radiation field characterization method based on index optimization in a mine weak light environment is characterized in that: The geometric feature and appearance feature mesh optimization method proposed in step S4 has the following specific steps: S4.1: Project the sparse point cloud obtained by the laser radar scanning onto a two-dimensional plane to obtain a depth map; S4.2: Perform depth completion on the depth map to obtain a dense depth map; S4.3: Find points with uncertain depth by indexing the dense depth map; S4.4: By extracting the features corresponding to the points with uncertain depth in the feature grid, inputting them into the two-dimensional optimized CNNN to obtain the optimized features, the C channel feature grid S is updated to obtain a more accurate feature grid S after optimization.

5. According to claim 3, a generalizable radiation field characterization method based on index optimization in a mine weak light environment is characterized in that: The method for constructing the normal illumination field and the dark field proposed in step S5 has the following specific steps: S5.1: Construct normal light field and dark field; S5.2: The position coordinate X of the sampling point is encoded to obtain the position feature, and the multi-layer perceptron is used to further encode and map the position feature. Then, the convolution layer is used to convolve it to simulate the uneven darkness. Finally, the adjustment weights of the light field and the dark field are obtained through the Sigmoid activation function.

6. According to claim 3, a generalizable radiation field characterization method based on index optimization in a mine weak light environment is characterized in that: The loss function of step S6 is constructed, and the loss function includes: a new MSE loss Metric loss Contrast loss Loss of color consistency Density Field Regularization Loss Dark Field Regularization Loss Feature grid optimization loss Where: Metric loss Adjust the enhancement level; Contrast loss Calculate the contrast loss between adjacent sampled pixels of the predicted normal light image and the true dark image; Loss of color consistency adjusting the color of the predicted normal light image; To improve the sparsity of tensor factor parameters, regularization losses are used for density field and dark field respectively. and Introducing feature grid optimization loss 7. According to claim 3, a generalizable radiation field characterization method based on index optimization in a mine weak light environment is characterized in that: The normal illumination field inference method proposed in step S7 has the following specific steps: The RGB images and lidar scanning data of a low-light scene in a mine collected for testing are input into the trained feature grid, and their accurate features are extracted and input into the trained normal light field and dark field to obtain the normal light representation and dark representation as well as the adjustment weights between them. By removing the influence of the dark field and the adjustment weights, and using the volume density and color value of normal light for rendering, the normal light image of the scene can be obtained.

Citation Information

Patent Citations

  • Semantic segmentation method for low-illumination scene

    CN110458844A

  • Infrared and RGB cross-modal feature point matching method

    CN114120013A

  • Hidden field-based unsupervised low-illumination enhancement method and device, and storage medium

    CN116452725A

  • Mine image three-dimensional reconstruction method based on neural radiation field

    CN118279490A

Cited By

  • Generalized radiation field characterization method based on index optimization in mine weak light environment

    CN120107316A