Three-dimensional point cloud reconstruction method based on multi-exposure image fusion
Through multi-exposure image fusion technology, combined with detail and color enhancement modules, the problem of poor reconstruction of three-dimensional point clouds in weak texture areas and noise surfaces is solved, and three-dimensional point cloud reconstruction with higher accuracy and visual perception effects is achieved.
Patent Information
- Application Number
- CN202411484335.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2025-02-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, when dealing with weak texture areas and noisy surfaces, it is difficult to extract sufficient feature points, resulting in poor reconstruction effect of three-dimensional point clouds.
Using a three-dimensional point cloud reconstruction method with multi-exposure image fusion, the design of the detail enhancement module and the color enhancement module, combined with the global-local adversarial learning network, generate images with richer details and more realistic brightness and color information, thereby improving the three-dimensional Reconstruction accuracy and efficiency.
It significantly improves the accuracy and visual perception effect of three-dimensional point cloud reconstruction, enhances contrast and detail retention, and ensures the fidelity of scene details and color.
Smart Images
Figure CN119399367A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine perception and machine vision, and specifically relates to a three-dimensional point cloud reconstruction method by fusing multiple exposure images. Background Art
[0002] With the development of computer vision technology, 3D reconstruction technology has been booming and has become one of the most popular research directions in computer vision. It involves multiple disciplines such as image processing, stereo vision, and pattern recognition. It uses computers to build three-dimensional models that express real objective scenes to meet the needs of production and life. With the continuous development of the industrialization process, the realization of various technologies depends on the acquisition of three-dimensional information of the target object.
[0003] With the development of deep learning technology, 3D reconstruction technology has entered a new stage of development. By using deep learning algorithms, the accuracy and efficiency of 3D reconstruction can be further improved, enabling the technology to handle more complex scenes and objects, and provide more accurate and rich 3D information for various application fields. Although the 3D reconstruction model based on deep learning can generate relatively accurate depth maps in most cases, in the case of weak texture areas, the weak texture areas lack sufficient feature information for matching. In the multi-view stereo model, there are not enough feature points for noisy and irregular surfaces, and it is difficult to extract enough feature points for restoration. As a result, partial information with incomplete contours is obtained, which ultimately leads to poor 3D reconstruction of point clouds.
[0004] Therefore, it is necessary to provide a three-dimensional point cloud reconstruction method based on multi-exposure image fusion to solve the above technical problems. Summary of the invention
[0005] In order to solve the technical problems existing in the prior art, the present invention provides a 3D point cloud reconstruction method by fusing multiple exposure images. A detail enhancement module is designed to comprehensively explore the details from the source input, and an enhanced image is effectively formed by finding the optimal local exposure to fully mine the information in the source image, enhance the contrast and preserve the details.
[0006] To achieve the above purpose, the technical solution adopted by the present invention is: a three-dimensional point cloud reconstruction method based on multi-exposure image fusion, the specific steps are as follows:
[0007] Step 1: Obtain the main view and N-1 views in other directions of the 3D model, and convert the N 3D model multi-view images from RGB format to YCbCr format.
[0008] Step 2: Design an adversarial learning network. Take the output of step 1 as input. Design a network that combines detail enhancement module and color enhancement module for global-local learning and outputs chromaticity component Cb f, chrominance component Cr f With the brightness component Y f Image, convert from YCbCr format to RGB image format.
[0009] Step 3: The detail enhancement module fuses different source images to generate a brightness component with richer details. The calculation formula of the brightness component is as follows:
[0010]
[0011] Among them, Q represents the total number of layers of the detail enhancement network, q represents the current layer of the detail enhancement network, and r q represents the hyperparameter, R^ q Represents the set of components of the source image, represents the perceptual loss calculation, l pix Represents each R^ q and Y f The normalized Manhattan distance between them, I1, I2 represent the source images with different exposures, Indicates the need to learn parameters, θ DEM Detail enhancement network.
[0012] Step 4: Use the color enhancement module to complete the color mapping from the target (fusion) brightness to the fusion image and from the source image to the fusion image. The calculation formula of the color mapping is as follows:
[0013]
[0014] Among them, θ CEM represents the detail enhancement network, Y f I1, I2 represent source images with different exposures. represents the hyperparameters to be learned, Cb f Represents the color information component, Cr f Represents the color saturation information component.
[0015] Step 5: Use the output of step 2 as the input of the feature extraction network. The feature extraction network uses 8 convolutional layers to obtain N groups of feature maps with a feature size of 32*H*W, where H represents the length of the feature size and W represents the width of the feature size.
[0016] Step 6: Use the feature map output in step 5 to construct the feature volume, and use the homography transformation matrix H i (d) Converted into a feature volume with a depth of di and a size of N*D*32*H*W, where N represents n viewing angles, D represents the depth value corresponding to a feature point on a single feature plane, and the homography transformation matrix H i (d) is calculated as follows:
[0017]
[0018] Where s represents the scale factor, f x 、f y ,γ,u0,v0 represent the five camera internal parameters, r1,r2,t represent the camera external parameters, X w , Y w , Z w Represents the world coordinate system.
[0019] Step 7: Calculate the variance of each point on each channel of each feature volume output in step 6, and convert the value of the point into a cost volume of size D*32*H*W. The calculation method is as follows:
[0020]
[0021] Among them, V1, V2, ..., V N represents the feature volume under N perspectives, and C represents the cost volume.
[0022] Step 8: A 3D-UNet network structure is used for the cost body, and the cost body attention method is introduced to sum up the attention cost body features. The calculation formula is as follows:
[0023]
[0024] Among them, P h,w,m Represented as the probability volume feature output by each layer. V h,w,d Represents the cost volume features before each layer input. V ～ ∈R D*H*W represents the attention cost body, M is the number of previous depth hypotheses, and C is the number of channels.
[0025] Step 9: The attention cost body compresses the number of channels to 1 through convolution, resulting in a size of probability body.
[0026] Step 10: For the size The probability volume is calculated for each point on the plane along the d direction to obtain the initial depth map.
[0027] Step 11: Normalize the depth map, perform information fusion through a 4-layer residual network, add the output single-channel feature map to its initial depth map, and obtain the optimized depth map.
[0028] Step 12: Calculate the loss of the initial depth map and the optimized depth map respectively, and then add the λ weight coefficient. The value of λ is 1.0. The loss calculation method is as follows:
[0029]
[0030] Among them, d(p) is the true depth value, is the depth value corresponding to the initial depth map, To optimize the depth value corresponding to the depth map, λ is the weight coefficient and its value is 1.0.
[0031] Step 13: Use internal and external parameter matrix transformation to convert the optimized depth map into three-dimensional point cloud data, and finally obtain a three-dimensional point cloud model.
[0032] The detail enhancement module designed in the present invention fully explores the details from the source input, effectively forms an enhanced image by finding the optimal local exposure, fully mines the information within the source image, enhances the contrast and preserves the details, and at the same time adds a color enhancement model to improve the appearance by learning the relationship between color and brightness in natural images of the same scene, thereby making the fused image more realistic and vivid, and significantly improving visual perception.
[0033] The present invention introduces cost body attention to capture the fine details and overall image context between the predicted image and the real image, while bringing quality gain of detail preservation and merging multiple images taken at different exposure levels into one image to preserve scene details and color fidelity. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Flowchart of 3D point cloud reconstruction based on multi-view.
[0035] Figure 2 This is an image enhancement diagram based on detail enhancement and color enhancement.
[0036] Figure 3 Schematic diagram of homography transformation. DETAILED DESCRIPTION
[0037] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0038] like Figure 1 As shown in the figure, a 3D point cloud reconstruction method based on multi-exposure image fusion is firstly provided with multi-view information of the model by surrounding the 3D model, and the color and detail of the multi-view image are enhanced by using an enhancement module. A multi-view stereo matching algorithm is used to densify the sparse reconstruction, and the reconstruction result is a dense point cloud model of the 3D model.
[0039] Use a drone to surround the 3D model and capture N 3D model images in multiple views to obtain the main view and N-1 other views. The other views provide more information for the 3D model, thereby inferring the depth value more accurately.
[0040] like Figure 2As shown in the figure, converting N 3D model images from RGB to YCbCr format can effectively convert brightness and texture (Y f Channels, i.e. brightness components) are separated from colors (C b and C r Channel, i.e. chrominance component), a global-local adversarial learning network is used for image enhancement.
[0041] The input of the adversarial learning network is given two source images I1 (Y channel) and I2 (C channel) with different exposure settings and 3 channels. b and C r ) are used as input respectively, and the network adopts detail enhancement module and color enhancement module to combine global-local learning.
[0042] The detail enhancement module fuses different source images to generate a brightness component with richer details. The calculation formula of the brightness component is as follows:
[0043]
[0044]
[0045] Among them, r q is a hyperparameter, l per Indicates that each R q and Y f The normalized Manhattan distance between represents the perceived loss, Indicates the need to learn parameters, θ DEM It is a detail enhancement module.
[0046] The detail enhancement module consists of a joint encoder and a discriminative encoder. First, the joint encoder is divided into encoding and decoding stages to extract relevant features between the two source images.
[0047] The encoding stage of the joint encoder includes two encoders, each of which contains 4 feature extraction layers, and two source images I1 and I2 are input as inputs. The outputs of the two encoders are merged.
[0048] After each feature extraction layer, the feature map is max-pooled and used as the input of the next feature extraction layer. The outputs of the two encoders are connected in series as the input of the discriminative encoder, aiming to encode each source image separately and tend to discover the representative information of each input.
[0049] Another discriminative encoder of the joint encoder contains 4 layers of feature extraction layers. After 3 upsampling, the output channel number is 32 feature maps to generate the final fused brightness component, i.e., Y f .
[0050] In order to enhance the low texture feature information and the original unsaturated color information to become saturated and rich, the color enhancement module is used to perform color mapping from the target (fused) brightness and source image to the appropriate chromaticity of the fused image. The color mapping calculation formula is as follows:
[0051]
[0052] in, Indicates the need to learn parameters, θ CEM Color enhancement module Color enhancement network, Y f It is the brightness component output by the layering module.
[0053] The color enhancement module is a 4-layer joint encoder-decoder structure that selects three images from each perspective, namely the two source images I1 and I2 and the fused brightness component (Y f ) as input to explore the color mapping relationship between input images.
[0054] The joint encoder-decoder performs three up-sampling operations by convolution, and finally restores the same size as the source image, outputting the chrominance component Cb f and Cr f ; Chroma component (Cb f Cr f ) and the brightness component (Y f ) The image is converted from YCbCr format to RGB format as the input of the feature extraction network.
[0055] The multi-view enhanced image is used as the input of the feature extraction network. The feature extraction network uses 8 layers of ConvBnReLU convolutional layers, and finally obtains N groups of feature maps with a feature size of 32*H*W. The input and output of each layer are shown in Table 1.
[0056] Table 1 Feature extraction network based on MVSnet
[0057]
[0058] Among them, in_channe and out_channe represent input and output channels; kernel represents the convolution kernel size; stride represents the step size; padding represents edge padding, which is generally taken as 0.
[0059] The feature volume is constructed using N sets of feature maps with a feature size of 32*H*W. An adaptive variable interval segmentation strategy is used to allocate denser depth hypothesis planes near the potential true depth value and relatively sparse depth hypothesis planes far away from the potential true depth value to perform pixel depth estimation. The depth interval [d1, d2] is set, and the resolution is set to Δd. The formula for calculating the depth segmentation plane is as follows:
[0060]
[0061] like Figure 3 As shown, D planes are obtained, each plane corresponds to a homography transformation matrix H, and the homography transformation matrix H corresponding to the feature of the i-th perspective at the depth value d i The calculation of (d) is as follows:
[0062]
[0063] Where s represents the scale factor, f x 、f y ,γ,u0,v0 represent the five camera internal parameters, r1,r2,t represent the camera external parameters, X w , Y w , Z w Represents the world coordinate system.
[0064] Each channel feature in each set of feature maps is transformed by D homography matrices H with depth in the interval [d1, d2] i (d) Convert to depth di and generate a feature volume of size N*D*32*H*W.
[0065] After N groups of feature maps are mapped to the corresponding three-dimensional space, each value of the N feature bodies of size D*32*H*W in the depth direction is the matching cost of a pixel at different depths, and the value range of D is between [d1, d2].
[0066] N feature volumes of size D*32*H*W are obtained by taking the variance of each point on each channel of each feature volume as the value of the point and converting it into a cost volume of size D*32*H*W. The calculation method is as follows:
[0067]
[0068] Among them, V1, V2, ..., V N represents the feature volume under N perspectives, and C represents the cost volume.
[0069] A 3D-UNet network structure is used for the cost body, and the cost body attention method is introduced to further explore the similarity between the reference view and the source view. The attention cost body features are summed up and the calculation formula is as follows:
[0070]
[0071] Among them, P h,w,m is a probability body, V h,w,c,d As the cost body, V ～ ∈R C*H*Wrepresents the attention cost body, M is the number of upper depth hypotheses, and C is the number of channels.
[0072] The attention cost body compresses the number of channels to 1 through convolution, resulting in a size of The probability volume represents the probability of each pixel along the depth direction in the D dimension.
[0073] For size The probability volume is calculated for each point in the (W*H) plane along the d direction. The expected value is the depth estimate of the point. The expectation is calculated for each pixel point, and a more informative depth map is obtained. The depth map is normalized and information is fused through a 4-layer residual network. The output single-channel feature map is added element by element with its initial depth map to obtain the optimized depth map.
[0074] The loss is calculated for the initial depth map and the optimized depth map respectively, and then added with the λ weight coefficient. The value of λ is 1.0. The loss is calculated as follows:
[0075]
[0076] in, is the initial depth map loss, To optimize the depth map loss.
[0077] The calculation process of converting the depth map to a 3D point cloud uses multi-view geometry, and its principle is obtained based on the internal and external parameter matrix transformation formula:
[0078]
[0079] Among them, x, y, z are the point cloud coordinate system, x ^ ,y ^ represents the image coordinate system, and D is the depth value.
[0080] The optimized depth map is transformed based on the internal and external parameter matrices to generate point cloud data, and finally a three-dimensional point cloud model is obtained.
[0081] The detail enhancement module designed in the present invention comprehensively explores details from the source input, effectively forms an enhanced image by finding the optimal local exposure, fully mines the information within the source image, enhances contrast and preserves details, and at the same time adds a color enhancement model to improve the appearance by learning the relationship between color and brightness in natural images of the same scene, thereby making the fused image more realistic and vivid and significantly improving visual perception.
[0082] The present invention introduces cost body attention to capture the fine details and overall image context between the predicted image and the real image, while bringing quality gain of detail preservation and merging multiple images taken at different exposure levels into one image to preserve scene details and color fidelity.
[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims
1. A three-dimensional point cloud reconstruction method based on multi-exposure image fusion, characterized in that: The specific steps are as follows: Step 1: Obtain N 3D model multi-view images through the 3D model, and convert the N 3D model multi-view images from RGB format to YCbCr format; Step 2: Design an adversarial learning network. Take the output result of step 1 as input, design an adversarial learning network to output an image in YCbCr format, and then convert the image in YCbCr format into an image in RGB format. Step 3: The detail enhancement module fuses different source images to generate brightness components. The calculation formula of the brightness component is as follows: Among them, Q represents the total number of layers of the detail enhancement network, q represents the current layer of the detail enhancement network, and r q represents the hyperparameter, R^ q Represents the set of components of the source image, represents the perceptual loss calculation, l pix Represents each R^ q and Y f The normalized Manhattan distance between them, I1, I2 represent the source images with different exposures, Indicates the need to learn parameters, θ DEM Detail enhancement network; Step 4: Use the color enhancement module to complete the color mapping from the target brightness to the fused image and from the source image to the fused image. The calculation formula for the color mapping is as follows: Among them, θ CEM represents the detail enhancement network, Y f I1, I2 represent source images with different exposures. represents the hyperparameters to be learned, Cb f Represents the color information component, Cr f Represents the color saturation information component; Step 5: Use the output of step 2 as the input of the feature extraction network. The feature extraction network uses 8 convolutional layers to obtain N groups of feature maps with a feature size of 32*H*W, where H represents the length of the feature size and W represents the width of the feature size. Step 6: Use the feature map output in step 5 to construct the feature volume, and use the homography transformation matrix H i (d) Converted into a feature volume with a depth of di and a size of N*D*32*H*W, where N represents n viewing angles, D represents the depth value corresponding to a feature point on a single feature plane, and the homography transformation matrix H i (d) is calculated as follows: Where s represents the scale factor, f x 、f y ,γ,u0,v0 represent the five camera internal parameters, r1,r2,t represent the camera external parameters, X w , Y w , Z w Represents the world coordinate system; Step 7: Calculate the variance of the points on each channel of each feature volume output in step 6, and convert the value of each point into a cost volume of size D*32*H*W. The calculation method is as follows: Among them, V1, V2, ..., V N represents the feature volume under N perspectives, and C represents the cost volume; Step 8: Use a 3D-UNet network structure for the cost body, introduce the cost body attention method, and sum up to get the attention cost body features. The calculation formula is as follows: Among them, P h,w,m Represented as the probability volume feature of each layer output, V h,w,d Represents the cost volume feature before each layer input, V ～ ∈R D*H*W represents the attention cost body, M is the number of upper depth hypotheses, and C is the number of channels; Step 9: The attention cost body compresses the number of channels to 1 through convolution, resulting in a size of The probability body of Step 10: For the size The probability volume is calculated for each point on the plane along the d direction to obtain the initial depth map; Step 11: normalize the initial depth map, perform information fusion through a 4-layer residual network, add the output single-channel feature map to the initial depth map, and obtain an optimized depth map; Step 12: Calculate the loss of the initial depth map and the optimized depth map respectively, and then add the λ weight coefficient. The value of λ is 1.
0. The loss calculation method is as follows: Among them, d(p) is the true depth value, is the depth value corresponding to the initial depth map, To optimize the depth value corresponding to the depth map, λ is the weight coefficient, and the value is 1.0; Step 13: Use internal and external parameter matrix transformation to convert the optimized depth map into three-dimensional point cloud data, and finally obtain a three-dimensional point cloud model.
2. The method for 3D point cloud reconstruction by multi-exposure image fusion according to claim 1, characterized in that: In step 1, the 3D model multi-view image consists of a main view and N-1 views in other directions.
3. The method for 3D point cloud reconstruction by multi-exposure image fusion according to claim 1, characterized in that: In step 2, the adversarial learning network is designed to use the detail enhancement module and the color enhancement module for learning, and the chromaticity component Cb is output respectively. f , chrominance component Cr f With the brightness component Y f image.