A fully automatic, fast, high-precision 3D reconstruction method and system

Abstract

The present invention discloses a fully automatic, fast and high-precision three-dimensional reconstruction method and system, and relates to the field of three-dimensional dynamic reconstruction. The present invention constructs a spectral-point cloud fusion model, introduces Monte Carlo ray tracing to calculate the dynamic reflectivity field, and combines the Fresnel formula and dynamic ray tracing technology to make the reflectivity calculation more in line with physical laws, thereby improving the accuracy of spectral mapping. The present invention also optimizes and adjusts the spectral data based on the error correction matrix through a spectral error compensation network, thereby reducing the error accumulation caused by factors such as measurement angle and surface roughness; in addition, in the three-dimensional reconstruction process, the present invention enhances the point cloud data according to the spectrum, and improves the continuity and spectral consistency of the three-dimensional model through Poisson surface reconstruction and spectral reflectivity-driven grid optimization method.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional dynamic reconstruction, and in particular to a fully automatic, fast and high-precision three-dimensional reconstruction method and system. Background Art

[0002] At present, 3D reconstruction technology is widely used in many fields such as computer vision, reverse engineering, medical imaging, and cultural heritage protection. Its main goal is to obtain the geometric structure of an object through various imaging and measurement methods, and reconstruct a high-precision 3D model. Traditional 3D reconstruction methods mainly include stereo vision, structured light scanning, LiDAR, multi-view geometric reconstruction, etc. The stereo vision method relies on images taken from different angles by multiple cameras. By matching feature points in the same scene, it uses triangulation to calculate the depth information of the object and finally constructs a 3D model.

[0003] Although the current 3D reconstruction technology has made significant progress, it still faces many challenges, especially in complex scenes or high-precision reconstruction requirements. The existing methods have the following problems: First, it cannot be fully automated. Human intervention is required during the reconstruction process, and the equipment needs to be adjusted manually. After the reconstruction is completed, human-computer interaction is required for mapping, which is very troublesome. It is also impossible to reconstruct the texture of the target while reconstructing the geometric shape of the target. In the end, only a white geometric model is obtained, which is not conducive to the observation of the target. In addition, most traditional 3D reconstruction methods only focus on the geometric shape of the object, while ignoring the acquisition and use of spectral information. Since the color, material and lighting conditions of the object surface have an important influence on the 3D reconstruction results, the reconstruction method lacking spectral information is easily affected by the ambient light and the reflective characteristics of the surface material, resulting in the reconstruction model not being realistic enough in visual effect, and even geometric errors. Secondly, point cloud data based on structured light, LiDAR and other technologies often have noise, especially in the case of special materials such as low reflectivity, transparent or metal surfaces. Traditional methods are difficult to effectively remove noise and optimize point cloud quality. In addition, in the 3D reconstruction process, due to the lack of accurate modeling of the reflectivity of the object surface, existing methods are usually unable to truly simulate the spectral reflectance characteristics of the object under different lighting conditions, thus affecting the rendering effect and the physical consistency of the model. At the same time, the surface reconstruction method based on point cloud still has certain limitations in mesh generation and optimization, resulting in the final 3D model may have problems such as uneven surface and discontinuous topological structure. Therefore, how to improve the existing technology so that the 3D reconstruction can not only accurately capture geometric information, but also effectively combine spectral information to improve reconstruction accuracy and visual effects has become an important direction of current research. Summary of the invention

[0004] The present invention proposes a fully automatic, fast, high-precision three-dimensional reconstruction method and system, which can effectively make up for the shortcomings of traditional methods and improve the spectral accuracy, reflectivity consistency and dynamic adaptability of three-dimensional reconstruction.

[0005] Among them, a fully automatic, fast, high-precision 3D reconstruction method includes the following steps:

[0006] S1. Through hyperspectral imaging and RGB imaging, the spectral data and image data of the target object surface are collected synchronously. The image-driven point cloud data is obtained through the Structure from Motion technology combined with the depth estimation algorithm. The spectral data and point cloud data are aligned through the joint calibration model, and the aligned data are pre-processed through adaptive noise suppression and iterative nearest point to obtain the spectral point cloud data set;

[0007] S2. Based on the spectral point cloud dataset, a spectral-point cloud fusion model is constructed, and point cloud rendering is performed using 3D Gauss Splating technology. The dynamic reflectivity field of the target object surface is calculated using Monte Carlo ray tracing, and the spectral mapping error of the dynamic reflectivity field is corrected in combination with a spectral error compensation network. The corrected spectral data is mapped to the point cloud coordinate system using the spectral-point cloud fusion model to form spectrally enhanced point cloud data.

[0008] S3. Generate a continuous surface based on Poisson surface reconstruction of spectrally enhanced point cloud data, and adjust the triangular mesh structure through spectral reflectance-driven mesh optimization to obtain a three-dimensional mesh model; generate texture information with object surface details based on the surface characteristics of image data and spectrally enhanced point cloud; drive texture generation through dynamic reflectance field to finally generate a texture with spectral information, and map the texture to the three-dimensional mesh model; generate the final 3DOBJ model with texture and spectral information.

[0009] Furthermore, a fully automatic, fast, high-precision 3D reconstruction system is provided, which is implemented based on any one of the above-mentioned fully automatic, fast, high-precision 3D reconstruction methods, and comprises:

[0010] The data acquisition module is used to collect the spectral data and point cloud data of the target object surface synchronously through hyperspectral imaging and RGB imaging combined with SFM technology, align the coordinates of the spectral data and point cloud data through the joint calibration model, and pre-process the aligned data through adaptive noise suppression and iterative nearest point to obtain the spectral point cloud data set;

[0011] The data processing module is used to construct a spectral-point cloud fusion model based on the spectral point cloud data set, and obtain spectrally enhanced point cloud data by combining the calculation of dynamic reflectivity field and spectral error compensation network.

[0012] A data reconstruction module is used to generate a continuous surface based on the Poisson surface reconstruction of the spectrally enhanced point cloud data, and adjust the triangular mesh structure through mesh optimization driven by spectral reflectance;

[0013] Among them, the data processing module specifically includes a dynamic reflectivity field construction unit for calculating the dynamic reflectivity field of the target object surface through Monte Carlo ray tracing, a spectral error compensation network correction unit for correcting the spectral mapping error of the dynamic reflectivity field, and a spectral enhanced point cloud data output unit for mapping spectral data to a point cloud coordinate system through a spectral-point cloud fusion model to form spectral enhanced point cloud data.

[0014] A computer-readable storage medium is used to store a computer program, which, when executed on a computer, enables the computer to execute a fully automatic, fast, high-precision three-dimensional reconstruction method as described in any one of the above.

[0015] An electronic device, comprising:

[0016] Memory, used to store computer programs;

[0017] A processor is used to execute the computer program to implement a fully automatic, fast, and high-precision three-dimensional reconstruction method as described in any one of the above items.

[0018] The beneficial effects of the invention are:

[0019] (1) The present invention constructs a spectrum-point cloud fusion model and introduces Monte Carlo ray tracing to calculate the dynamic reflectivity field. Combining the Fresnel formula and dynamic ray tracing technology, the reflectivity calculation is more in line with physical laws, thereby improving the accuracy of spectral mapping. In addition, in order to further reduce the spectral error, the present invention also optimizes and adjusts the spectral data based on the error correction matrix through a spectral error compensation network, thereby reducing the error accumulation caused by factors such as measurement angle and surface roughness;

[0020] (2) In the 3D reconstruction process, the present invention uses spectral enhanced point cloud data to improve the continuity and spectral consistency of the 3D model through Poisson surface reconstruction and spectral reflectance-driven mesh optimization methods. Compared with traditional Delaunay triangulation or direct point cloud interpolation methods, the present invention uses spectral data as an optimization constraint to ensure that the mesh structure is consistent with the spectral reflectance characteristics of the real object, thereby providing more realistic and accurate 3D reconstruction results in dynamic scenes.

[0021] (3) The present invention uses multiple cameras to simultaneously image the target at 720°, and uses a fully automatic cutout technology based on artificial intelligence to remove the background on the image. It then uses the latest 3D Gauss Splating rendering technology to perform three-dimensional reconstruction, thereby achieving fully automatic, fast, and high-precision three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A method flow chart of a fully automatic, fast and high-precision three-dimensional reconstruction method provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the structure of a fully automatic, fast, high-precision three-dimensional reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention. It should be noted that relational terms such as the terms "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0027] Moreover, the terms "comprises," "comprising," or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or machine that includes a list of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, article, or machine. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or machine that includes the element.

[0028] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0029] Embodiment 1

[0030] like Figure 1 , a fully automatic, fast, high-precision 3D reconstruction method, comprising the following steps:

[0031] S1. Through the combination of hyperspectral imaging and RGB imaging with SFM technology, the spectral data and point cloud data of the target object surface are collected synchronously, the spectral data and point cloud data are aligned through the joint calibration model, and the aligned data are preprocessed through adaptive noise suppression and iterative nearest point to obtain the spectral point cloud data set;

[0032] S2. Based on the spectral point cloud dataset, a spectral-point cloud fusion model is constructed, and point cloud rendering is performed using 3D Gauss Splating technology. The dynamic reflectivity field of the target object surface is calculated using Monte Carlo ray tracing, and the spectral mapping error of the dynamic reflectivity field is corrected in combination with a spectral error compensation network. The spectral data is mapped to the point cloud coordinate system using the spectral-point cloud fusion model to form spectrally enhanced point cloud data.

[0033] S3. Generate a continuous surface based on Poisson surface reconstruction of spectrally enhanced point cloud data, and adjust the triangular mesh structure through spectral reflectance-driven mesh optimization; generate texture information with object surface details based on the surface characteristics of image data and spectrally enhanced point cloud; drive texture generation through dynamic reflectance field, finally generate texture with spectral information, and map the texture to the 3D mesh model; generate the final 3D OBJ model with texture and spectral information.

[0034] Specifically, the specific implementation principle of the above embodiment is:

[0035] S1. Through hyperspectral imaging and RGB imaging, the spectral data and image data of the target object surface are collected synchronously, and the image-driven point cloud data are obtained through the SFM (Structure from Motion) technology combined with the depth estimation algorithm. The spectral data and point cloud data are aligned by the joint calibration model, and the aligned data are pre-processed by adaptive noise suppression and iterative closest point to obtain a spectral point cloud data set; preferably, due to the existence of noise interference and coordinate errors in the acquisition process, the noise in the spectral and point cloud data is further removed by an adaptive noise suppression method, and the coordinate alignment accuracy is optimized by the iterative closest point (ICP) algorithm, finally forming a high-quality spectral point cloud data set.

[0036] S2. Based on the spectral point cloud data set, a spectral-point cloud fusion model is constructed, and point cloud rendering is performed using the 3D Gauss Splating technology. The dynamic reflectivity field of the target object surface is calculated using Monte Carlo ray tracing, and the spectral mapping error of the dynamic reflectivity field is corrected in combination with the spectral error compensation network. The corrected spectral data is mapped to the point cloud coordinate system through the spectral-point cloud fusion model to form spectrally enhanced point cloud data. Since the spectral reflectivity characteristics of the object surface are not only related to its own material, but also affected by the ambient light, the Monte Carlo ray tracing method is used to calculate the dynamic reflectivity field, and the reflectivity distribution at different wavelengths is estimated by simulating multiple reflections and scattering of light on the object surface. However, due to the complexity of the light propagation model and the influence of measurement errors, there may be certain errors in the mapping of spectral data. Therefore, a spectral error compensation network is further introduced. The error of the spectral data is minimized by training the network, and an error correction matrix is ​​output. The correction matrix is ​​used to compensate for the spectral mapping error in the dynamic reflectivity field, thereby improving the accuracy of the fusion of spectral data and point cloud data. The spectrum-point cloud fusion model maps the corrected spectrum data to the point cloud coordinate system, so that each point cloud data point contains accurate spectrum information, forming spectrum enhanced point cloud data.

[0037] S3. Generate a continuous surface based on Poisson surface reconstruction of spectrally enhanced point cloud data, and adjust the triangular mesh structure through spectral reflectivity-driven mesh optimization to obtain a three-dimensional mesh model; generate texture information with object surface details based on the surface characteristics of image data and spectrally enhanced point cloud; drive texture generation through dynamic reflectivity field, finally generate texture with spectral information, and map the texture to the three-dimensional mesh model; generate the final 3DOBJ model with texture and spectral information. However, due to the discreteness of point cloud data, the directly reconstructed surface may have geometric errors, so a spectral reflectivity-driven mesh optimization strategy is further introduced to adjust the triangular mesh structure. In the mesh optimization process, the normal vector of each point in the spectrally enhanced point cloud data is first calculated, and the initial triangular mesh is constructed based on the normal vector. The Poisson reconstructed surface is further discretized into triangular patches through the Delaunay triangulation method, and the spectral reflectance value of each patch is determined by the spectral information of the adjacent point cloud data. Preferably, in order to improve the spectral consistency and geometric accuracy of the model, the Lagrange multiplier method optimization algorithm is used to match the normal vector of the triangular facet with the normal vector of the point cloud data, and the position of the mesh vertices is adjusted through iterative optimization, ultimately obtaining a high-precision spectrally enhanced three-dimensional reconstruction model.

[0038] Furthermore, in step S1, the specific implementation principle process of obtaining the spectral point cloud data set is as follows:

[0039] Assume that each point cloud data point has spatial coordinates , the wavelength of the corresponding spectral data is ; Map the spectral data to a low-dimensional space through principal component analysis and maintain its geometric structure. Through the above mapping, combine the spectral data and the point cloud coordinate data to generate a fused spectral point cloud data set; preferably, use deep learning technology (such as multimodal neural network) to jointly train the point cloud data and the spectral data to learn the complex relationship between the two. This method can automatically adjust the weight of the spectral features according to the geometric features of the point cloud, so that the fused result can better reflect the real optical properties of the object. In the above manner, the obtained spectral point cloud data set includes coordinate values ​​and spectral values. The coordinate values ​​represent the spatial position of the point on the surface of the object, which is described by three-dimensional coordinates (x, y, z); represents the information of the geometric shape and structure of the object; and the spectral values ​​represent the reflection characteristics of the surface of the object in different bands, reflecting the absorption and reflection characteristics of the substance in different spectral regions. Further, by fusing the point cloud and spectral data, a data set including comprehensive data of spatial information (i.e., surface shape) and spectral information (i.e., material and surface reflectivity) is obtained, which can more accurately perform surface analysis, defect detection, material classification and surface property evaluation of the object.

[0040] Furthermore, in step S2, calculating the dynamic reflectivity field of the target object surface by Monte Carlo ray tracing specifically includes the following sub-steps:

[0041] S2011. Calculate the reflectivity of each point cloud data by ray tracing according to the Fresnel formula, and summarize the reflectivity of all point cloud data by ray tracing on the surface of the target object to obtain an initial reflectivity field;

[0042] S2012. According to the surface material type of the target object, setting the reflectivity coefficient of the material type at different wavelengths;

[0043] S2013. Calculate the dynamic reflectivity field based on the set reflectivity coefficient and the initial reflectivity field.

[0044] Furthermore, the Fresnel formula is specifically expressed as:

[0045] ;

[0046] Among them, the Indicates wavelength and the angle of incidence The reflectivity under represents the refractive index of the object, Represents the reflection angle. Specifically, the Monte Carlo method is used to simulate the reflection process of light multiple times to calculate the reflectivity value of each point on the target surface. In addition, for the surface material type of the target object, material parameters are introduced into the reflectivity calculation according to the material of the target object (such as metal, ceramic, plastic, etc.). These material properties (such as glossiness, absorption coefficient, refractive index, etc.) directly affect the calculation of the reflectivity field. For each material, the reflectivity calculation formula needs to be adjusted according to its spectral characteristics, thereby optimizing the calculation of the reflectivity field.

[0047] For example, the metal surface has strong specular reflection characteristics, and the reflectivity changes with the incident angle. The Fairbrun model can be used to reflect light and calculate the reflectivity; the surface of ceramic materials reflects strong diffuse light, so the Lambertian reflection model is used for calculation.

[0048] Furthermore, the step S2011 specifically includes the following sub-steps:

[0049] S20111. Calculate the normal vector of each point cloud data;

[0050] S20112. Set the directions of the incident light and the reflected light for each point cloud data, calculate the incident angle by the normal and the direction of the light, and calculate the reflection angle by the angle between the reflected light and the surface normal;

[0051] S20113. Calculate the reflectivity of the ray tracing of each point cloud data by using the Fresnel formula according to the calculated incident angle and reflection angle;

[0052] S20114. Summarize the reflectivity of each point to obtain the initial reflectivity field of the entire target object surface.

[0053] Furthermore, in step S2013, the specific process of calculating the dynamic reflectivity field is expressed as follows:

[0054] ;

[0055] Among them, the Representing point cloud data At wavelength The dynamic reflectivity field under represents the wavelength, Represents the spatial coordinates of point cloud data, Indicates the reflectivity coefficient of the surface material type of the target object at different wavelengths. Representing point cloud data At wavelength Specifically, the initial reflectivity field It is a function of spatial coordinates and wavelength, and is related to material properties.

[0056] Furthermore, represents the reflectivity at a certain wavelength and a certain incident angle, and is used to describe the reflective properties of the interaction between light and the surface. In the above embodiment, it is based on the optical properties of the material; and It represents the spectral reflectance of each point on the surface of the target object, that is, the reflection characteristics of a certain wavelength at a spatial position. Each point on the target surface needs to be associated with the incident angle, which is related to the propagation path of the light and the surface normal. The reflectance depends on the incident angle and the reflection angle. These two angles are related to the surface normal of each point. For a surface point, its normal vector is obtained through the point cloud data; for each point, the reflectance of the point at a certain wavelength is calculated through the normal direction of the point, the direction of the incident light and the direction of the reflected light through the spectral reflectance formula, that is: ; wherein said Represent the incident light, reflected light and surface normal respectively. It represents a geometric factor used to describe the relationship between incident and reflected light and the surface normal, which is calculated according to a reflection model. Exemplarily, it can be calculated using a Lambertian reflection model or a Phong reflection model.

[0057] Furthermore, in step S2, in combination with the spectral error compensation network, the spectral mapping error of the dynamic reflectivity field is corrected, which specifically includes the following sub-steps:

[0058] S2021. Taking minimizing the error of spectral data as the optimization goal, defining the error function of the spectral error compensation network, and training the spectral error compensation network by minimizing the error function;

[0059] S2022. Outputting an error correction matrix through the trained spectral error compensation network;

[0060] S2023. Correct the spectral data of each point cloud data point according to the error correction matrix.

[0061] Specifically, the spectral error compensation network is trained using a deep learning method, and learns how to correct errors by comparing with real reflectance data. The input of the network is the original spectral reflectance data, and the output is the spectral data after error compensation. For the reflectance value of each point cloud data, the spectral error compensation network identifies the error based on its spectral characteristics and automatically adjusts the corresponding value. For example, if the reflectance value in certain bands is too high or too low, the network will adjust according to the compensation strategy obtained during training, thereby improving the accuracy of the spectral data.

[0062] Furthermore, in step S2021, the error function of the spectral error compensation network is expressed as:

[0063] ;

[0064] Among them, the represents the error function of the spectral error compensation network, represents the error correction matrix, Represents the total number of point cloud data. Indicates the number index of point cloud data. Represents the i-th point cloud data The corresponding original spectral data is obtained through historical data. Represents the i-th point cloud data The corresponding real spectral data is obtained through actual measurement.

[0065] Furthermore, in step S2022, the error correction matrix is ​​expressed as:

[0066] ;

[0067] Said It means that the solution can minimize the error correction matrix , that is, to find an optimal , so that the corrected spectral data Closest to the real spectral data.

[0068] Specifically, the error correction matrix is ​​used to correct the spectrum value of each wavelength, which is defined as: Assuming the error correction matrix is an n×n matrix, where n represents the measurement data of n wavelengths at each point. Each element of represents the correction coefficient of the spectrum data at the i-th wavelength to the spectrum data at the j-th wavelength. For example, assuming n=3, then the correction matrix and the original spectral data as follows:

[0069] ;

[0070] ;

[0071] Through matrix multiplication, the corrected spectral data is obtained :

[0072] .

[0073] Furthermore, the spectrally enhanced point cloud data includes point cloud coordinates and the spectral reflectance of each point, namely:

[0074] ;

[0075] Among them, the represents spectrally enhanced point cloud data, Represent point cloud data The coordinates of Indicates that the i-th point cloud data has a wavelength of The spectral reflectance under represents the mth wavelength, Represents the i-th point cloud data.

[0076] Furthermore, the step S3 specifically includes the following sub-steps:

[0077] S301. Calculate the normal vector of each point in the spectral enhanced point cloud data;

[0078] S302. Generate a three-dimensional surface based on the normal vector data of the point cloud by a Poisson reconstruction algorithm;

[0079] S303. discretizing the Poisson reconstructed surface into a triangular mesh by Delaunay triangulation, wherein the triangular mesh is composed of triangular facets, each facet corresponding to three adjacent points in the point cloud, wherein each point has a spectral reflectance value;

[0080] S304. The normal vector of each triangle is consistent with the normal vector of the corresponding point in the point cloud data as the objective function of the spectrum-driven optimization, and the objective function is minimized by the Lagrange multiplier method optimization algorithm;

[0081] S305. Optimize the mesh structure by iteratively updating the positions of the mesh vertices.

[0082] Furthermore, in step S304, the objective function of the spectrum-driven optimization is specifically expressed as:

[0083] ;

[0084] Among them, the represents the objective function of spectrum-driven optimization, Represents a triangle The normal vector of the patch, Represents the i-th point cloud data The normal vector, Represents a patch, the represents the influence factor of spectral reflectance on grid optimization. Representing point cloud data In wavelength The spectral reflectance under Representation patch In wavelength The spectral reflectance under Represents the mth wavelength. Specifically, the mesh vertex positions are adjusted by a spectral reflectance-driven optimization method to improve the quality of the mesh. The optimization goal is to make the normal vector of each triangle consistent with the normal vector of the corresponding point in the point cloud data, thereby maintaining the consistency of geometric and spectral information. The objective function consists of two parts. The first part is the geometric consistency term, which ensures that the normal vector of the triangular mesh face is consistent with the normal vector of the point cloud data. The second part is the spectral consistency term, which ensures that the spectral reflectance data can be effectively transmitted and maintained in the mesh.

[0085] Furthermore, as a preferred implementation of the above embodiment, a verification and post-processing method for the optimized grid is proposed, which includes the following steps:

[0086] S401. Verify the geometric quality of the optimized mesh by using mesh smoothness index and triangle shape quality, wherein Delaunay property and edge length ratio are used as evaluation criteria;

[0087] S402. Verify the spectral consistency by calculating the error between the spectral reflectance of each patch and the reflectance of adjacent point cloud data points;

[0088] S403. Output the optimized mesh as a three-dimensional mesh file.

[0089] Furthermore, in step S401, the specific process of verifying the geometric quality of the optimized mesh by the mesh smoothness index and the triangle shape quality is as follows: the geometric accuracy of the surface reconstruction is verified by calculating the error between the mesh vertices and the original point cloud data, that is:

[0090] ;

[0091] Among them, the Represents the error between the mesh vertex and the original point cloud data. Optimized grid and points The closest point is extracted by random sampling.

[0092] Furthermore, the specific process of step S402 is expressed as follows:

[0093] ;

[0094] Among them, the Represents the error between the spectral reflectance of each patch and the reflectance of the adjacent point cloud data points. represents the i-th patch.

[0095] Embodiment 2

[0096] Furthermore, as a preferred implementation of the above embodiment, a fully automatic, fast, high-precision 3D reconstruction system is proposed, comprising:

[0097] The data acquisition module is used to collect the spectral data and point cloud data of the target object surface synchronously through hyperspectral imaging and RGB imaging combined with Structure from Motion technology, align the coordinates of the spectral data and point cloud data through the joint calibration model, and pre-process the aligned data through adaptive noise suppression and iterative nearest point to obtain the spectral point cloud data set;

[0098] The data processing module is used to construct a spectrum-point cloud fusion model based on the spectrum point cloud data set; calculate the dynamic reflectivity field of the target object surface through Monte Carlo ray tracing, and correct the spectrum mapping error of the dynamic reflectivity field in combination with the spectrum error compensation network; map the spectrum data to the point cloud coordinate system through the spectrum-point cloud fusion model to form spectrum enhanced point cloud data;

[0099] The data reconstruction module is used to generate a continuous surface based on the Poisson surface reconstruction of the spectrally enhanced point cloud data, and adjust the triangular mesh structure through spectral reflectance-driven mesh optimization.

[0100] Furthermore, the data processing module specifically includes a dynamic reflectance field building unit, a spectral error compensation network correction unit and a spectral enhanced point cloud data output unit, wherein:

[0101] The dynamic reflectivity field construction unit specifically includes:

[0102] The initial reflectivity field calculation subunit is used to calculate the reflectivity of the ray tracing of each point cloud data according to the Fresnel formula, and to summarize the reflectivity of the ray tracing of all point cloud data on the surface of the target object to obtain the initial reflectivity field;

[0103] The reflectivity coefficient calculation subunit is used to set the reflectivity coefficient of the material type at different wavelengths according to the surface material type of the target object;

[0104] The dynamic reflectivity long calculation subunit is used to calculate the dynamic reflectivity field according to the set reflectivity coefficient and the initial reflectivity field.

[0105] Among them, the specific process of calculating the dynamic reflectivity field is expressed as:

[0106] ;

[0107] Among them, the Representing point cloud data At wavelength The dynamic reflectivity field under represents the wavelength, Represents the spatial coordinates of point cloud data, Indicates the reflectivity coefficient of the surface material type of the target object at different wavelengths. Representing point cloud data At wavelength Initial reflectivity field under .

[0108] Furthermore, the spectral error compensation network correction unit specifically includes:

[0109] An optimization target definition subunit is used to take minimizing the error of spectral data as the optimization target, define the error function of the spectral error compensation network, and train the spectral error compensation network by minimizing the error function;

[0110] A network training subunit, used for outputting an error correction matrix through a trained spectral error compensation network;

[0111] The error correction subunit is used to correct the spectral data of each point cloud data point according to the error correction matrix.

[0112] Furthermore, the error function of the spectral error compensation network is expressed as:

[0113] ;

[0114] Among them, the represents the error function of the spectral error compensation network, represents the error correction matrix, Represents the total number of point cloud data. Indicates the number index of point cloud data. Represents the i-th point cloud data The corresponding original spectral data is obtained through historical data. Represents the i-th point cloud data The corresponding real spectral data is obtained through actual measurement.

[0115] Furthermore, the error correction matrix is ​​expressed as:

[0116] ;

[0117] Among them, the It means that the solution can minimize the error correction matrix , that is, to find an optimal , so that the corrected spectral data Closest to the real spectral data.

[0118] Furthermore, the spectrally enhanced point cloud data includes point cloud coordinates and the spectral reflectance of each point, namely:

[0119] ;

[0120] Among them, the represents spectrally enhanced point cloud data, Represent point cloud data The coordinates of Indicates that the i-th point cloud data has a wavelength of The spectral reflectance under represents the mth wavelength, Represents the i-th point cloud data.

[0121] Furthermore, the data reconstruction module specifically includes:

[0122] A normal vector calculation unit, used to calculate the normal vector of each point in the spectral enhanced point cloud data;

[0123] A 3D reconstruction unit, used to generate a 3D surface through a Poisson reconstruction algorithm based on normal vector data of the point cloud;

[0124] Constructing a processing unit for discretizing the Poisson reconstructed surface into a triangular mesh by Delaunay triangulation, wherein the triangular mesh is composed of triangular facets, each facet corresponding to three adjacent points in the point cloud, wherein each point has a spectral reflectance value;

[0125] The spectral driven optimization unit is used to make the normal vector of each triangle consistent with the normal vector of the corresponding point in the point cloud data as the objective function of the spectral driven optimization, and minimize the objective function through the Lagrange multiplier method optimization algorithm;

[0126] An update iterative optimization unit is used to iteratively update the positions of mesh vertices and optimize the mesh structure;

[0127] The texture mapping generation unit is used to drive texture generation through a dynamic reflectivity field, ultimately generate a texture with spectral information, and map the texture onto a three-dimensional mesh model; thereby generating a final 3D OBJ model with texture and spectral information.

[0128] Furthermore, the objective function of the spectrum-driven optimization is specifically expressed as:

[0129] ;

[0130] Among them, the represents the objective function of spectrum-driven optimization, Represents a triangle The normal vector of the patch, Represents the i-th point cloud data The normal vector, Represents a patch, the represents the influence factor of spectral reflectance on grid optimization. Representing point cloud data In wavelength The spectral reflectance under Representation patch In wavelength The spectral reflectance under represents the mth wavelength.

[0131] Embodiment 3

[0132] Based on the first embodiment, this embodiment proposes a fully automatic, fast, high-precision 3D reconstruction device, the hardware design diagram of which is shown in FIG. Figure 2 As shown, the target image can be quickly acquired through synchronous acquisition of multiple cameras, large depth of field imaging can be achieved through stacked acquisition, and background interference can be reduced through automatic cutout by artificial intelligence. This further accelerates the reconstruction process and achieves high-precision three-dimensional reconstruction of the target.

[0133] The workflow is as follows

[0134] 1. Through multiple vertically arranged automatic zoom and autofocus cameras, the vertical 360° image of the target object is synchronously acquired; through the rotation of the turntable, the horizontal 360° image of the target object is acquired to achieve 720° all-round imaging. The artificial intelligence algorithm is used to automatically remove the background to ensure that the image data only contains the target object; the SFM (Structure-from-Motion) technology is used to restore the camera posture when the image was taken. Through multi-view geometric matching, the sparse three-dimensional point cloud of the target object is calculated; combined with the depth estimation algorithm, the point cloud density is optimized to obtain image-driven high-precision point cloud data. Hyperspectral imaging is used to synchronously acquire the spectral information of the object surface; through the joint calibration model, the spectral data is aligned with the image point cloud coordinates to form a spectral point cloud data set;

[0135] 2. Construct a spectrum-point cloud fusion model, and establish a spectrum-enhanced point cloud model through the mapping relationship between point cloud coordinates and spectrum data; use the optimized 3D Gauss Splating technology to render the point cloud, so that the spectrum data is evenly distributed in three-dimensional space;

[0136] 3. Monte Carlo ray tracing is used to calculate the dynamic reflectivity field (D-SRF) of the target object surface; combined with the spectral error compensation network, the error correction matrix is ​​calculated and the spectral mapping accuracy is optimized;

[0137] 4. Through the spectrum-point cloud fusion model, the spectral data is mapped to the point cloud coordinate system to form high-precision spectral enhanced point cloud data.

[0138] 5. Use the Poisson surface reconstruction algorithm to convert the spectrally enhanced point cloud into a continuous 3D surface model; use spectral reflectance-driven triangular mesh optimization to adjust the mesh structure to make the spectral data more uniform on the geometric surface; use the image data in the previous step (obtained from the image acquisition process) and the surface characteristics of the spectrally enhanced point cloud to generate texture information with object surface details; combine image data and spectral data, and drive texture generation through a dynamic reflectance field (corrected spectral reflectance data) to ensure that the visual effect of the texture is consistent with the spectral information; finally generate textures with spectral information, and map these textures to the three-dimensional mesh model to generate the final 3D OBJ model with texture and spectral information.

[0139] Embodiment 4

[0140] Based on the first embodiment, this embodiment proposes a terminal device for fully automatic, fast, and high-precision three-dimensional reconstruction. The terminal device includes at least one memory, at least one processor, and a bus connecting different platform systems.

[0141] The memory may include readable media in the form of volatile memory, such as RAM 211 and / or cache memory, and may further include ROM 213 .

[0142] The memory also stores a computer program, which can be executed by the processor, so that the processor executes any one of the above-mentioned fully automatic, fast, and high-precision three-dimensional reconstruction methods in the embodiments of the present application. The specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiments of the above-mentioned method, and some contents are not repeated here. The memory may also include a program / utility with a set (at least one) of program modules, such program modules include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of these examples or some combination may include the implementation of a network environment.

[0143] Accordingly, the processor may execute the above-mentioned computer program, and may execute the program / utility.

[0144] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0145] The terminal device can also communicate with one or more external devices such as keyboards, pointing devices, Bluetooth devices, etc., and can also communicate with one or more devices that can interact with the terminal device, and / or communicate with any device (such as a router, a modem, etc.) that enables the terminal device to communicate with one or more other computing devices. This communication can be carried out through an I / O interface. In addition, the terminal device can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter. The network adapter can communicate with other modules of the terminal device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0146] Embodiment 5

[0147] Based on the first embodiment, this embodiment proposes a computer-readable storage medium for fully automatic, fast, and high-precision 3D reconstruction, on which instructions are stored, and when the instructions are executed by the processor, any of the above-mentioned fully automatic, fast, and high-precision 3D reconstruction methods is implemented. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above method, and some contents are not repeated here.

[0148] The present embodiment provides a program product for implementing the above method, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited to this. In the present embodiment, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than the readable storage medium, which may send, propagate, or transmit a program used by or in combination with an instruction execution system, an apparatus, or a device. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operation of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device, partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0150] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A fully automatic, fast, high-precision 3D reconstruction method, characterized in that: The following steps are involved: S1. Through hyperspectral imaging and RGB imaging, the spectral data and image data of the target object surface are collected synchronously. The image-driven point cloud data is obtained through the Structure from Motion technology combined with the depth estimation algorithm. The spectral data and point cloud data are aligned through the joint calibration model, and the aligned data are pre-processed through adaptive noise suppression and iterative nearest point to obtain the spectral point cloud data set; S2. Based on the spectral point cloud dataset, a spectral-point cloud fusion model is constructed, and point cloud rendering is performed using 3D Gauss Splating technology. The dynamic reflectivity field of the target object surface is calculated using Monte Carlo ray tracing, and the spectral mapping error of the dynamic reflectivity field is corrected in combination with a spectral error compensation network. The corrected spectral data is mapped to the point cloud coordinate system through the spectrum-point cloud fusion model to form spectrally enhanced point cloud data; S3. Generate a continuous surface based on Poisson surface reconstruction of spectrally enhanced point cloud data, and adjust the triangular mesh structure through spectral reflectance-driven mesh optimization to obtain a three-dimensional mesh model; generate texture information with object surface details based on the surface characteristics of image data and spectrally enhanced point cloud; The texture generation is driven by the dynamic reflectivity field, and finally a texture with spectral information is generated, and the texture is mapped onto the three-dimensional mesh model; the final 3D OBJ model with texture and spectral information is generated.

2. A fully automatic, fast, high-precision 3D reconstruction method as claimed in claim 1, characterized in that: In step S2, calculating the dynamic reflectivity field of the target object surface by Monte Carlo ray tracing specifically includes the following sub-steps: S2011. Calculate the reflectivity of each point cloud data by ray tracing according to the Fresnel formula, and summarize the reflectivity of all point cloud data by ray tracing on the surface of the target object to obtain an initial reflectivity field; S2012. According to the surface material type of the target object, setting the reflectivity coefficient of the material type at different wavelengths; S2013. Calculate the dynamic reflectivity field based on the set reflectivity coefficient and the initial reflectivity field.

3. A fully automatic, fast, high-precision 3D reconstruction method as claimed in claim 2, characterized in that: The step S2011 specifically includes the following sub-steps: S20111. Calculate the normal vector of each point cloud data; S20112. Set the directions of the incident light and the reflected light for each point cloud data, calculate the incident angle by the normal and the direction of the incident light, and calculate the reflection angle by the angle between the reflected light and the surface normal; S20113. Calculate the reflectivity of the ray tracing of each point cloud data by using the Fresnel formula according to the calculated incident angle and reflection angle; S20114. Summarize the reflectivity of each point to obtain the initial reflectivity field of the entire target object surface.

4. A fully automatic, fast, high-precision 3D reconstruction method as claimed in claim 2, characterized in that: In step S2013, the specific process of calculating the dynamic reflectivity field is expressed as follows: ； Among them, the Representing point cloud data At wavelength The dynamic reflectivity field under represents the wavelength, Represents the spatial coordinates of point cloud data, Indicates the reflectivity coefficient of the surface material type of the target object at different wavelengths. Indicates that the point cloud data has a wavelength of Initial reflectivity field under .

5. The fully automatic, fast, high-precision 3D reconstruction method according to claim 1, characterized in that: In step S2, in combination with the spectral error compensation network, the spectral mapping error of the dynamic reflectance field is corrected, which specifically includes the following sub-steps: S2021. Taking minimizing the error of spectral data as the optimization goal, defining the error function of the spectral error compensation network, and training the spectral error compensation network by minimizing the error function; S2022. Outputting an error correction matrix through the trained spectral error compensation network; S2023. Correct the spectral data of each point cloud data point according to the error correction matrix.

6. A fully automatic, fast, high-precision three-dimensional reconstruction method as claimed in claim 5, characterized in that: In step S2021, the error function of the spectral error compensation network is expressed as: ； Among them, the represents the error function of the spectral error compensation network, represents the error correction matrix, Represents the total number of point cloud data. Represents the point cloud index, Represents the i-th point cloud data The corresponding original spectral data is obtained through historical data. Represents the i-th point cloud data The corresponding real spectral data is obtained through actual measurement.

7. A fully automatic, fast, high-precision 3D reconstruction method as claimed in claim 6, characterized in that: In step S2022, the error correction matrix is ​​expressed as: , It means that the solution can minimize the error correction matrix , that is, to find an optimal , so that the corrected spectral data Closest to the real spectral data.

8. The fully automatic, fast, high-precision 3D reconstruction method according to claim 1, characterized in that: The spectrally enhanced point cloud data includes the point cloud coordinates and the spectral reflectance of each point, namely: ； Among them, the represents spectrally enhanced point cloud data, Represent point cloud data The three-dimensional coordinates of Indicates that the i-th point cloud data has a wavelength of The spectral reflectance under represents the mth wavelength, Represents the i-th point cloud data.

9. The fully automatic, fast, high-precision 3D reconstruction method according to claim 1, characterized in that: In step S3, the specific process of generating a continuous surface based on the Poisson surface reconstruction of the spectrally enhanced point cloud data and adjusting the triangular mesh structure through spectral reflectance-driven mesh optimization includes the following sub-steps: S301. Calculate the normal vector of each point in the spectral enhanced point cloud data; S302. Generate a three-dimensional surface based on the normal vector data of the point cloud by a Poisson reconstruction algorithm; S303. discretizing the Poisson reconstructed surface into a triangular mesh by Delaunay triangulation, wherein the triangular mesh is composed of triangular facets, each facet corresponding to three adjacent points in the point cloud, wherein each point has a spectral reflectance value; S304. The normal vector of each triangle is consistent with the normal vector of the corresponding point in the point cloud data as the objective function of the spectrum-driven optimization, and the objective function is minimized by the Lagrange multiplier method optimization algorithm; S305. Optimize the mesh structure by iteratively updating the positions of the mesh vertices.

10. The fully automatic, fast, high-precision 3D reconstruction method according to claim 9, characterized in that: In step S304, the objective function of the spectrum-driven optimization is specifically expressed as: ； Among them, the represents the objective function of spectrum-driven optimization, Represents a triangle The normal vector of the patch, Represents the i-th point cloud data The normal vector, Represents a patch, the represents the influence factor of spectral reflectance on grid optimization. Representing point cloud data In wavelength The spectral reflectance under Representation patch In wavelength The spectral reflectance under represents the mth wavelength.

11. A fully automatic, fast, high-precision 3D reconstruction system, which is implemented based on the fully automatic, fast, high-precision 3D reconstruction method according to any one of claims 1 to 10, characterized in that: The system includes: The data acquisition module is used to synchronously collect the spectral data and image data of the target object surface through hyperspectral imaging and RGB imaging, obtain the image-driven point cloud data through the Structure from Motion technology combined with the depth estimation algorithm, and align the coordinates of the spectral data and point cloud data through the joint calibration model; The data processing module is used to construct a spectrum-point cloud fusion model based on the spectrum point cloud data set, and perform point cloud rendering through 3D GaussSplating technology; calculate the dynamic reflectivity field of the target object surface through Monte Carlo ray tracing, and correct the spectrum mapping error of the dynamic reflectivity field in combination with the spectrum error compensation network; map the corrected spectrum data to the point cloud coordinate system through the spectrum-point cloud fusion model to form spectrum enhanced point cloud data; The data reconstruction module is used to generate a continuous surface based on the Poisson surface reconstruction of the spectrally enhanced point cloud data, and adjust the triangular mesh structure through the mesh optimization driven by the spectral reflectivity to obtain a 3D mesh model; generate texture information with object surface details based on the surface characteristics of the image data and the spectrally enhanced point cloud; drive texture generation through the dynamic reflectivity field, and finally generate a texture with spectral information, and map the texture to the 3D mesh model; generate the final 3D OBJ model with texture and spectral information; Among them, the data processing module specifically includes a dynamic reflectivity field construction unit for calculating the dynamic reflectivity field of the target object surface through Monte Carlo ray tracing, a spectral error compensation network correction unit for correcting the spectral mapping error of the dynamic reflectivity field, and a spectral enhanced point cloud data output unit for mapping spectral data to a point cloud coordinate system through a spectral-point cloud fusion model to form spectral enhanced point cloud data.

12. The fully automatic, fast, high-precision 3D reconstruction system according to claim 11, characterized in that: The dynamic reflectivity field construction unit specifically includes: The initial reflectivity field calculation subunit is used to calculate the reflectivity of the ray tracing of each point cloud data according to the Fresnel formula, and to summarize the reflectivity of the ray tracing of all point cloud data on the surface of the target object to obtain the initial reflectivity field; The reflectivity coefficient calculation subunit is used to set the reflectivity coefficient of the material type at different wavelengths according to the surface material type of the target object; The dynamic reflectivity long calculation subunit is used to calculate the dynamic reflectivity field according to the set reflectivity coefficient and the initial reflectivity field.

13. The fully automatic, fast, high-precision 3D reconstruction system according to claim 11, characterized in that: The spectral error compensation network correction unit specifically includes: An optimization target definition subunit is used to take minimizing the error of spectral data as the optimization target, define the error function of the spectral error compensation network, and train the spectral error compensation network by minimizing the error function; A network training subunit, used for outputting an error correction matrix through a trained spectral error compensation network; The error correction subunit is used to correct the spectral data of each point cloud data point according to the error correction matrix.

14. The fully automatic, fast, high-precision three-dimensional reconstruction system according to claim 11, characterized in that: The data reconstruction module specifically includes: A normal vector calculation unit, used to calculate the normal vector of each point in the spectral enhanced point cloud data; A 3D reconstruction unit, used to generate a 3D surface through a Poisson reconstruction algorithm based on normal vector data of the point cloud; Constructing a processing unit for discretizing the Poisson reconstructed surface into a triangular mesh by Delaunay triangulation, wherein the triangular mesh is composed of triangular facets, each facet corresponding to three adjacent points in the point cloud, wherein each point has a spectral reflectance value; The spectral driven optimization unit is used to make the normal vector of each triangle consistent with the normal vector of the corresponding point in the point cloud data as the objective function of the spectral driven optimization, and minimize the objective function through the Lagrange multiplier method optimization algorithm; An update iterative optimization unit is used to iteratively update the positions of mesh vertices and optimize the mesh structure; The texture mapping generation unit is used to drive texture generation through a dynamic reflectivity field, ultimately generate a texture with spectral information, and map the texture onto a three-dimensional mesh model; thereby generating a final 3D OBJ model with texture and spectral information.

15. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed on a computer, enables the computer to execute a fully automatic, fast, high-precision three-dimensional reconstruction method as described in any one of claims 1 to 10.

16. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, used to execute the computer program to implement a fully automatic, fast, high-precision three-dimensional reconstruction method as described in any one of claims 1 to 10.

Images