Fully automated, high-speed, and highly accurate three-dimensional reconstruction method and system

Abstract

This invention discloses a fully automated, high-speed, and highly accurate three-dimensional reconstruction method and system, relating to the field of three-dimensional dynamic reconstruction. The invention introduces the construction of a spectral-point cloud fusion model and the calculation of a dynamic reflectance field by Monte Carlo ray tracing. By combining this with Fresnel's equation and dynamic ray tracing techniques, the reflectance calculation conforms to physical laws, thereby improving the accuracy of spectral mapping. Furthermore, the invention optimizes spectral data based on an error correction matrix using a spectral error compensation network, reducing the accumulation of errors caused by factors such as measurement angle and surface roughness. Moreover, in the three-dimensional reconstruction process, the invention improves the continuity and spectral consistency of the three-dimensional model by using Poisson surface reconstruction and a spectral reflectance-driven mesh optimization method based on spectrally enhanced point cloud data.

Description

[Technical Field]

[0001] The present invention relates to the field of three-dimensional dynamic reconstruction, and more specifically, to a fully automated, high-speed, and highly accurate three-dimensional reconstruction method and system. [Background technology]

[0002] Currently, three-dimensional reconstruction technology is widely applied in multiple fields such as computer vision, reverse engineering, medical imaging, and cultural heritage preservation. Its main purpose is to acquire the geometric structure of an object using various imaging and measurement methods and reconstruct a highly accurate three-dimensional model. Conventional three-dimensional reconstruction methods mainly include stereo vision-based methods, structural optical scanning, laser radar (LiDAR), and multi-view geometric reconstruction. Stereo vision-based methods rely on images taken from different angles by multiple cameras, and calculate depth information of the object using triangulation by matching feature points within the same scene, ultimately constructing a three-dimensional model.

[0003] However, despite the remarkable advancements in current 3D reconstruction technology, it still faces many challenges, particularly in complex scenes or when high-precision reconstruction is required. Existing methods have the following problems: First, complete automation is not possible, requiring human intervention during the reconstruction process and manual adjustment of the equipment. Even after reconstruction is complete, texture application must be performed using human-machine interaction, which is extremely cumbersome. Furthermore, it is not possible to reconstruct the texture of the object simultaneously with reconstructing its geometric outline, resulting in only a white geometric model, which is unfavorable for observing the object. In addition, many conventional 3D reconstruction methods focus only on the geometric form of the object, ignoring the acquisition and use of spectral information. Since the color, material, and lighting conditions of the object's surface significantly affect the 3D reconstruction results, reconstruction methods lacking spectral information are susceptible to the influence of ambient light and the reflective properties of the surface material, resulting in visual effects of the reconstructed model that are not sufficiently realistic, and even causing errors in the geometric form. Next, point cloud data based on technologies such as structural optics and LiDAR often contains noise, and in particular, for special materials such as low reflectivity, transparent or metallic surfaces, it is difficult to effectively remove noise and optimize point cloud quality using conventional methods. Furthermore, because precise modeling of the reflectivity of the object surface is lacking in the three-dimensional reconstruction process, existing methods usually cannot truly simulate the spectral reflectance characteristics of an object under different lighting conditions, which affects rendering effects and the physical consistency of the model. At the same time, point cloud-based surface reconstruction methods still have certain limitations in terms of mesh generation and optimization, which can lead to problems such as surface unevenness and discontinuities in topological structure in the final three-dimensional model. Therefore, improving existing technologies so that three-dimensional reconstruction not only accurately captures geometric information but also effectively combines spectral information to improve reconstruction accuracy and visual effects is an important direction in current research. [Overview of the Initiative]

[0004] This invention proposes a fully automated, high-speed, and highly accurate three-dimensional reconstruction method and system, effectively compensating for the shortcomings of conventional methods and improving the spectral accuracy, reflectance consistency, and dynamic adaptability of three-dimensional reconstruction.

[0005] A fully automated, high-speed, and highly accurate three-dimensional reconstruction method, including the following steps S1 to S3, Step S1: Hyperspectral imaging and RGB imaging are used to synchronously collect spectral and image data from the surface of the target object. Image-driven point cloud data is obtained by combining SFM technology and depth estimation algorithms. Coordinate alignment of spectral data and point cloud data is performed using a Joint Calibration Model. Preprocessing of the aligned data is performed using adaptive noise suppression and iterative nearest neighbor method to obtain a spectral point cloud dataset. Step S2: A spectral-point cloud fusion model is constructed based on the spectral point cloud dataset, point cloud rendering is performed using 3D Gaussian Splatting technology, the dynamic reflectance field of the target object surface is calculated by Monte Carlo ray tracing, the spectral mapping error of the dynamic reflectance field is corrected by combining it with a spectral error compensation network, and spectrally enhanced point cloud data is formed by mapping the corrected spectral data to the point cloud coordinate system using the spectral-point cloud fusion model. Step S3: Generate a continuous surface by Poisson surface reconstruction based on the spectral feature point cloud data, adjust the triangular mesh structure by spectral reflectance-driven mesh optimization to obtain a three-dimensional mesh model, generate texture information with details of the object surface based on the image data and the surface characteristics of the spectral feature point cloud, drive texture generation by the dynamic reflectance field, finally generate a texture with spectral information, map the texture to the three-dimensional mesh model, and generate a final 3D OBJ model with texture and spectral information. A three-dimensional reconstruction method.

[0006] Furthermore, a fully automatic, high-speed, and high-precision three-dimensional reconstruction system is provided. This system is realized based on the fully automatic, high-speed, and high-precision three-dimensional reconstruction method described in any of the above. A data collection module that combines hyperspectral imaging and RGB imaging with SFM technology to synchronously collect spectral data and point cloud data of the target object surface, perform coordinate alignment between the spectral data and the point cloud data by an integrated calibration model, and perform preprocessing on the aligned data by adaptive noise suppression and the iterative closest point method to obtain a spectral point cloud data set. A data processing module that constructs a spectral-point cloud fusion model based on the spectral point cloud data set and obtains spectral feature point cloud data by combining the calculation of the dynamic reflectance field and the spectral error compensation network. A data reconstruction module that generates a continuous surface by Poisson surface reconstruction based on the spectral feature point cloud data and adjusts the triangular mesh structure by spectral reflectance-driven mesh optimization. including, wherein the data processing module specifically includes a dynamic reflectance field construction unit for calculating a dynamic reflectance field on the surface of the target object by Monte Carlo ray tracing, a spectral error compensation network correction unit for correcting the spectral mapping error of the dynamic reflectance field, and a spectral enhancement point cloud data output unit for mapping spectral data to a point cloud coordinate system by a spectral-point cloud fusion model to form spectral enhancement point cloud data.

[0007] A computer-readable storage medium used for storing a computer program, which causes a computer to execute the fully automatic, high-speed, and high-precision three-dimensional reconstruction method described in any of the above when the computer program is executed on the computer.

[0008] A memory for storing a computer program, A processor for realizing the fully automatic, high-speed, and high-precision three-dimensional reconstruction method described in any of the above by executing the computer program, An electronic device including.

[0009] The beneficial effects of the present invention are as follows. (1) The present invention constructs a spectral-point cloud fusion model, introduces Monte Carlo ray tracing to calculate a dynamic reflectance field, and further combines Fresnel's formula and dynamic ray tracing technology to adapt the reflectance calculation to physical laws and improve the accuracy of spectral mapping. Also, to further reduce spectral errors, the present invention uses a spectral error compensation network to optimize and adjust spectral data based on an error correction matrix, thereby reducing the accumulation of errors caused by factors such as measurement angle and surface roughness. (2) In the three-dimensional reconstruction process, the present invention improves the continuity of the three-dimensional model and spectral consistency by employing Poisson surface reconstruction and a spectral reflectance-driven mesh optimization method based on spectrally enhanced point cloud data. Compared with conventional Delaunay triangulation or direct interpolation methods of point clouds, the present invention utilizes spectral data as an optimization constraint and ensures that the mesh structure matches the spectral reflectance characteristics of the actual object, thereby providing more realistic and highly accurate three-dimensional reconstruction results in dynamic scenes. (3) The present invention simultaneously captures images of a target object from all 360 degrees using multiple cameras, removes the background from the images using fully automated cropping technology based on artificial intelligence, and further performs three-dimensional reconstruction using the latest 3D Gaussian Splatting rendering technology, thereby achieving fully automated, high-speed, and highly accurate three-dimensional reconstruction. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows a flowchart of a fully automated, high-speed, and high-precision three-dimensional reconstruction method provided by an embodiment of the present invention. [Figure 2] This is a schematic diagram of the configuration of a fully automated, high-speed, and high-precision three-dimensional reconstruction apparatus provided by an embodiment of the present invention. [Modes for carrying out the invention]

[0011] The technical solutions of the present invention will be described in more detail below with reference to the attached drawings, but the scope of protection of the present invention is not limited to the following description.

[0012] To further clarify the object, technical solution, and advantages of the present invention, the present invention will be described in more detail with reference to the accompanying drawings and examples. The specific examples described herein are for illustrative purposes only and do not limit the present invention; the described examples represent only a portion of the present invention and do not encompass all examples. Typically, the components of the embodiments of the present invention shown and illustrated herein can be arranged and designed in various different configurations.

[0013] Accordingly, the detailed description of embodiments of the present invention shown in the drawings below is not intended to limit the claimed scope of protection of the present invention, but merely to illustrate selected embodiments of the present invention. All other embodiments that can be obtained by those skilled in the art based on embodiments of the present invention without creative effort are included within the scope of protection of the present invention. The relational terms such as "first," "second," etc., are used merely to distinguish one entity or action from another entity or action, and do not necessarily require or suggest that there is an actual relationship or sequence between these entities or actions.

[0014] Furthermore, “includes,” “incorporates,” or any variation thereof implies non-exclusive inclusion; therefore, a process, method, article, or apparatus that includes a set of elements does not include only those elements, but may also include other elements not expressly enumerated, or elements specific to that process, method, article, or apparatus. Moreover, unless otherwise specified, an element defined as “including…” does not preclude the existence of further identical elements in a process, method, article, or apparatus that includes that element.

[0015] The features and performance of the present invention will be described in more detail below with reference to examples.

[0016] Example 1 As shown in Figure 1, the fully automated, high-speed, and highly accurate three-dimensional reconstruction method includes the following steps. S1. By combining hyperspectral imaging and RGB imaging with SFM technology, spectral data and point cloud data of the target object surface are acquired synchronously. Coordinate alignment of the spectral data and point cloud data is performed using an integrated calibration model. Preprocessing of the aligned data is performed using adaptive noise suppression and iterative nearest neighbor method to acquire a spectral point cloud dataset. S2. A spectral-point cloud fusion model is constructed based on the spectral point cloud dataset, point cloud rendering is performed using 3D Gaussian Splatting technology, the dynamic reflectance field of the target object surface is calculated by Monte Carlo ray tracing, the spectral mapping error of the dynamic reflectance field is corrected by combining it with a spectral error compensation network, and spectral-enhanced point cloud data is formed by mapping the spectral data to a point cloud coordinate system using the spectral-point cloud fusion model. S3. Based on the spectrally enhanced point cloud data, a continuous surface is generated by Poisson surface reconstruction, the triangular mesh structure is adjusted by spectral reflectance-driven mesh optimization, texture information with details of the object surface is generated based on the image data and surface characteristics of the spectrally enhanced point cloud, texture generation is driven by the dynamic reflectance field, and finally a texture with spectral information is generated and mapped to a three-dimensional mesh model, thereby generating a final 3D OBJ model with texture and spectral information.

[0017] Specifically, the implementation principle of the above embodiment is as follows. S1. By using hyperspectral imaging and RGB imaging, spectral and image data of the target object surface are acquired synchronously. Image-driven point cloud data is acquired by combining SFM technology and depth estimation algorithms. Coordinate alignment of spectral data and point cloud data is performed using an integrated calibration model. A spectral point cloud dataset is obtained by performing adaptive noise suppression and iterative nearest neighbor preprocessing on the aligned data. Preferably, since noise interference and coordinate errors exist during the acquisition process, noise in the spectral data and point cloud data is further removed using an adaptive noise suppression method, and the coordinate alignment accuracy is optimized using an iterative nearest neighbor (ICP) algorithm to finally form a high-quality spectral point cloud dataset.

[0018] S2. A spectral-point cloud fusion model is constructed based on the spectral point cloud dataset, point cloud rendering is performed using 3D Gaussian Splatting technology, the dynamic reflectance field of the target object surface is calculated by Monte Carlo ray tracing, and the spectral mapping error of the dynamic reflectance field is corrected by combining it with a spectral error compensation network. The corrected spectral data is then mapped to the point cloud coordinate system using the spectral-point cloud fusion model to form spectrally enhanced point cloud data. Since the spectral reflectance characteristics of an object surface are related not only to the material itself but also to the influence of ambient lighting, the present invention calculates the dynamic reflectance field using the Monte Carlo ray tracing method and estimates the reflectance distribution at different wavelengths by simulating the process of multiple reflections and scattering of light rays at the object surface. However, due to the complexity of the ray propagation model and the influence of measurement errors, a certain amount of error may occur in the mapping of spectral data. Therefore, a spectral error compensation network is introduced and trained to minimize the error in spectral data and output an error correction matrix. The aforementioned error correction matrix is ​​used to compensate for spectral mapping errors in the dynamic reflectance field, thereby improving the fusion accuracy of spectral data and point cloud data. The spectral-point cloud fusion model maps the corrected spectral data to the point cloud coordinate system, forming spectrally enhanced point cloud data so that each point in the point cloud data contains accurate spectral information.

[0019] S3. Based on the spectrally enhanced point cloud data, a continuous surface is generated by Poisson surface reconstruction, and the triangular mesh structure is adjusted by spectral reflectance-driven mesh optimization to obtain a three-dimensional mesh model. Furthermore, texture information with details of the object surface is generated based on the surface characteristics of the image data and spectrally enhanced point cloud, texture generation is driven by a dynamic reflectance field, and finally a texture with spectral information is generated and mapped to the three-dimensional mesh model to generate a 3D OBJ model with texture and spectral information. However, due to the discreteness of the point cloud data, geometric errors may occur in the directly reconstructed surface. Therefore, the present invention further introduces a spectral reflectance-driven mesh optimization strategy to adjust the triangular mesh structure. In the mesh optimization process, first, the normal vector of each point in the spectrally enhanced point cloud data is calculated, and an initial triangular mesh is constructed based on the normal vector. Furthermore, the Poisson-reconstructed surface is discretized into triangular patches using the Delaunay triangulation method, and the spectral reflectance value of each patch is determined by the spectral information of the adjacent point cloud data. Preferably, to improve the spectral consistency and geometric accuracy of the model, an optimization algorithm using the Lagrange multiplier method is used to match the normal vectors of the triangular patches with the normal vectors of the point cloud data, and the mesh vertex positions are adjusted by iterative optimization to ultimately obtain a highly accurate spectrally enhanced three-dimensional reconstruction model.

[0020] Furthermore, the specific implementation principle flow for obtaining the spectral point cloud dataset in step S1 is as follows: Each point cloud data point is assumed to have spatial coordinates P=(x,y,z), and its corresponding spectral data is assumed to be λ. The spectral data is mapped to a low-dimensional space by principal component analysis, and its geometric structure is preserved. Based on this mapping, the spectral data and point cloud coordinate data are combined to generate a fused spectral point cloud dataset. Preferably, deep learning techniques (e.g., multimodal neural networks) are used to co-learn the point cloud data and spectral data, learning the complex relationship between them. This method allows for automatic adjustment of the weights of spectral features based on the geometric features of the point cloud, so that the fused result more accurately reflects the actual optical properties of the object. The spectral point cloud dataset obtained by the method described above includes coordinate values ​​and spectral values. The coordinate values ​​represent the spatial position of a point on the object's surface, are described by three-dimensional coordinates (x, y, z), and represent the object's geometric shape and structural information. On the other hand, the spectral values ​​represent the reflection characteristics of the object's surface in different wavelength bands and reflect the absorption and reflection characteristics of the material in different spectral regions. Furthermore, by fusing the point cloud data and spectral data, a comprehensive dataset containing spatial information (i.e., surface shape) and spectral information (i.e., material and surface reflectance) is obtained. This dataset enables more accurate surface analysis, defect detection, material classification, and surface property evaluation of objects.

[0021] Furthermore, the specific process for calculating the dynamic reflectance field of the target object surface by Monte Carlo ray tracing in step S2 includes the following substeps. S2011. Based on Fresnel's equation, the ray-tracing reflectance of each point cloud data is calculated, and the ray-tracing reflectance of all point cloud data on the surface of the target object is aggregated to obtain the initial reflectance field. S2012. Based on the surface material type of the target object, the reflectance coefficients at different wavelengths for that material type are set. S2013. The dynamic reflectance field is calculated based on the set reflectance coefficient and the initial reflectance field.

[0022] Furthermore, Fresnel's equation can be expressed specifically as follows:

number

[0023] For example, metal surfaces have strong specular reflectivity, and their reflectivity changes with the angle of incidence; therefore, the Verbrante model is used to calculate the reflection and reflectivity of light rays. On the other hand, ceramic surfaces exhibit strong diffuse reflection, so calculations are performed based on the Lambertian reflection model.

[0024] Furthermore, step S2011 specifically includes the following substeps. S20111. Calculate the normal vector for each point cloud data. S20112. Set the directions of incident and reflected rays for each point cloud data, calculate the incident angle based on the normal and ray direction, and calculate the reflection angle based on the angle between the reflected ray and the surface normal. S20113. Based on the calculated angle of incidence and angle of reflection, the ray tracing reflectance of each point cloud data is calculated using Fresnel's formula. S20114. The reflectance of each point is aggregated to obtain the initial reflectance field of the entire surface of the target object.

[0025] Furthermore, the specific processing flow for calculating the dynamic reflectance field in step S2013 is expressed as follows. R dyn (λ,P)=a λ · R(λ,P) Here, R dyn (λ,P) represents the dynamic reflectance field of the point cloud data P at wavelength λ, where λ is the wavelength and P is the spatial coordinate of the point cloud data. λ The `R(λ,P)` parameter represents the reflectance coefficients of the target object's surface at different wavelengths depending on the material type, while `R(λ,P)` represents the initial reflectance field of the point cloud data P at wavelength λ. Specifically, the initial reflectance field R(λ,P) is a function of spatial coordinates and wavelength, and depends on the material attributes.

[0026] Furthermore, R(λ,θ) represents the reflectance at a certain wavelength and angle of incidence, and is used to describe the reflective properties in the interaction between light and the surface. In the above example, it is based on the optical properties of the material. On the other hand, R dyn (λ,P) represents the spectral reflectance at each point on the surface of the object, i.e., the reflectance characteristic exhibited by a certain wavelength at a given spatial position. Furthermore, each point on the surface must be associated with an angle of incidence, which is related to the propagation path of the light ray and the surface normal. The reflectance depends on the angle of incidence and the angle of reflection, and both of these angles are related to the surface normal of each point. For each point on the surface, its normal vector is obtained from point cloud data. For each point, the reflectance at a certain wavelength at that point is calculated using the spectral reflectance formula based on the normal direction of the point, the direction of the incident ray, and the direction of the reflected ray, and is specifically expressed by the following formula.

number

number

number

[0027] Furthermore, the process in step S2 to correct the spectral mapping error of the dynamic reflectance field by combining a spectral error compensation network specifically includes the following substeps. S2021. With the optimization goal of minimizing the error in spectral data, we define the error function of the spectral error compensation network and train the spectral error compensation network by minimizing the error function. S2022. The error correction matrix is ​​output using the trained spectral error compensation network. S2023. Based on the error correction matrix, the spectral data of each point cloud data point is corrected.

[0028] Specifically, the spectral error compensation network is trained using deep learning techniques and learns how to correct errors by comparing them with true reflectance data. The input to the network is the original spectral reflectance data, and the output is the spectral data after error compensation. For each reflectance value in the point cloud data, the spectral error compensation network identifies the error based on its spectral characteristics and automatically adjusts the corresponding value. For example, if the reflectance value is excessively high or low in a certain wavelength band, the network adjusts that value based on the compensation strategy acquired during the training process, thereby improving the accuracy of the spectral data.

[0029] Furthermore, in step S2021, the error function of the spectral error compensation network is expressed by the following equation.

number

[0030] Furthermore, in the step S2022, the error correction matrix is represented by the following formula.

Number

[0031] Specifically, the error correction matrix is used to correct the spectral values at each wavelength and is defined as follows. That is, the error correction matrix M is an n×n matrix where n represents the n wavelength data measured at each point. Each element n ij of the matrix represents the correction coefficient of the spectral data at the i-th wavelength with respect to the spectral data at the j-th wavelength. Exemplarily, when n = 3, the correction matrix M and the original spectral data S raw (P i ) are as follows.

Number

Number

[0032] Furthermore, the spectral enhanced point cloud data includes the point cloud coordinates and the spectral reflectivity of each point and is represented by the following formula.

Number

[0033] Furthermore, step S3 specifically includes the following substeps. S301. Calculate the normal vector for each point in the spectrally enhanced point cloud data. S302. A three-dimensional surface is generated using the Poisson reconstruction algorithm based on the normal vector data of the point cloud. S303. The Poisson-reconstructed surface is discretized into a triangular mesh by Delaunay triangulation, and the triangular mesh is composed of triangular patches, each patch corresponding to three adjacent points in the point cloud, and each point has a spectral reflectance value. S304. The objective function of spectrum-driven optimization is to make the normal vector of each triangular patch match the normal vector of the corresponding point in the point cloud data, and the objective function is minimized using an optimization algorithm based on the Lagrange multiplier method. S305. Optimize the mesh structure by iteratively updating the positions of the mesh vertices.

[0034] Furthermore, the objective function of the spectrum-driven optimization in step S304 is expressed by the following equation.

number

[0035] Furthermore, as a preferred embodiment in the above example, a verification and post-processing method for the optimized mesh is proposed, which includes the following steps. S401. The geometric quality of the optimized mesh is verified based on the mesh smoothness index and triangular shape quality, and the Delaunay property and side length ratio are used as evaluation criteria. S402. Spectral consistency is verified by calculating the error between the spectral reflectance of each patch and the reflectance of adjacent point cloud data points. S403. Output the optimized mesh as a 3D mesh file.

[0036] Furthermore, the specific flow for verifying the geometric quality of the optimized mesh based on the mesh smoothness index and triangular shape quality in step S401 is as follows. The geometric accuracy of the surface reconstruction is verified by calculating the error between the mesh vertices and the original point cloud data, and is specifically expressed by the following formula.

number

[0037] Furthermore, the specific flow of step S402 is represented by the following formula.

number

[0038] Example 2 Furthermore, as a preferred embodiment in the above example, a fully automated, high-speed, and high-precision three-dimensional reconstruction system is proposed, which includes a data acquisition module, a data processing module, and a data reconstruction module. The data acquisition module combines hyperspectral imaging and RGB imaging with SFM technology to synchronously collect spectral and point cloud data from the surface of a target object. It then performs coordinate alignment of the spectral and point cloud data using an integrated calibration model, and preprocesses the aligned data using adaptive noise suppression and iterative nearest neighbor method to obtain a spectral point cloud dataset. The data processing module constructs a spectral-point cloud fusion model based on the spectral point cloud dataset, calculates the dynamic reflectance field of the target object surface by Monte Carlo ray tracing, corrects the spectral mapping error of the dynamic reflectance field in combination with a spectral error compensation network, and maps the corrected spectral data to a point cloud coordinate system using the spectral-point cloud fusion model to form spectrally enhanced point cloud data. The data reconstruction module generates a continuous surface by Poisson surface reconstruction based on the spectrally enhanced point cloud data, and adjusts the triangular mesh structure by spectral reflectance-driven mesh optimization.

[0039] Furthermore, the data processing module includes a dynamic reflectance field construction unit, a spectral error compensation network correction unit, and a spectral enhancement point cloud data output unit. The dynamic reflectance field construction unit further includes an initial reflectance field calculation subunit, a reflectance coefficient calculation subunit, and a dynamic reflectance field calculation subunit. The initial reflectance field calculation subunit calculates the reflectance in the ray tracing of each point cloud data based on Fresnel's equation, and obtains the initial reflectance field by aggregating the ray tracing reflectances of all point cloud data on the surface of the target object. The reflectance coefficient calculation subunit sets the reflectance coefficients for different wavelengths based on the material type of the target object's surface. The dynamic reflectance field calculation subunit calculates the dynamic reflectance field based on the set reflectance coefficient and the initial reflectance field.

[0040] The specific flow for calculating the dynamic reflectance field is expressed by the following equation.

number

[0041] Furthermore, the spectral error compensation network correction unit includes an optimization goal definition subunit, a network learning subunit, and an error correction subunit. The optimization goal definition subunit defines the error function of the spectral error compensation network with the optimization goal of minimizing the error in spectral data, and trains the network by minimizing the error function. The network learning subunit outputs an error correction matrix using the trained spectral error compensation network. The error correction subunit corrects the spectral data of each point cloud data point based on the error correction matrix.

[0042] Furthermore, the error function of the spectral error compensation network is expressed by the following equation.

number

[0043] Furthermore, the error correction matrix is ​​expressed by the following formula.

number

[0044] Furthermore, the spectrally enhanced point cloud data includes the point cloud coordinates and the spectral reflectance of each point, and is expressed by the following formula.

number

[0045] Furthermore, the data reconstruction module includes a normal vector calculation unit, a three-dimensional reconstruction unit, a construction processing unit, a spectrum-driven optimization unit, an update iteration optimization unit, and a texture mapping generation unit. The normal vector calculation unit calculates the normal vector for each point in the spectrally enhanced point cloud data. The three-dimensional reconstruction unit generates a three-dimensional surface using the Poisson reconstruction algorithm based on the normal vector data of the point cloud. The construction processing unit discretizes the Poisson reconstruction surface into a triangular mesh using Delaunay triangulation, and the triangular mesh is composed of triangular patches, each patch corresponding to three adjacent points in the point cloud, and each point has a spectral reflectance value. The spectrum-driven optimization unit sets the objective function of spectrum-driven optimization as matching the normal vector of each triangular patch with the normal vector of the corresponding point in the point cloud data, and minimizes this objective function using an optimization algorithm based on the Lagrange multiplier method. The update iterative optimization unit optimizes the mesh structure by iteratively updating the positions of the mesh vertices. The texture mapping generation unit drives texture generation using a dynamic reflectance field, ultimately generating a texture with spectral information, and then maps this texture to a three-dimensional mesh model to generate a final 3D OBJ model containing both the texture and spectral information.

[0046] Furthermore, the objective function of the spectrum-driven optimization is expressed by the following equation.

number

[0047] Example 3 Based on Example 1, this embodiment proposes a fully automated, high-speed, and high-precision three-dimensional reconstruction apparatus. The schematic hardware design of the apparatus is shown in Figure 2. It acquires target images at high speed through synchronous acquisition by multiple cameras, achieves high depth-of-field imaging through stacking acquisition, reduces background interference by performing automatic cutout using artificial intelligence, further accelerates the reconstruction process, and achieves high-precision three-dimensional reconstruction of the target.

[0048] The operation flow is as follows: 1. Multiple vertically positioned auto-zoom and auto-focus cameras synchronously acquire vertical 360° images of the target object. By rotating a turntable, horizontal 360° images of the target object are acquired, achieving 720° omnidirectional imaging. An artificial intelligence algorithm automatically removes the background, ensuring that the image data contains only the target object. SFM (Structure-from-Motion) technology is used to restore the camera orientation at the time of image acquisition. Multi-view geometric matching calculates a sparse three-dimensional point cloud of the target object. Point cloud density is optimized in combination with a depth estimation algorithm to acquire image-driven, high-precision point cloud data. Hyperspectral imaging synchronously acquires spectral information of the object surface. An integrated calibration model aligns spectral data with image point cloud coordinates to form a spectral point cloud dataset. 2. Construct a spectral-point cloud fusion model and build a spectrally enhanced point cloud model based on the mapping relationship between point cloud coordinates and spectral data. Perform point cloud rendering using optimized 3D Gaussian Splatting technology to ensure that the spectral data is uniformly distributed in three-dimensional space. 3. The dynamic reflectance field (D-SRF) of the target object surface is calculated using Monte Carlo ray tracing. An error correction matrix is ​​calculated in combination with a spectral error compensation network to optimize spectral mapping accuracy. 4. A spectral-point cloud fusion model is used to map spectral data to a point cloud coordinate system, forming high-precision spectrally enhanced point cloud data. 5. The spectrally emphasized point cloud is converted into a continuous 3D surface model using a Poisson surface reconstruction algorithm. The mesh structure is adjusted using spectral reflectance-driven triangular mesh optimization to ensure that the spectral data is more uniform on the geometric surface. Based on the image data (obtained in the image acquisition process) and the surface characteristics of the spectrally emphasized point cloud in the above step, texture information with details of the object surface is generated. The image data and spectral data are combined, and texture generation is driven by a dynamic reflectance field (corrected spectral reflectance data) to ensure consistency between the visual effect of the texture and the spectral information. Finally, a texture with spectral information is generated, and this texture is mapped to the three-dimensional mesh model to generate a final 3D OBJ model with texture and spectral information.

[0049] Example 4 Based on Example 1, this embodiment proposes a fully automated, high-speed, and high-precision three-dimensional reconstruction terminal device. The terminal device includes at least one memory, at least one processor, and a bus for connecting different platform systems.

[0050] The memory may include a readable medium in the form of volatile memory such as RAM211 and / or cache memory, and may also include ROM213.

[0051] Here, a computer program is further stored in the memory, and the computer program is executed by the processor, causing the processor to perform any one of the fully automatic, high-speed, and high-precision three-dimensional reconstruction methods described in the embodiments of this application. The specific implementation is consistent with the implementation and technical effects achieved in the embodiments of the above method, so some redundant descriptions are omitted. The memory may also further include a set (at least one) of program modules of program / utility tools, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each or any combination of these examples may include an implementation of a network environment.

[0052] In response to this, the processor can execute the aforementioned computer programs, and can also run programs / utility tools.

[0053] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerator port, a processor, or a local bus using any of the several types of bus structures.

[0054] The terminal device may communicate with one or more external devices such as keyboards, pointing devices, and Bluetooth devices, as well as with one or more devices that can interact with the terminal device, and / or with any device that enables the terminal device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication can be performed via an I / O interface. Furthermore, the terminal device may communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks such as the Internet) via a network adapter. The network adapter may communicate with other modules of the terminal device via a bus. It should be understood that, although not illustrated, other hardware and / or software modules, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, may be used in combination with the terminal device.

[0055] Example 5 Based on Example 1, this embodiment proposes a fully automated, high-speed, and high-precision computer-readable storage medium for three-dimensional reconstruction. Instructions are stored in this computer-readable storage medium, and when these instructions are executed by a processor, one of the above fully automated, high-speed, and high-precision three-dimensional reconstruction methods is realized. The specific implementation method is consistent with the implementation method and technical effects achieved in the embodiment of the above method, so some redundant descriptions are omitted.

[0056] This embodiment provides a program product for realizing the above method, which employs a portable compact disc read-only memory (CD-ROM), includes program code, and can be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this embodiment, the readable medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not an exhaustive list) of readable storage media include electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any appropriate combination of the above.

[0057] A computer-readable storage medium may contain data signals propagated within the baseband or as part of a carrier, the data signals carrying readable program code. Such propagated data signals may take multiple forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may be any other readable medium, which can transmit, propagate, or transmit programs for use by or in combination with instruction execution systems, apparatus, or devices. The program code contained on the readable storage medium may be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof. The program code for performing the operation of the present invention may be written in any combination of one or more programming languages, which include object-oriented programming languages ​​such as Java and C++, and also include conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computing device, or entirely on a remote computing device or server. If a remote computing device is included, it may be connected to the user's computer via any type of network, including a LAN or WAN, or it may be connected to an external computing device (for example, via the Internet using an Internet service provider).

[0058] The above description is merely a preferred embodiment of the present invention, and it should be understood that the present invention is not limited to the forms disclosed herein. It should not be considered as excluding other embodiments, which can be applied to various other combinations, modifications, and environments, and can be modified within the scope of the ideas described herein, based on the above teachings or the art or knowledge of the relevant technical field. Modifications and alterations by those skilled in the art should all be within the scope of protection of the claims appended to the present invention, provided that they do not deviate from the spirit and scope of the present invention.

Claims

1. A fully automated, high-speed, and highly accurate three-dimensional reconstruction method, comprising the following steps S1 to S3, Step S1: Hyperspectral imaging and RGB imaging are used to synchronously collect spectral and image data of the target object surface. Image-driven point cloud data is acquired by combining SFM technology and a depth estimation algorithm. Coordinate alignment of spectral data and point cloud data is performed using an integrated calibration model. Preprocessing of the aligned data is performed using adaptive noise suppression and iterative nearest neighbor method to acquire a spectral point cloud dataset. Step S2: A spectral-point cloud fusion model is constructed based on the spectral point cloud dataset, point cloud rendering is performed using 3D Gaussian Platting technology, the dynamic reflectance field of the target object surface is calculated by Monte Carlo ray tracing, the spectral mapping error of the dynamic reflectance field is corrected in combination with a spectral error compensation network, and the corrected spectral data is mapped to the point cloud coordinate system by the spectral-point cloud fusion model to form spectrally enhanced point cloud data. Step S3: A three-dimensional reconstruction method characterized by generating a continuous surface by Poisson surface reconstruction based on the spectrally emphasized point cloud data, adjusting the triangular mesh structure by spectral reflectance-driven mesh optimization, obtaining a three-dimensional mesh model, generating texture information having details of the object surface based on the surface characteristics of the image data and spectrally emphasized point cloud, driving texture generation by the dynamic reflectance field, finally generating a texture having spectral information, mapping the texture to the three-dimensional mesh model, and generating a final 3D OBJ model having the texture and spectral information.

2. In step S2, the step of calculating the dynamic reflectance field of the target object surface by Monte Carlo ray tracing includes the following steps S2011 to S2013: S2011: Based on Fresnel's equation, the reflectance of each point cloud data point trace is calculated, and the reflectance of all point cloud data points on the surface of the target object is aggregated to obtain the initial reflectance field. S2012: Based on the material type of the surface of the target object, set the reflectance coefficients for different wavelengths of the said material type. S2013: The three-dimensional reconstruction method according to claim 1, characterized in that a dynamic reflectance field is calculated based on a set reflectance coefficient and the initial reflectance field.

3. Step S2011 includes the following steps S20111 to S20114, S20111: Calculate the normal vector of each point cloud data, S20112: Set the direction of the incident and reflected rays for each point cloud data, calculate the incident angle based on the direction of the normal and incident rays, and calculate the reflection angle based on the angle between the reflected rays and the surface normal. S20113: Based on the calculated angle of incidence and angle of reflection, the reflectance of the ray tracing for each point cloud data is calculated using Fresnel's formula. S20114: The three-dimensional reconstruction method according to claim 2, characterized by aggregating the reflectance of each point to obtain an initial reflectance field of the entire surface of the target object.

4. In step S2013, the specific flow for calculating the dynamic reflectance field is given by the following equation: R ｄｙｎ (λ, P) = a λ ・R (λ, P) It is represented as, In the formula, R ｄｙｎ (λ, P) represents the dynamic reflectance field of the point cloud data P at wavelength λ, where λ is the wavelength and P is the spatial coordinate of the point cloud data, a λ The three-dimensional reconstruction method according to claim 2, characterized in that R(λ, P) represents the reflectance coefficient at different wavelengths for different surface material types of the target object, and R(λ, P) represents the initial reflectance field at wavelength λ of the point cloud data P.

5. In step S2, the step of correcting the spectral mapping error of the dynamic reflectance field in combination with a spectral error compensation network includes the following steps S2021 to S2023: S2021: With the optimization goal of minimizing the error in spectral data, the error function of the spectral error compensation network is defined, and the spectral error compensation network is trained by minimizing the error function. S2022: Output the error correction matrix using the trained spectral error compensation network. S2023: The three-dimensional reconstruction method according to claim 1, characterized in that spectral data of each point cloud data point is corrected based on the error correction matrix.

6. The error function of the spectral error compensation network in step S2021 is given by the following equation: [Math 1] is represented by the formula, where E ｏｐｔ represents the error function of the spectral error compensation network, M represents the error correction matrix, N represents the total number of point cloud data, i represents the point cloud index, and S ｒａｗ (P ｉ ) represents the original spectral data obtained from the history data corresponding to the i-th point cloud data P ｉ , and S ｔｒｕｅ (P ｉ ) represents the true spectral data obtained by actual measurement corresponding to the i-th point cloud data P ｉ . The three-dimensional reconstruction method according to claim 5, characterized in that

7. The error correction matrix in step S2022 is given by the following formula: [Math 2] The three-dimensional reconstruction method according to claim 5, characterized by being represented as follows.

8. The spectrally enhanced point cloud data includes the point cloud coordinates and the spectral reflectance of each point, and is expressed by the following formula: [Math 3] It is expressed as, in the formula, P ｆｉｎａｌ The x represents spectrally weighted point cloud data, ｉ , y ｉ , z ｉ These are point cloud data P, respectively. ｉ The three-dimensional coordinates are shown, S ｃｏｒｒｅｃｔｅｄ λ is the wavelength of the i-th point cloud data. ｍ The spectral reflectance at λ is shown. ｍ This indicates the mth wavelength, P ｉ The three-dimensional reconstruction method according to claim 1, characterized in that represents the i-th point cloud data.

9. In step S3, generating a continuous surface by Poisson surface reconstruction based on spectrally enhanced point cloud data and adjusting the triangular mesh structure by spectral reflectance-driven mesh optimization includes the following steps S301 to S305: S301: Calculate the normal vector of each point in the spectrally enhanced point cloud data. S302: Based on the normal vector data of the point cloud, a three-dimensional surface is generated using the Poisson reconstruction algorithm. S303: The Poisson reconstructed surface is discretized into a triangular mesh by Delaunay triangulation, the triangular mesh is composed of triangular patches, each patch corresponds to three adjacent points in the point cloud, and each point has a spectral reflectance value. S304: The objective function of spectrum-driven optimization is to match the normal vector of each triangular patch with the normal vector of the corresponding point in the point cloud data, and the objective function is minimized using an optimization algorithm based on the Lagrange multiplier method. S305: The three-dimensional reconstruction method according to claim 1, characterized by optimizing the mesh structure by iteratively updating the positions of mesh vertices.

10. The objective function for spectral-driven optimization in step S304 is given by the following equation: [Math 4] It is expressed as, in the formula, E ｏｐｔ，ｄｒｉｖｅｒ represents the objective function of spectrum-driven optimization, N(T) represents the normal vector of the triangular patch T, and N(P ｉ ) is the i-th point cloud data P ｉ The normal vector is shown, T represents the patch, and λ ０ This shows the influencing factors of spectral reflectance for mesh optimization, and S(λ) ｍ , P ｉ ) is point cloud data P ｉ The spectral reflectance at wavelength λm is shown, and S(λ) ｍ ,T) is the wavelength λ of patch T. ｍ The spectral reflectance at λ is shown. ｍ The three-dimensional reconstruction method according to claim 9, characterized in that represents the mth wavelength.

11. A fully automated, high-speed, and high-precision three-dimensional reconstruction system realized by a three-dimensional reconstruction method according to any one of claims 1 to 10, comprising a data acquisition module, a data processing module, and a data reconstruction module, The data acquisition module is configured to synchronously acquire spectral and image data of the target object surface using hyperspectral imaging and RGB imaging, acquire image-driven point cloud data by combining SFM technology and a depth estimation algorithm, and perform coordinate alignment of spectral data and point cloud data using an integrated calibration model. The data processing module is configured to construct a spectral-point cloud fusion model based on the spectral point cloud dataset, perform point cloud rendering using 3D Gaussian Platting technology, calculate the dynamic reflectance field of the target object surface by Monte Carlo ray tracing, correct the spectral mapping error of the dynamic reflectance field in combination with a spectral error compensation network, and map the corrected spectral data to a point cloud coordinate system using the spectral-point cloud fusion model, thereby forming spectrally enhanced point cloud data. The data reconstruction module is configured to generate a continuous surface by Poisson surface reconstruction based on the spectrally emphasized point cloud data, adjust the triangular mesh structure by spectral reflectance-driven mesh optimization, acquire a three-dimensional mesh model, generate texture information with details of the object surface based on the image data and surface characteristics of the spectrally emphasized point cloud, drive texture generation with the dynamic reflectance field, finally generate a texture with spectral information, map the texture to the three-dimensional mesh model, and generate a final 3D OBJ model with texture and spectral information. The data processing module is a three-dimensional reconstruction system characterized by including a dynamic reflectance field construction unit for calculating the dynamic reflectance field of the surface of a target object by Monte Carlo ray tracing, a spectral error compensation network correction unit for correcting spectral mapping errors of the dynamic reflectance field, and a spectral-enhanced point cloud data output unit for mapping spectral data to a point cloud coordinate system using a spectral-point cloud fusion model to form spectrally enhanced point cloud data.

12. The dynamic reflectance field construction unit includes an initial reflectance field calculation subunit, a reflectance coefficient calculation subunit, and a dynamic reflectance field calculation subunit. The initial reflectance field calculation subunit is configured to calculate the reflectance of each point cloud data point trace based on Fresnel's equation, and to obtain the initial reflectance field by aggregating the reflectances of all point cloud data points on the surface of the target object. The reflectance coefficient calculation subunit is configured to set the reflectance coefficient at different wavelengths for the material type based on the material type of the surface of the target object. The three-dimensional reconstruction system according to claim 11, characterized in that the dynamic reflectance field calculation subunit is configured to calculate a dynamic reflectance field based on a set reflectance coefficient and the initial reflectance field.

13. The spectral error compensation network correction unit includes an optimization goal definition subunit, a network learning subunit, and an error correction subunit. The aforementioned optimization goal definition subunit defines the error function of the spectral error compensation network with the optimization goal of minimizing the error in the spectral data, and is configured to train the spectral error compensation network by minimizing the error function. The network learning subunit is configured to output an error correction matrix using a trained spectral error compensation network. The three-dimensional reconstruction system according to claim 11, characterized in that the error correction subunit is configured to correct the spectral data of each point cloud data point based on an error correction matrix.

14. The data reconstruction module includes a normal vector calculation unit, a three-dimensional reconstruction unit, a construction processing unit, a spectrum-driven optimization unit, an update iterative optimization unit, and a texture mapping generation unit. The normal vector calculation unit is configured to calculate the normal vector of each point in the spectrally enhanced point cloud data. The three-dimensional reconstruction unit is configured to generate a three-dimensional surface using a Poisson reconstruction algorithm based on the normal vector data of the point cloud. The construction processing unit is configured to discretize the Poisson reconstruction surface into a triangular mesh by Delaunay triangulation, the triangular mesh is composed of triangular patches, each patch corresponds to three adjacent points in the point cloud, and each point has a spectral reflectance value. The spectral-driven optimization unit is configured to minimize the objective function of spectral-driven optimization, which is to match the normal vector of each triangular patch with the normal vector of the corresponding point in the point cloud data, using an optimization algorithm based on Lagrange multipliers. The update iterative optimization unit is configured to optimize the mesh structure by iteratively updating the positions of mesh vertices. The three-dimensional reconstruction system according to claim 11, characterized in that the texture mapping generation unit is configured to drive texture generation by a dynamic reflectance field, ultimately generate a texture including spectral information, map it to a three-dimensional mesh model, and generate a final 3D OBJ model having the texture and spectral information.

15. A computer-readable storage medium for storing computer programs, A computer-readable storage medium characterized in that, when the computer program is executed on the computer, the computer is made to execute the three-dimensional reconstruction method described in any one of claims 1 to 10.

16. Memory for storing computer programs, A processor for realizing the three-dimensional reconstruction method described in any one of claims 1 to 10 by executing the computer program, An electronic device characterized by including

Images