High-precision three-dimensional surface reconstruction method and device based on polarization information enhancement

Abstract

The embodiments of the present disclosure disclose a high-precision three-dimensional surface reconstruction method and apparatus based on polarization information enhancement. A specific implementation of the method includes: using a polarization camera to capture polarization images from multiple perspectives to obtain a polarization image set; performing structured light initialization processing on the target image to generate initialized target image information; inversely projecting the precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data; performing Gaussian point cloud dereflection processing on the initialized target image information to generate dereflected Gaussian point cloud data; generating Gaussian point cloud data based on the enhanced Gaussian point cloud data and the dereflected Gaussian point cloud data; and performing data optimization and reconstruction processing on the Gaussian point cloud data to generate three-dimensional triangular facets. This implementation can improve the accuracy and efficiency of three-dimensional surface reconstruction.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a high-precision three-dimensional surface reconstruction method and apparatus based on polarization information enhancement. Background Art

[0002] With the rise of neural radiance field technology, high-precision 3D scene rendering and mesh extraction have become possible by mapping continuous coordinates to color and volume density through multi-layer perceptrons. Currently, high-precision 3D surface reconstruction is typically achieved using methods based on structured light and multi-view stereo. These methods typically rely on photometric information and depth estimation from multi-view images. However, when reconstructing textureless and reflective areas, geometric details are lost or the surface reconstruction is incomplete.

[0003] However, in practice, it is found that when using the above method to reconstruct a high-precision three-dimensional surface, the following technical problems often occur:

[0004] In the reconstruction of textureless and reflective areas, the lack of photometric information makes it difficult to recover geometric details, resulting in missing or discontinuous surface reconstruction results. Especially when dealing with highly reflective surfaces, traditional methods have difficulty dealing with photometric inconsistencies, resulting in holes or incomplete models.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the Invention

[0006] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] Some embodiments of the present disclosure propose high-precision three-dimensional surface reconstruction methods, devices, electronic devices, and computer-readable media based on polarization information enhancement to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a high-precision three-dimensional surface reconstruction method based on polarization information enhancement, the method comprising: using a polarization camera to collect polarization images of multiple perspectives to obtain a polarization image set; performing linear decomposition processing on each polarization image in the polarization image set to generate polarization information to obtain a polarization information set, wherein the polarization information in the polarization information set includes: linear polarization angle and polarization degree; performing image fusion processing on each polarization image in the polarization image set to generate a target image; performing structured light initialization processing on the target image to generate initialized target image information, wherein the initialized target image information includes: camera posture parameters, initialized Gaussian point cloud data, image color value; performing polarization enhancement processing on each polarization information in the above polarization information set to generate enhanced polarization information and obtain an enhanced polarization information set; performing precise depth processing on the above enhanced polarization information set to generate a precise depth information set; inversely projecting the above precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data; based on the above polarization information set, the above image color value and the above target image, performing Gaussian point cloud de-reflection processing on the above initialized target image information to generate de-reflected Gaussian point cloud data; generating Gaussian point cloud data based on the above enhanced Gaussian point cloud data and the above de-reflected Gaussian point cloud data; performing data optimization and reconstruction processing on the above Gaussian point cloud data to generate a three-dimensional triangular facet.

[0009] In a second aspect, some embodiments of the present disclosure provide a high-precision three-dimensional surface reconstruction device based on polarization information enhancement, the device comprising: an acquisition unit, configured to use a polarization camera to acquire polarization images of multiple perspectives to obtain a polarization image set; a linear decomposition unit, configured to perform linear decomposition processing on each polarization image in the polarization image set to generate polarization information to obtain a polarization information set, wherein the polarization information in the polarization information set includes: linear polarization angle and polarization degree; a fusion unit, configured to perform image fusion processing on each polarization image in the polarization image set to generate a target image; an initialization unit, configured to perform structured light initialization processing on the target image to generate initialized target image information, wherein the initialized target image information includes: camera pose parameters, initialized Gaussian point cloud data, and image color values; a polarization enhancement unit, It is configured to perform polarization enhancement processing on each polarization information in the above-mentioned polarization information set to generate enhanced polarization information and obtain an enhanced polarization information set; the precise depth unit is configured to perform precise depth processing on the above-mentioned enhanced polarization information set to generate a precise depth information set; the back projection unit is configured to back-project the above-mentioned precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data; the de-reflection unit is configured to perform Gaussian point cloud de-reflection processing on the above-mentioned initialized target image information based on the above-mentioned polarization information set, the above-mentioned image color value and the above-mentioned target image to generate de-reflected Gaussian point cloud data; the generation unit is configured to generate Gaussian point cloud data based on the above-mentioned enhanced Gaussian point cloud data and the above-mentioned de-reflected Gaussian point cloud data; the optimization and reconstruction unit is configured to perform data optimization and reconstruction processing on the above-mentioned Gaussian point cloud data to generate three-dimensional triangular facets.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0012] The various embodiments disclosed above have the following beneficial effects: The high-precision 3D surface reconstruction method based on polarization information enhancement, as described in some embodiments of the present application, can improve the accuracy and efficiency of 3D surface reconstruction. Specifically, the phenomenon of holes or incomplete models is caused by the lack of photometric information in the reconstruction of textureless and reflective areas, making it difficult to recover geometric details, resulting in missing or discontinuous surface reconstruction results. This is especially true when processing highly reflective surfaces, where traditional methods struggle to cope with photometric inconsistencies. Based on this, the high-precision 3D surface reconstruction method based on polarization information enhancement in some embodiments of the present application first uses a polarization camera to capture polarization images from multiple perspectives to generate a polarization image set. This allows the polarization characteristics of the target object to be obtained, providing basic data for subsequent polarization information processing. Second, each polarization image in the polarization image set is linearly decomposed to generate polarization information, resulting in a polarization information set. The polarization information in the polarization information set includes linear polarization angle and degree of polarization. This allows further analysis and extraction of the geometric and physical properties of the target object. Third, each polarization image in the polarization image set is fused to generate a target image. This enhances the visual quality and information content of the image, providing richer data for subsequent 3D reconstruction. Next, the target image undergoes structured light initialization processing to generate initial target image information. This initial target image information includes camera pose parameters, initial Gaussian point cloud data, and image color values. This provides an accurate starting point and foundational data for 3D reconstruction. Next, polarization enhancement processing is performed on each polarization information in the polarization information set to generate enhanced polarization information, resulting in an enhanced polarization information set. This further improves the quality and usability of the polarization information. The enhanced polarization information set then undergoes precise depth processing to generate a precise depth information set. This allows for higher-precision depth data to be obtained, providing critical geometric information for 3D reconstruction. The precise depth information set is then inversely projected into 3D space to generate enhanced Gaussian point cloud data. This converts 2D image information into 3D spatial data, laying the foundation for subsequent 3D modeling. Finally, based on the polarization information set, the image color values, and the target image, Gaussian point cloud de-reflection processing is performed on the initial target image information to generate de-reflected Gaussian point cloud data. This eliminates the effects of reflections on 3D reconstruction, improving the accuracy and completeness of reconstruction. Gaussian point cloud data is generated based on the enhanced Gaussian point cloud data and the dereflected Gaussian point cloud data. This allows the integration of multiple data sources to create a more comprehensive 3D model. Data optimization and reconstruction processing are performed on the Gaussian point cloud data to generate 3D triangular facets. This completes the conversion from Gaussian point clouds to triangular facets, achieving high-precision 3D surface reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flowchart of some embodiments of a high-precision three-dimensional surface reconstruction method based on polarization information enhancement according to the present disclosure;

[0015] Figure 2 is a schematic structural diagram of some embodiments of a high-precision three-dimensional surface reconstruction device based on polarization information enhancement according to the present disclosure;

[0016] Figure 3 It is a structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1A process 100 of some embodiments of a high-precision three-dimensional surface reconstruction method based on polarization information enhancement according to the present disclosure is shown. The high-precision three-dimensional surface reconstruction method based on polarization information enhancement includes the following steps:

[0024] Step 101: Use a polarization camera to collect polarization images from multiple perspectives to obtain a polarization image set.

[0025] In some embodiments, an entity (e.g., a computing device) executing a high-precision 3D surface reconstruction method based on polarization information enhancement may use a polarization camera to capture polarization images from multiple perspectives, generating a polarization image set. The polarization images in the polarization image set may include, but are not limited to, 0-degree polarization images, 45-degree polarization images, 90-degree polarization images, and 135-degree polarization images. For example, the polarization camera may be a large constant image polarization camera.

[0026] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0027] Step 102 : performing linear decomposition processing on each polarization image in the polarization image set to generate polarization information, thereby obtaining a polarization information set.

[0028] In some embodiments, the execution entity may perform linear decomposition processing on each polarization image in the polarization image set using a preset vector to generate polarization information, thereby obtaining a polarization information set. The polarization information in the polarization information set includes: angle of linear polarization (AoLP) and degree of polarization (DoLP).

[0029] For example, the preset vector may be a Stokes vector. The Stokes vector can be expressed as follows:

[0030] stokes=(s0, s1, s2, s3), S0=I 0° +I 90° , S1=I 0° -I 90° , S2=I 45° -I 135° , S3=I 0° +I 90° -2I 45° .

[0031] The above linear polarization angle can be obtained by the following formula:

[0032] AoLP = 0.5*arctan(S2 / S1).

[0033] The above polarization degree can be expressed by the following formula:

[0034] DoLP=sqμt(S1*S1+S2*S2) / S0.

[0035] S0 represents the total intensity of light, which is the sum of the light intensities in all polarization directions. S1 represents the polarization difference between 0 and 90 degrees. S2 represents the polarization difference between 45 and 135 degrees. S3 represents the circularly polarized component of light. AoLP represents the angle between the polarization direction of light and the horizontal. DoLP represents the purity of the polarization state of light, that is, the degree of polarization of light.

[0036] Step 103 : performing image fusion processing on each polarization image in the polarization image set to generate a target image.

[0037] In some embodiments, the execution entity may perform image fusion processing on each polarization image in the polarization image set using a preset image fusion algorithm to generate a target image. For example, the preset image fusion algorithm may be a wavelet transform fusion algorithm.

[0038] Step 104 : performing structured light initialization processing on the target image to generate initialized target image information.

[0039] In some embodiments, the execution entity may perform structured light initialization processing on the target image using a preset structured light initialization algorithm to generate initialized target image information. The initialized target image information includes: camera pose parameters, initialized Gaussian point cloud data, and image color values. For example, the preset structured light initialization algorithm may be Structure from Motion technology. The camera pose parameters may include: the camera's rotation matrix and translation vector. The initialized Gaussian point cloud data may be: 25.5 points / m2. The image color values ​​may be: red, green, and blue.

[0040] In practice, the execution entity may perform structured light initialization processing on the target image through the following steps to generate initialized target image information:

[0041] The first step is to perform parameter matching on the target image to generate camera pose parameters. In practice, the execution entity can perform parameter matching on the target image to generate camera pose parameters using a preset parameter matching algorithm. The preset parameter matching algorithm can be a brute force matching algorithm.

[0042] The second step is to perform 3D reconstruction on the camera pose parameters to generate an initial point cloud. In practice, the execution entity may use a preset 3D reconstruction algorithm to perform 3D reconstruction on the camera pose parameters to generate the initial point cloud. The preset 3D reconstruction algorithm may be a triangulation algorithm.

[0043] The third step is to add Gaussian noise to the initial point cloud to generate initial Gaussian point cloud data. In practice, the execution entity can first use a preset function to create Gaussian noise. Then, the execution entity can add the Gaussian noise to each point in the initial point cloud to generate initial Gaussian point cloud data. For example, the preset function can be a random number generator function.

[0044] The fourth step is to perform color mapping on the target image to generate image color values. In practice, the execution subject can perform color mapping on the target image to generate image color values ​​using a preset color mapping algorithm. The preset color mapping algorithm can be a grayscale mapping algorithm.

[0045] In the fifth step, the camera pose parameters, the initialized Gaussian point cloud data and the image color values ​​are determined as the initialized target image information.

[0046] Step 105 : Perform polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information, thereby obtaining an enhanced polarization information set.

[0047] In some embodiments, the execution entity may perform polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information and obtain an enhanced polarization information set.

[0048] In practice, the execution entity may perform polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information, thereby obtaining an enhanced polarization information set:

[0049] The first step is to generate a preset depth group and a preset normal candidate value group based on the polarization information. The preset depths in the preset depth group can be depth values ​​for each Gaussian point extracted from the geometric model. The preset normal candidate values ​​in the preset normal candidate value group can be normal vectors for each Gaussian point extracted from the geometric model. For example, the geometric model can be a 3D Gaussian Splatting (3DGS) rendering model.

[0050] The following formula is used to generate a preset depth group and a preset normal candidate value group based on the polarization information:

[0051] H={(d,n), (dptb,n), (d,n_AoLPA i),(d_ptb,n_AoLP A i)}, i=1,2,3,a4.

[0052] Where d represents the depth value obtained from the geometric model. n represents the normal vector obtained from the geometric model. d_ptb is used to generate candidate depth values. n_AoLP A iCandidate normal values ​​derived from the angle of linear polarization (AoLP) and degree of polarization (DoLP).

[0053] In the second step, based on the preset depth group and the preset normal candidate value group, a polarization consistency score is generated. For example, the polarization consistency score may be 0.85.

[0054] The polarization consistency score is generated based on the preset depth group and the preset normal candidate value group using the following formula:

[0055] Sp(n * )=min{w·exp(-(φi-φ) * 2 / τ)}Sp(n * )=min{w·exp(-(φi-φ) * 2 / τ)}.

[0056] Where φ represents the azimuth of the currently assumed normal direction, φi represents the adjusted angle of linear polarization (AoLP), and τ represents the factor that controls sensitivity.

[0057] The third step is to combine the polarization consistency score with a preset photometric matching score to obtain a polarization cost function. The polarization cost function can be used to select candidate target depths and target normals for each pixel. For example, the preset photometric matching score can be 0.25.

[0058] The polarization cost function is obtained by combining the polarization consistency score and the preset photometric matching score using the following formula:

[0059] S=Sc+λ1*Sp+λ2*Sdn}.

[0060] Here, λ1 is a weight factor used to adjust the polarization consistency score, Sp, in the overall cost function. λ2 is used to adjust the normal and depth consistency scores, Sdn, in the overall cost function. Sc measures the color consistency of the same object point under different viewpoints. Sp represents the angle of linear polarization (AoLP) and degree of polarization (DoLP) scores to improve matching accuracy. Sdn represents the relationship between the depth gradient and the normal to constrain the candidate value.

[0061] In the fourth step, based on the polarization cost function, a target depth is selected from the preset depth group, and a target normal candidate value is selected from the preset normal candidate value group. In practice, first, the execution entity may use the polarization cost function to screen each preset depth in the preset depth group to generate a target depth. Second, the execution entity may use the polarization cost function to screen each preset normal candidate value in the preset normal candidate value group to generate a target normal candidate value.

[0062] Step 5: Generate enhanced polarization information based on the target depth and the candidate target normal value. In practice, the execution entity may generate enhanced polarization information based on the target depth and the candidate target normal value using a preset enhancement algorithm. For example, the preset enhancement algorithm may be a non-subsampled contourlet transform algorithm.

[0063] Step 106 : Perform precise depth processing on the enhanced polarization information set to generate a precise depth information set.

[0064] In some embodiments, the execution entity may perform precise depth processing on the enhanced polarization information set using a preset precise depth algorithm to generate a precise depth information set. For example, the preset precise depth algorithm may be a stereo matching algorithm.

[0065] Step 107 : reverse-project the above-mentioned precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data.

[0066] In some embodiments, the execution entity may inversely project the precise depth information set into a three-dimensional space to obtain enhanced Gaussian point cloud data. The three-dimensional space may be a space relative to a reference coordinate system. For example, the reference coordinate system may include a camera coordinate system, a world coordinate system, and a local coordinate system.

[0067] Step 108 : Based on the polarization information set, the image color value, and the target image, perform Gaussian point cloud de-reflection processing on the initialized target image information to generate de-reflected Gaussian point cloud data.

[0068] In some embodiments, the execution entity may perform Gaussian point cloud de-reflection processing on the initialized target image information based on the polarization information set, the image color value, and the target image to generate de-reflected Gaussian point cloud data.

[0069] In practice, the execution entity may perform Gaussian point cloud de-reflection processing on the initialized target image information based on the polarization information set, the image color value, and the target image through the following steps to generate de-reflection Gaussian point cloud data:

[0070] The first step is to locate the reflective area using the polarization information set and the image color values. In practice, the execution entity can use a preset positioning algorithm to locate the reflective area using the polarization information set and the image color values. The preset positioning algorithm can be a threshold segmentation algorithm.

[0071] The second step is to generate a chromaticity image based on the reflective area and the target image. In practice, first, the execution entity may convert the target image into a color space to obtain a color image. Second, the execution entity may extract characteristic colors from the color image. Then, the execution entity may perform brightness sorting on the characteristic colors to generate a characteristic image. Finally, the execution entity may perform reflective area recognition on the characteristic image to generate a chromaticity diagram. For example, the color space may be a red, green, and blue (RGB) color space.

[0072] The third step is to detect and process the reflective area based on the polarization information set and the chromaticity image to generate a reflection color loss term. In practice, the execution entity may use a preset image segmentation algorithm to detect and process the reflective area based on the polarization information set and the chromaticity image to generate a reflection color loss term. For example, the preset image segmentation algorithm may be an edge detection algorithm.

[0073] In the fourth step, based on the reflection color loss term, Gaussian point cloud de-reflection processing is performed on the initialized Gaussian point cloud data to generate de-reflected Gaussian point cloud data. In practice, the execution entity may use a preset de-reflection algorithm to perform Gaussian point cloud de-reflection processing on the initialized Gaussian point cloud data based on the reflection color loss term to generate de-reflected Gaussian point cloud data. For example, the preset de-reflection algorithm may be a Gaussian filtering algorithm.

[0074] Step 109 : Generate Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data.

[0075] In some embodiments, the execution entity may generate Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data.

[0076] In practice, the execution entity may generate Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data through the following steps:

[0077] The first step is to perform back-projection processing on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data based on the Gaussian point cloud depth group and the Gaussian point cloud normal candidate value group to generate back-projected Gaussian point cloud data. The Gaussian point cloud depth in the Gaussian point cloud depth group is the target depth corresponding to the enhanced polarization information in the enhanced polarization information set corresponding to the enhanced Gaussian point cloud data, and the Gaussian point cloud normal candidate value in the Gaussian point cloud normal candidate value group is the target normal candidate value corresponding to the enhanced polarization information in the enhanced polarization information set corresponding to the enhanced Gaussian point cloud data. In practice, the execution entity can perform back-projection processing on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data based on the Gaussian point cloud depth group and the Gaussian point cloud normal candidate value group through a preset back-projection algorithm to generate back-projected Gaussian point cloud data. The preset back-projection algorithm can be: a filtered back-projection algorithm.

[0078] In the second step, based on the above target normal candidate value, the above back-projected Gaussian point cloud data is geometrically adjusted to generate Gaussian point cloud data.

[0079] The back-projected Gaussian point cloud data is geometrically adjusted based on the target normal candidate value using the following formula to generate Gaussian point cloud data:

[0080]

[0081] Where d represents the distance from each point in the point cloud to its neighboring points. σ represents the standard deviation of the point cloud data. e represents the base of the natural logarithm.

[0082] Step 110 : performing data optimization and reconstruction processing on the Gaussian point cloud data to generate three-dimensional triangular facets.

[0083] In some embodiments, the execution entity may perform data optimization and reconstruction processing on the Gaussian point cloud data to generate three-dimensional triangular facets.

[0084] In practice, the execution entity may perform data optimization and reconstruction processing on the Gaussian point cloud data to generate three-dimensional triangular facets through the following steps:

[0085] In the first step, the Gaussian point cloud data and the reflective area are reconstructed using a preset reflection color loss function, a preset normal consistency loss function, and a preset depth normal consistency loss function to generate a depth image. For example, the preset reflection color loss function may be a mean square error loss function. The preset normal consistency loss function may be a cross entropy loss function. The preset depth normal consistency loss function may be a structural similarity loss function.

[0086] In the second step, the depth image is geometrically optimized using a preset normal consistency function to generate a three-dimensional triangular facet. The three-dimensional triangular facet can be a two-dimensional geometric model. For example, the preset normal consistency function can be a truncated signed distance function (TSDF).

[0087] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a high-precision three-dimensional surface reconstruction device based on polarization information enhancement. These embodiments of the high-precision three-dimensional surface reconstruction device based on polarization information enhancement are similar to Figure 1 Corresponding to the method embodiments shown, the high-precision three-dimensional surface reconstruction device based on polarization information enhancement can be specifically applied to various electronic devices.

[0088] like Figure 2As shown, some embodiments of the high-precision three-dimensional surface reconstruction device 200 based on polarization information enhancement include: an acquisition unit 201, a linear decomposition unit 202, a fusion unit 203, an initialization unit 204, a polarization enhancement unit 205, a precise depth unit 206, a back projection unit 207, a dereflection unit 208, a generation unit 209, and an optimization and reconstruction unit 210. Among them, the acquisition unit 201 is configured to use a polarization camera to acquire polarization images of multiple perspectives to obtain a polarization image set; the linear decomposition unit 202 is configured to perform linear decomposition processing on each polarization image in the above polarization image set to generate polarization information to obtain a polarization information set, wherein the polarization information in the polarization information set includes: linear polarization angle and polarization degree; the fusion unit 203 is configured to perform image fusion processing on each polarization image in the above polarization image set to generate a target image; the initialization unit 204 is configured to perform structured light initialization processing on the above target image to generate initialized target image information, wherein the above initialized target image information includes: camera posture parameters, initialized Gaussian point cloud data, and image color values; the polarization enhancement unit 205 is configured to perform structured light initialization processing on each polarization information in the above polarization information set The polarization enhancement processing is performed to generate enhanced polarization information and obtain an enhanced polarization information set; the precise depth unit 206 is configured to perform precise depth processing on the above-mentioned enhanced polarization information set to generate a precise depth information set; the back projection unit 207 is configured to back-project the above-mentioned precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data; the de-reflection unit 208 is configured to perform Gaussian point cloud de-reflection processing on the above-mentioned initialized target image information based on the above-mentioned polarization information set, the above-mentioned image color value and the above-mentioned target image to generate de-reflected Gaussian point cloud data; the generation unit 209 is configured to generate Gaussian point cloud data based on the above-mentioned enhanced Gaussian point cloud data and the above-mentioned de-reflected Gaussian point cloud data; the optimization and reconstruction unit 210 is configured to perform data optimization and reconstruction processing on the above-mentioned Gaussian point cloud data to generate three-dimensional triangular facets.

[0089] It is understandable that the units described in the high-precision three-dimensional surface reconstruction device 200 based on polarization information enhancement are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units included therein, and will not be repeated here.

[0090] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0092] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0093] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0094] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0095] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0096] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: uses a polarization camera to collect polarization images of multiple perspectives to obtain a polarization image set; performs linear decomposition processing on each polarization image in the polarization image set to generate polarization information to obtain a polarization information set, wherein the polarization information in the polarization information set includes: linear polarization angle and polarization degree; performs image fusion processing on each polarization image in the polarization image set to generate a target image; performs structured light initialization processing on the target image to generate initialized target image information, wherein the initialized target image information includes: camera pose parameters, initialized Gaussian point cloud data, etc. data, image color values; performing polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information and obtain an enhanced polarization information set; performing precise depth processing on the enhanced polarization information set to generate a precise depth information set; inversely projecting the precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data; based on the polarization information set, the image color values ​​and the target image, performing Gaussian point cloud de-reflection processing on the initialized target image information to generate de-reflected Gaussian point cloud data; generating Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data; performing data optimization and reconstruction processing on the Gaussian point cloud data to generate a three-dimensional triangular facet.

[0097] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, they may be described as: a processor comprising: an acquisition unit, a linear decomposition unit, a fusion unit, an initialization unit, a polarization enhancement unit, a precise depth unit, a back projection unit, a dereflection unit, a generation unit, an optimization and reconstruction unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the generation unit may also be described as "a unit for generating Gaussian point cloud data based on the above-mentioned enhanced Gaussian point cloud data and the above-mentioned dereflected Gaussian point cloud data".

[0100] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0101] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A high-precision three-dimensional surface reconstruction method based on polarization information enhancement, comprising: Use a polarization camera to collect polarization images from multiple perspectives to obtain a polarization image set; Performing linear decomposition processing on each polarization image in the polarization image set to generate polarization information, thereby obtaining a polarization information set, wherein the polarization information in the polarization information set includes: a linear polarization angle and a degree of polarization; performing image fusion processing on each polarization image in the polarization image set to generate a target image; Performing structured light initialization processing on the target image to generate initialized target image information, wherein the initialized target image information includes: camera pose parameters, initialized Gaussian point cloud data, and image color values; performing polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information, thereby obtaining an enhanced polarization information set; performing precise depth processing on the enhanced polarization information set to generate a precise depth information set; Inversely projecting the precise depth information set into three-dimensional space to obtain enhanced Gaussian point cloud data; performing Gaussian point cloud de-reflection processing on the initialized target image information based on the polarization information set, the image color value, and the target image to generate de-reflected Gaussian point cloud data; Generating Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data; The Gaussian point cloud data is optimized and reconstructed to generate three-dimensional triangular facets.

2. The method according to claim 1, wherein The performing structured light initialization processing on the target image to generate initialized target image information includes: Performing parameter matching processing on the target image to generate camera pose parameters; Performing three-dimensional reconstruction processing on the camera pose parameters to generate an initial point cloud; Performing Gaussian noise addition processing on the initial point cloud to generate initialized Gaussian point cloud data; Performing color mapping processing on the target image to generate image color values; The camera pose parameters, the initialized Gaussian point cloud data, and the image color value are determined as initialized target image information.

3. The method according to claim 1, wherein The performing polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information includes: generating a preset depth group and a preset normal candidate value group based on the polarization information; generating a polarization consistency score based on the preset depth group and the preset normal candidate value group; Combining the polarization consistency score with a preset photometric matching score to obtain a polarization cost function; Based on the polarization cost function, selecting a target depth from the preset depth group and selecting a target normal candidate value from the preset normal candidate value group; Enhanced polarization information is generated based on the target depth and the target normal candidate value.

4. The method according to claim 3, wherein: The performing Gaussian point cloud de-reflection processing on the initialized target image information based on the polarization information set, the image color value, and the target image to generate de-reflected Gaussian point cloud data includes: Locating a reflective area using the polarization information set and the image color value; generating a colorimetric image based on the reflective area and the target image; performing detection processing on the reflective area based on the polarization information set and the chromaticity image to generate a reflection color loss term; Based on the reflection color loss term, Gaussian point cloud de-reflection processing is performed on the initialized Gaussian point cloud data to generate de-reflected Gaussian point cloud data.

5. The method according to claim 4, wherein The generating of Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data includes: Based on the Gaussian point cloud depth group and the Gaussian point cloud normal candidate value group, back-projection processing is performed on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data to generate back-projected Gaussian point cloud data, wherein the Gaussian point cloud depth in the Gaussian point cloud depth group is a target depth corresponding to the enhanced polarization information in the enhanced polarization information set corresponding to the enhanced Gaussian point cloud data, and the Gaussian point cloud normal candidate value in the Gaussian point cloud normal candidate value group is a target normal candidate value corresponding to the enhanced polarization information in the enhanced polarization information set corresponding to the enhanced Gaussian point cloud data; Based on the target normal candidate value, geometric adjustment processing is performed on the back-projected Gaussian point cloud data to generate Gaussian point cloud data.

6. The method according to claim 5, wherein: The performing data optimization and reconstruction processing on the Gaussian point cloud data to generate a three-dimensional triangular facet includes: Performing image reconstruction processing on the Gaussian point cloud data and the reflective area using a preset reflection color loss function, a preset normal consistency loss function, and a preset depth normal consistency loss function to generate a depth image; The depth image is geometrically optimized using a preset normal consistency function to generate a three-dimensional triangular facet.

7. A high-precision three-dimensional surface reconstruction device based on polarization information enhancement, comprising: an acquisition unit configured to acquire polarization images of multiple perspectives using a polarization camera to obtain a polarization image set; a linear decomposition unit configured to perform linear decomposition processing on each polarization image in the polarization image set to generate polarization information, thereby obtaining a polarization information set, wherein the polarization information in the polarization information set includes: a linear polarization angle and a degree of polarization; a fusion unit configured to perform image fusion processing on each polarization image in the polarization image set to generate a target image; an initialization unit configured to perform structured light initialization processing on the target image to generate initialized target image information, wherein the initialized target image information includes: camera pose parameters, initialized Gaussian point cloud data, and image color values; a polarization enhancement unit configured to perform polarization enhancement processing on each polarization information in the polarization information set to generate enhanced polarization information, thereby obtaining an enhanced polarization information set; a precise depth unit configured to perform precise depth processing on the enhanced polarization information set to generate a precise depth information set; a back-projection unit configured to back-project the precise depth information set into a three-dimensional space to obtain enhanced Gaussian point cloud data; a dereflection unit configured to perform Gaussian point cloud dereflection processing on the initialized target image information based on the polarization information set, the image color value, and the target image to generate dereflected Gaussian point cloud data; a generating unit configured to generate Gaussian point cloud data based on the enhanced Gaussian point cloud data and the de-reflected Gaussian point cloud data; The optimization and reconstruction unit is configured to perform data optimization and reconstruction processing on the Gaussian point cloud data to generate three-dimensional triangular facets.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images