Three-dimensional reconstruction method, reconstruction device and reconstruction system based on non-Lambert model

Abstract

The invention discloses a three-dimensional reconstruction method, device and system based on a non-Lambert model, and belongs to the field of three-dimensional reconstruction. The non-Lambert model-based three-dimensional reconstruction method provided by the embodiment of the invention comprises the steps of performing image acquisition on a to-be-reconstructed object to obtain multiple groups of reconstructed images with different illumination angles; calculating a corresponding incident light vector according to the illumination angle of the reconstructed image; calculating a surface normal vector of the to-be-reconstructed object according to a reflection formula of a non-Lambert model based on the incident light vector and the reconstructed image; and performing height reconstruction on the to-be-reconstructed object based on the surface normal vector to obtain a reconstruction result. Therefore, three-dimensional reconstruction with relatively high precision can be realized, a relatively good reconstruction effect on an object with a large amount of highlight information is also achieved, and the adaptability is relatively high.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of photometric stereo vision, in particular to a three-dimensional reconstruction method based on a non-Lambert model, a three-dimensional reconstruction device, a three-dimensional reconstruction system and a computer readable storage medium. BACKGROUND

[0002] As an important branch of computer vision, the three-dimensional reconstruction field aims to use a computer to establish a three-dimensional model expressing an objective object in the real world, and to meet the needs of production and life. In recent years, researchers have found that two-dimensional images contain rich information about objects, and have proposed three-dimensional reconstruction based on images. Among them, photometric stereo vision has attracted much attention because it can restore high-precision surface details.

[0003] Currently, photometric stereo vision is based on the Lambert model, which assumes that the surface of the scene obeys the law of diffuse reflection, and restores surface normal information from photometric images to perform three-dimensional reconstruction. However, the material in the real environment varies greatly, and the diffuse reflection law cannot well describe the geometric and physical relationship of the material surface. Therefore, the three-dimensional reconstruction method based on the Lambert model cannot handle objects with a large amount of highlight information. SUMMARY

[0004] The embodiments of the present application provide a three-dimensional reconstruction method based on a non-Lambert model, a three-dimensional reconstruction device, a three-dimensional reconstruction system and a computer readable storage medium to solve at least one of the above technical problems.

[0005] The three-dimensional reconstruction method of the embodiments of the present application comprises: image acquisition of the object to be reconstructed to obtain a plurality of sets of reconstruction images with different illumination angles; calculating the corresponding incident light vector according to the illumination angle of the reconstruction image; calculating the surface normal vector of the object to be reconstructed according to the reflection formula of the non-Lambert model based on the incident light vector and the reconstruction image; high reconstruction of the object to be reconstructed based on the surface normal vector to obtain a reconstruction result.

[0006] In some embodiments, the image acquisition of the object to be reconstructed to obtain a plurality of sets of reconstruction images with different illumination angles comprises: controlling a plurality of light sources located at different illumination angles to emit light in turn; controlling the image acquisition device to perform image acquisition of the object to be reconstructed under the condition that each of the light sources emits light, to obtain a plurality of sets of the reconstruction images; The calculation of the corresponding incident light vector according to the illumination angle of the reconstruction image comprises: According to the device parameters of the image acquisition device and the illumination angle of the reconstructed image, a corresponding incident light vector is calculated.

[0007] In some embodiments, the device parameters include focal length and pixel size, the illumination angle includes rotation angle and tilt angle, and the calculation of the corresponding incident light vector according to the device parameters of the image acquisition device and the illumination angle of the reconstructed image includes: According to the focal length, the pixel size, the rotation angle and the tilt angle of the reconstructed image, the incident light vector of each pixel point in each reconstructed image is calculated.

[0008] In some embodiments, the three-dimensional reconstruction method further includes: Setting the material parameters of the non-Lambertian model; The calculation of the surface normal vector of the object to be reconstructed according to the reflection formula of the non-Lambertian model based on the incident light vector and the reconstructed image includes: According to the reflection formula of the non-Lambertian model, the surface normal vector of each corresponding position point in the object to be reconstructed is calculated based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters.

[0009] In some embodiments, the non-Lambertian model is a Cook-Torrance model, and the material parameters include any one or more of base reflectance, roughness, diffuse reflection color, diffuse reflection coefficient, and specular reflection coefficient.

[0010] In some embodiments, the calculation of the surface normal vector of each corresponding position point in the object to be reconstructed according to the reflection formula of the non-Lambertian model based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters includes: Setting the initial value of the to-be-solved normal vector in the reflection formula; According to the reflection formula, a first brightness of each corresponding position point in the object to be reconstructed is calculated based on the incident light vector of each pixel point in each reconstructed image and the material parameters. According to the first brightness and the reconstructed image, the to-be-solved normal vector is iteratively optimized according to a nonlinear least mean square algorithm to obtain the corresponding surface normal vector of each position point.

[0011] In some embodiments, the iterative optimization of the to-be-solved normal vector according to the nonlinear least mean square algorithm based on the first brightness and the reconstructed image to obtain the corresponding surface normal vector of each position point includes: acquiring a second brightness of each pixel point in each of the reconstructed images; calculating a brightness difference between the first brightness of each of the position points and the second brightness of the corresponding pixel point in each of the reconstructed images, to obtain a plurality of brightness differences corresponding to the reconstructed images; performing optimization iteration on the to-be-solved normal vector according to a sum of squares of the plurality of brightness differences, to obtain the surface normal vector corresponding to each of the position points.

[0012] In some embodiments, the height reconstruction of the to-be-reconstructed object based on the surface normal vector to obtain the reconstruction result comprises: height reconstructing the to-be-reconstructed object based on the surface normal vector according to a height reconstruction algorithm to obtain the reconstruction result; The height reconstruction algorithm comprises any one of a global iteration algorithm, a frequency domain discrete cosine transform algorithm, and a pyramid reconstruction algorithm.

[0013] The three-dimensional reconstruction device based on the non-Lambert model in the embodiments of the present application comprises: an image acquisition module configured to acquire images of a to-be-reconstructed object to obtain a plurality of reconstructed images with different illumination angles; a first calculation module configured to calculate corresponding incident light vectors according to the illumination angles of the reconstructed images; a second calculation module configured to calculate a surface normal vector of the to-be-reconstructed object according to a reflection formula of the non-Lambert model based on the incident light vectors; a height reconstruction module configured to height reconstruct the to-be-reconstructed object based on the surface normal vector to obtain a reconstruction result.

[0014] The three-dimensional reconstruction system based on the non-Lambert model in the embodiments of the present application comprises one or more processors and a memory, the memory stores a computer program, and the computer program is executed by the processor to implement the three-dimensional reconstruction method of any one of the above embodiments.

[0015] The computer readable storage medium in the embodiments of the present application stores a computer program, and the program is executed by the processor to implement the three-dimensional reconstruction method of any one of the above embodiments.

[0016] In the three-dimensional reconstruction method, the three-dimensional reconstruction device, the three-dimensional reconstruction system and the computer readable storage medium based on the non-Lambert model according to the embodiments of the application, the three-dimensional reconstruction is performed based on the non-Lambert model, a plurality of groups of reconstruction images with different light angles are collected, the corresponding incident light vectors are calculated according to the light angles of the reconstruction images, the surface normal vectors of the object to be reconstructed are calculated according to the reflection formula of the non-Lambert model, the height reconstruction of the object to be reconstructed is performed based on the surface normal vectors, and the reconstruction result is obtained. In this way, the three-dimensional reconstruction with high precision can be realized, the object with a large amount of highlight information also has good reconstruction effect, and the adaptability is strong.

[0017] Additional aspects and advantages of the embodiments of the application will be in part apparent and in part pointed out hereinafter in the description. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, including the drawings, in which: Figure 1 is one of the flow diagrams of the three-dimensional reconstruction method according to some embodiments of the application; Figure 2 is a schematic diagram of setting a plurality of light sources with different light angles according to some embodiments of the application; Figure 3 is a schematic diagram of a reconstruction image according to some embodiments of the application; Figure 4 is a schematic diagram of imaging based on the Lambert model according to some embodiments of the application; Figure 5 is a schematic diagram of imaging based on the non-Lambert model according to some embodiments of the application; Figure 6 is a schematic diagram of the reflection effect of an object with different roughness according to some embodiments of the application; Figure 7 is a schematic diagram of a half-angle vector according to some embodiments of the application; Figure 8 is a schematic diagram of light ray occlusion between micro-surfaces according to some embodiments of the application; Figure 9 is another flow diagram of the three-dimensional reconstruction method according to some embodiments of the application; Figure 10 is a schematic diagram of a scene for collecting a reconstruction image according to some embodiments of the application; Figure 11 is a schematic diagram of the rotation angle and the inclination angle of a light source according to some embodiments of the application; Figure 12 is a third flow diagram of the three-dimensional reconstruction method according to some embodiments of the application; Figure 13 Fig. 4 is a flowchart of a method for three-dimensional reconstruction according to some embodiments of the present application; Figure 14 Fig. 5 is a flowchart of a method for three-dimensional reconstruction according to some embodiments of the present application; Figure 15 Fig. 6 is a flowchart of a method for height reconstruction based on a frequency domain discrete cosine transform algorithm according to some embodiments of the present application; Figure 16 Fig. 7 is a block diagram of a three-dimensional reconstruction device according to some embodiments of the present application; Figure 17 Fig. 8 is a block diagram of a three-dimensional reconstruction system according to some embodiments of the present application; Figure 18 Fig. 9 is a diagram of a connection state of a computer readable storage medium and a processor according to some embodiments of the present application.

[0019] Reference Signs List: Object to be reconstructed 10, light source 20, image acquisition device 30, three-dimensional reconstruction device 200, image acquisition module 210, first calculation module 220, second calculation module 230, height reconstruction module 240, three-dimensional reconstruction system 300, processor 310, memory 320, computer readable storage medium 400, computer program 410, processor 420. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be further described below with reference to the drawings. The same or similar reference signs are used throughout the drawings to refer to the same or like parts or elements with the same or similar function. In addition, the embodiments of the present application described below with reference to the drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as a limitation of the present application.

[0021] An important goal of photometric stereo vision is to solve the illumination reflection model through the image difference caused by the light source, to obtain the normal information of the scene surface, and to restore the three-dimensional geometric model of the object. In the related art, three-dimensional reconstruction is performed based on the Lambert model. It is assumed that the incident light is dispersed at the same intensity in each direction, regardless of the position of the observer. Such a surface is an ideal diffuse reflector. The reflected light intensity depends only on the incident light intensity, the cosine value of the included angle between the surface normal and the incident light direction, and the diffuse reflectivity of the surface. The diffuse reflection formula of a point light source illuminating the surface is as follows:

[0022] where I is the light intensity value of each pixel in the image, is the diffuse reflection coefficient, n is the normal vector of the object surface, and s is the normal light source direction of the point light source. The three-dimensional reconstruction based on the Lambert model is suitable for objects with rough surfaces and no obvious gloss, and cannot handle objects with a large amount of highlight information, such as metals, porcelain, and jade.

[0023] Therefore, referring to Figure 1 and Figure 2 , the embodiment of the present application provides a three-dimensional reconstruction method based on a non-Lambert model, which comprises the following steps: 010: image acquisition is performed on the object to be reconstructed 10 to obtain a plurality of groups of reconstruction images with different illumination angles; 020: the corresponding incident light vectors are calculated according to the illumination angles of the reconstruction images; 030: the surface normal vectors of the object to be reconstructed 10 are calculated according to the reflection formula of the non-Lambert model based on the incident light vectors and the reconstruction images; 040: high reconstruction is performed on the object to be reconstructed 10 based on the surface normal vectors to obtain a reconstruction result.

[0024] In the three-dimensional reconstruction method of the embodiment of the present application, the three-dimensional reconstruction is performed based on the non-Lambert model, a plurality of groups of reconstruction images with different illumination angles are collected, the corresponding incident light vectors are calculated according to the illumination angles of the reconstruction images, the surface normal vectors of the object to be reconstructed 10 are calculated according to the reflection formula of the non-Lambert model, and high reconstruction is performed on the object to be reconstructed 10 based on the surface normal vectors to obtain a reconstruction result. In this way, the three-dimensional reconstruction with high precision can be realized, and the object with a large amount of highlight information also has good reconstruction effect, and the adaptability is strong.

[0025] Specifically, image acquisition is performed on the object to be reconstructed 10 under different illumination angles to obtain a plurality of groups of reconstruction images with different illumination angles, each group including at least one reconstruction image. It can be understood that the incident light received by the object to be reconstructed 10 at the same position is different under different illumination angles; according to the illumination angles of the reconstruction images, the incident light vectors corresponding to the reconstruction images under different illumination angles can be calculated to represent the incident light conditions of the corresponding positions in the reconstruction images.

[0026] A suitable non-Lambert model is selected, and the non-Lambert model can adopt any one of the Blinn-Phong model, the Kajiya-Kay model, the Torrance-Sparrow model, and the Cook-Torrance model, and of course, any other non-Lambert model can also be adopted.

[0027] The reflection formula of the non-Lambert model is a formula for deriving the reflected light intensity based on the incident light intensity and the normal vector, and the parameters in the reconstructed image can represent the reflected light intensity. Therefore, based on the incident light vector and the reconstructed image, the surface normal vector of the object to be reconstructed 10 can be solved according to the reflection formula of the non-Lambert model.

[0028] After obtaining the surface normal vector of the object to be reconstructed 10, the height reconstruction can be performed based on the surface normal vector, the three-dimensional geometric model of the object to be reconstructed 10 is reconstructed, and the reconstruction result corresponding to the object to be reconstructed 10 is obtained.

[0029] It is found through research that, for a copper concave pit with a height of 200 um, the three-dimensional reconstruction is performed based on the Lambert model, the height of the reconstructed pit is 750 um, and the error is 500 um. The reconstruction is performed based on the non-Lambert model, and four reconstructed images with different illumination angles are obtained, as shown in Figure 3 a, b, c, and d in FIG. 4, the height of the reconstructed pit is 250 um, and the error is 50 um.

[0030] In the embodiments of the present application, the photometric stereo vision method is used for three-dimensional reconstruction based on the non-Lambert model. The Lambert model only considers ideal diffuse reflection, while the non-Lambert model is used to describe the reflection characteristics of complex materials in the real world. The reflection formula of the non-Lambert model considers not only diffuse reflection but also specular reflection, and can more realistically simulate the lighting effect of the object surface, especially for objects with a large amount of highlight information, so as to perform more accurate three-dimensional reconstruction, and the reconstruction accuracy is better than that of the Lambert model.

[0031] As shown in Figure 4 and Figure 5 For the same size convex, the Lambert model and the non-Lambert model are used for imaging, Figure 4 corresponding to the Lambert model, Figure 5 corresponding to the non-Lambert model, it can be easily seen that there is a significant difference between the imaging of the Lambert model and the non-Lambert model, and the imaging effect of the non-Lambert model is closer to the real object surface.

[0032] In the related art, in order to solve the problem that the three-dimensional reconstruction method based on the Lambert model cannot process objects with a large amount of highlight information, the normal solving of photometric stereo is performed by removing the highlights of the image and separating the diffuse reflection component. However, the method of removing the highlights usually needs to perform image segmentation as a preprocessing operation, which requires that the reflection highlight area of the object be as concentrated as possible to ensure the integrity of the highlight area during segmentation, and the adaptability is poor. In the embodiments of the present application, the normal solving is performed by comprehensively considering the diffuse reflection and specular reflection, and the highlight removal is not needed, and the adaptability is strong.

[0033] In some embodiments, the non-Lambert model is a Cook-Torrance model.

[0034] Specifically, the Cook-Torrance model is a kind of physical-based micro-surface lighting model, which is improved on the basis of the Torrance-Sparrow model, and combines micro-surface theory and Fresnel reflection. Compared with the Phong model, the Cook-Torrance model can more realistically simulate the lighting effect of the object surface, and has better applicability.

[0035] The Cook-Torrance lighting model regards the rough surface of an object as being composed of a large number of micro-planes, each of which can be regarded as an ideal mirror reflector. The roughness of the object surface is represented by the change of the slope of the micro-planes. The rougher the surface is, the greater the change of the slope is, and vice versa.

[0036] The reflection formula of the Cook-Torrance model is as follows:

[0037] wherein, represents the outgoing light intensity along the observation direction at the point , that is, the line-of-sight direction; represents the incident light intensity, represents the incident light vector; n represents the surface normal (that is, the surface normal vector); represents the hemispherical integration region; represents the diffuse reflection coefficient; represents the specular reflection coefficient; represents the diffuse reflection color; D represents the micro-plane distribution function; F represents the Fresnel reflection function; and G represents the geometric attenuation coefficient.

[0038] The diffuse reflection coefficient is used to control the proportion of diffuse reflection in the total reflection, and the value range is between 0 and 1. The greater the diffuse reflection coefficient is, the more obvious the diffuse reflection effect is, and the smaller the diffuse reflection is, and the relatively stronger the specular reflection is.

[0039] The diffuse reflection color c represents the diffuse reflection color presented by the surface of the object to be reconstructed 10 under indirect lighting or ambient lighting, which is determined by the material properties of the object to be reconstructed 10, such as a red apple, a blue sky, etc. The diffuse reflection color c determines the basic color tone of the object to be reconstructed 10 without direct lighting, and the color of the reflected light in the diffuse reflection process.

[0040] The specular reflection coefficient is used to control the proportion of specular reflection in the total reflection, which is similar to the diffuse reflection coefficient , and the value range is between 0 and 1, and , The greater the value, the more obvious the mirror reflection effect, and the relatively weaker the diffuse reflection. The smaller the value, the weaker the mirror reflection, and the relatively stronger the diffuse reflection.

[0041] The microfacet distribution function D is used to describe the probability of the surface normal n aligning with the half-angle vector, and reflects the roughness of the micro surface. The higher the roughness, the smaller the D value, the more dispersed the reflected light, and the larger the highlight range. The lower the roughness, the greater the D value, the more concentrated the reflected light, and the sharper the highlight. For example, for a smooth metal surface, the D value is large, and the highlight is concentrated and bright. For a rough wall surface, the D value is small, and the highlight range is large and dim.

[0042] As shown in Figure 6 , the roughness of objects (1) to (5) is 0.1, 0.3, 0.5, 0.8, and 1.0, respectively. It can be easily seen that the lower the roughness, the more the microfacets aligned with the intermediate vector are highly concentrated in a small radius range, and finally a very bright spot is generated.

[0043] The calculation formula of the microfacet distribution function D is as follows:

[0044] Wherein, a represents the roughness parameter, n represents the macroscopic normal (i.e. the surface normal), and m represents the micro surface normal.

[0045] The Fresnel reflection function F is used to describe the distribution ratio of the reflected light between mirror reflection and diffuse reflection when the angle between the line of sight direction and the normal changes. When the angle between the line of sight direction and the normal is small, the F value is large, and the reflected light tends to be mirror reflection, and the object surface looks more shiny. When the angle between the line of sight direction and the normal increases, the F value decreases, and the reflected light tends to be diffuse reflection. For example, when observing the water surface from the side, the reflected light is weak, and more is diffuse reflection. When observed vertically, the reflected light is very strong, and mainly is mirror reflection.

[0046] The calculation formula of the Fresnel reflection function F is as follows:

[0047] Wherein, represents the base reflectivity; represents the line of sight direction; h represents the half-angle vector, and the calculation formula of the half-angle vector h is as follows:

[0048] Wherein, represents the incident light vector (i.e. the aforementioned ). As shown in Figure 7 , the half-angle vector h refers to the incident light vector and the line of sight direction intermediate vectors between them.

[0049] The geometric attenuation coefficient G is used to describe the self-shielding and shadow effect between micro-surfaces, and the value range is between 0 and 1. The smaller the G value, the more obvious the shielding and shadow effect. When G = 1, there is no shielding and shadow, and the light can be fully irradiated and reflected; when G approaches 0, the unevenness of the micro-surface causes a large amount of light to be blocked, as shown in Figure 8 , so that the reflected light intensity is weakened.

[0050] The calculation formula of the geometric attenuation coefficient G is as shown in the following formula:

[0051]

[0052]

[0053]

[0054] Please refer to Figure 2 and Figure 9 , in some embodiments, image acquisition is performed on the object to be reconstructed 10 to obtain a plurality of groups of reconstruction images (i.e. 010) with different illumination angles, including: 011: control a plurality of light sources 20 located at different illumination angles to emit light in turn; 012: control the image acquisition device 30 to perform image acquisition on the object to be reconstructed 10 under the condition that each light source 20 emits light, to obtain a plurality of groups of reconstruction images; At this time, according to the illumination angle of the reconstruction image, the corresponding incident light vector (i.e. 020) is calculated, including: 021: according to the device parameters of the image acquisition device 30 and the illumination angle of the reconstruction image, the corresponding incident light vector is calculated.

[0055] Specifically, a plurality of light sources 20 are provided, and the plurality of light sources 20 are located at different illumination angles, as shown in Figure 2 , s1, s2, s3 respectively represent the normal light source 20 direction of the three light sources 20, n represents the normal vector, and v represents the line of sight direction. Control the plurality of light sources 20 to emit light in turn, and perform image acquisition on the object to be reconstructed 10 through the image acquisition device 30. Control the image acquisition device 30 to perform image acquisition on the object to be reconstructed 10 when each light source 20 emits light.

[0056] The number of image acquisition devices 30 is at least one. As shown in Figure 10 , in the case that the number of image acquisition devices 30 is 1, the image acquisition device 30 can be controlled to perform image acquisition on the object to be reconstructed 10 once when each light source 20 emits light, to obtain one reconstruction image, which constitutes a group of reconstruction images.

[0057] In the case that the number of image acquisition devices 30 is multiple, the multiple image acquisition devices 30 can be respectively arranged at different orientations to have different shooting angles of view, and the multiple image acquisition devices 30 can be controlled to respectively perform image acquisition on the object 10 to be reconstructed when each light source 20 emits light, so as to obtain multiple reconstruction images to form a group of reconstruction images.

[0058] According to the device parameters of the image acquisition device 30 and the illumination angle of the reconstruction image, the incident light vector corresponding to each reconstruction image can be calculated.

[0059] In the related art, three-dimensional reconstruction is performed based on a bidirectional reflectance distribution function (BRDF), and the BRDF is suitable for more materials. However, this method has high requirements for the structure of the light source and the image acquisition device, for example, under ideal conditions, an isotropic material requires that the light sources are uniformly distributed on a ring to perform related calculations.

[0060] In the embodiments of the present application, the multiple light sources 20 only need to be arranged at different illumination angles, and whether the light sources 20 are uniformly distributed is not limited, and the light sources 20 and the image acquisition device 30 do not need to be specially arranged. In this way, the operation is simple and has strong adaptability.

[0061] Please refer to Figure 10 In some embodiments, the device parameters include a focal length and a pixel size, and the illumination angle includes a rotation angle and an inclination angle. According to the device parameters of the image acquisition device 30 and the illumination angle of the reconstruction image, the corresponding incident light vector (i.e. 021) is calculated, including: According to the focal length, the pixel size, the rotation angle and the inclination angle of the reconstruction image, the incident light vector of each pixel point in each reconstruction image is calculated.

[0062] Specifically, the device parameters of the image acquisition device 30 include a focal length and a pixel size of a chip. In one example, the focal length of the image acquisition device 30 is 35 mm, and the pixel size of the chip is 1.85 μm.

[0063] The illumination angle includes a rotation angle and an inclination angle, as shown in Figure 11 The rotation angle is represented by Tilt, and the rotation angle represents the azimuth angle parameter of the light source 20, representing the rotation angle of the light source 20 in the horizontal direction, that is, the included angle between the light source 20 and the positive direction of the x axis counterclockwise, which can also be called the azimuth angle, and the value range is [0, 360°]. The inclination angle is represented by Slant, and the inclination angle represents the inclination angle parameter of the light source 20, representing the included angle between the light source 20 and the positive direction of the z axis, which can also be called the polar angle, and the value range is [0, 90°].

[0064] In one example, four light sources 20 are set, the tilt angles of the four light sources 20 are all set to 30°, and the rotation angles are set to 0°, 90°, 180°, and 270° respectively. The four light sources 20 emit light in sequence, and the four reconstructed images collected in sequence are as follows: Figure 3 shown.

[0065] According to the focal length, pixel size, rotation angle and tilt angle of the reconstructed image, combined with the imaging geometry and illumination model of the image acquisition device 30, the incident light vector of each pixel point in each reconstructed image can be calculated using trigonometric functions.

[0066] See also Figure 10 and Figure 12 In some embodiments, the three-dimensional reconstruction method further comprises: 050: setting material parameters of the non-Lambertian model according to the material of the object to be reconstructed 10; At this time, based on the incident light vector and the reconstructed image, the surface normal vector (ie, 030) of the object to be reconstructed 10 is calculated according to the reflection formula of the non-Lambertian model, including: 031: Based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters, the surface normal vector of each corresponding position point in the object to be reconstructed 10 is calculated according to the reflection formula of the non-Lambertian model.

[0067] Specifically, the material parameters of the non-Lambertian model may be set according to the material of the object to be reconstructed 10. In some embodiments, the material parameters include any one or more of basic reflectivity, roughness, diffuse reflection color, diffuse reflection coefficient, and specular reflection coefficient.

[0068] The specific types of material parameters can be determined according to the selected non-Lambertian model. When the non-Lambertian model is the Cook-Torrance model, the material parameters include base reflectivity, roughness, diffuse color, diffuse reflection coefficient, and specular reflection coefficient. In an example, base reflectivity = (0.04, 0.04, 0.04); roughness parameter a = 0.7; diffuse color c = (0.95, 0.63, 0.54); =0.2; =1- =0.8.

[0069] After setting the material parameters, based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters, the surface normal vector of each position point in the object to be reconstructed 10 can be calculated according to the reflection formula of the non-Lambertian model. In this way, the non-Lambertian model can more realistically simulate the lighting effect of the surface of the object to be reconstructed 10, so as to calculate more accurate surface normal vectors, which is beneficial to improve the three-dimensional reconstruction accuracy. The specific process of calculating the surface normal vector is described in detail below.

[0070] Please refer to Figure 10 and Figure 13 In some embodiments, based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters, the surface normal vector of each position point in the object to be reconstructed 10 is calculated according to the reflection formula of the non-Lambertian model (i.e. 031), including: 0311: Set the initial value of the to-be-solved normal vector in the reflection formula; 0312: Based on the incident light vector of each pixel point in each reconstructed image and the material parameters, the first brightness of each position point in the object to be reconstructed 10 is calculated according to the reflection formula; 0313: Based on the first brightness and the reconstructed image, the to-be-solved normal vector is iteratively optimized according to the non-linear least squares algorithm to obtain the corresponding surface normal vector of each position point.

[0071] Specifically, in the reflection formula of the non-Lambertian model, n is the to-be-solved normal vector, and the initial value of the to-be-solved normal vector is set to (0, 0, 1). At the same time, the value of the observation direction can be set, and in one example, the observation direction is set to (0, 0, 1).

[0072] According to the reflection formula of the non-Lambertian model, the calculation formula of the first brightness of each position point in the object to be reconstructed 10 can be determined, and the calculation formula of the first brightness is as follows:

[0073] wherein, represents the first brightness. The incident light vector and the material parameters of each pixel point in each reconstructed image are substituted into the calculation formula of the first brightness, so that the first brightness of each position point in the object to be reconstructed 10 can be calculated.

[0074] Based on the first brightness and the reconstructed image, the to-be-solved normal vector can be iteratively optimized according to the non-linear least squares algorithm, so as to obtain the corresponding surface normal vector of each position point. The specific process of iterative optimization is described in detail below.

[0075] Please refer to Figure 14In some embodiments, based on the first brightness and the reconstructed image, the to-be-solved normal vector is iteratively optimized according to a non-linear least square algorithm to obtain a corresponding surface normal vector of each position point (i.e., 0313), including: 03131: obtaining a second brightness of each pixel point in each reconstructed image; 03132: calculating a brightness difference between the first brightness of each position point and the second brightness of the corresponding pixel point in each reconstructed image to obtain a plurality of brightness differences corresponding to a plurality of groups of reconstructed images; 03133: iteratively optimizing the to-be-solved normal vector according to the sum of squares of the plurality of brightness differences to obtain a corresponding surface normal vector of each position point.

[0076] Specifically, the brightness difference between the first brightness calculated based on the reflection formula and the brightness of the reconstructed image can be taken as an objective function of the non-linear least square algorithm optimization iteration. It can be understood that each position point in the to-be-reconstructed image has a corresponding pixel point in the reconstructed image, and the reconstructed image has a plurality of groups. Therefore, after the first brightness is calculated, the second brightness of each pixel point in each reconstructed image is obtained. The brightness difference between the first brightness of each position point and the second brightness of the corresponding pixel point in each reconstructed image is calculated to obtain a plurality of brightness differences corresponding to a plurality of groups of reconstructed images.

[0077] In one example, the reconstructed image has 4 groups, and the brightness difference between a position point (x, y) of the to-be-reconstructed object 10 and the corresponding pixel points in the 4 groups of reconstructed images is Rn1, Rn2, Rn3, and Rn4. The objective function can be expressed as:

[0078] wherein, represents the objective function, that is, the objective function is the sum of squares of a plurality of brightness differences. By minimizing the objective function, the to-be-solved normal vector is iteratively optimized, and the corresponding surface normal vector of each position point is obtained.

[0079] Please refer to Figure 10 In some embodiments, the to-be-reconstructed object 10 is height-reconstructed based on the surface normal vector to obtain a reconstruction result (i.e., 040), including: The to-be-reconstructed object 10 is height-reconstructed based on the surface normal vector according to a height-reconstruction algorithm to obtain a reconstruction result; The height-reconstruction algorithm includes any one of a global iteration algorithm, a frequency domain discrete cosine transform algorithm, and a pyramid reconstruction algorithm.

[0080] Specifically, the height of the object 10 to be reconstructed is reconstructed according to the surface normal vector based on a height reconstruction algorithm to obtain a reconstruction result. The height reconstruction algorithm can adopt any height reconstruction algorithm, which is not limited herein. For example, the height reconstruction algorithm can adopt any one of a global iterative algorithm, a Discrete Cosine Transform (DTC) algorithm, and a pyramid reconstruction algorithm.

[0081] For example, taking the DTC algorithm as the height reconstruction algorithm, as shown in FIG. 2, the gradient data can be derived according to the surface normal vector, and the surface height data of the object 10 to be reconstructed can be obtained by solving the Poisson equation according to the gradient data, so as to realize the height reconstruction and obtain the reconstruction result. The process of solving the Poisson equation can include performing the DTC, high-pass filtering, and inverse DTC (IDTC). Figure 15

[0082] For example, the gradient divergence field can be calculated according to the gradient data, the DTC is performed on the divergence field, the Poisson equation is converted from the spatial domain to the frequency domain, the solution of the Poisson equation is obtained in the frequency domain, the high-pass filter can be used for filtering to suppress the low-frequency noise or the baseline drift, and finally, the frequency domain solution of the Poisson equation is converted back to the spatial domain by the IDTC to obtain the surface height data.

[0083] When the global iterative algorithm, the pyramid reconstruction algorithm, or other algorithms are used as the height reconstruction algorithm, the height reconstruction can be performed in a manner and logic known to those skilled in the art, which is not described herein.

[0084] Please refer to Figure 2 and Figure 16 The embodiment of the present application further provides a three-dimensional reconstruction device 200 based on a non-Lambert model. The three-dimensional reconstruction device 200 includes an image acquisition module 210, a first calculation module 220, a second calculation module 230, and a height reconstruction module 240. The image acquisition module 210 is configured to acquire images of the object 10 to be reconstructed to obtain a plurality of groups of reconstruction images with different illumination angles. The first calculation module 220 is configured to calculate corresponding incident light vectors according to the illumination angles of the reconstruction images. The second calculation module 230 is configured to calculate the surface normal vector of the object 10 to be reconstructed according to the reflection formula of the non-Lambert model based on the incident light vectors and the reconstruction images. The height reconstruction module 240 is configured to reconstruct the height of the object 10 to be reconstructed based on the surface normal vector to obtain a reconstruction result.

[0085] ​In some embodiments, the image acquisition module 210 is specifically configured to control the multiple light sources 20 at different illumination angles to emit light in sequence; and in the case that each light source 20 emits light, the image acquisition device 30 is controlled to acquire images of the object 10 to be reconstructed, thereby obtaining multiple sets of reconstructed images. At this time, the first calculation module 220 is specifically configured to calculate the corresponding incident light vector according to the device parameters of the image acquisition device 30 and the illumination angle of the reconstructed image.

[0086] In some embodiments, the device parameters include focal length and pixel size, and the illumination angle includes rotation angle and tilt angle. The first calculation module 220 is specifically configured to calculate the incident light vector of each pixel point in each reconstructed image according to the focal length, the pixel size, the rotation angle and the tilt angle of the reconstructed image.

[0087] In some embodiments, the three-dimensional reconstruction device 200 further comprises a parameter setting module. The parameter setting module is specifically configured to set the material parameters of the non-Lambertian model. At this time, the second calculation module 230 is specifically configured to calculate the surface normal vector of each position point in the object 10 to be reconstructed according to the reflection formula of the non-Lambertian model based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters.

[0088] In some embodiments, the second calculation module 230 is specifically configured to set the initial value of the normal vector to be solved in the reflection formula; calculate the first brightness of each position point in the object 10 to be reconstructed according to the reflection formula based on the incident light vector of each pixel point in each reconstructed image and the material parameters; and perform optimization iteration on the normal vector to be solved based on the first brightness and the reconstructed image according to the nonlinear least square algorithm, thereby obtaining the corresponding surface normal vector of each position point.

[0089] In some embodiments, the height reconstruction module 240 is specifically configured to obtain the second brightness of each pixel point in each reconstructed image; calculate the brightness difference between the first brightness of each position point and the second brightness of the corresponding pixel point in each reconstructed image, thereby obtaining multiple brightness differences corresponding to the multiple sets of reconstructed images; and perform optimization iteration on the normal vector to be solved according to the sum of squares of the multiple brightness differences, thereby obtaining the corresponding surface normal vector of each position point.

[0090] In some embodiments, the height reconstruction module 240 is specifically configured to perform height reconstruction on the object 10 to be reconstructed according to the surface normal vector based on a height reconstruction algorithm, thereby obtaining the reconstruction result; wherein the height reconstruction algorithm includes any one of a global iteration algorithm, a frequency domain discrete cosine transform algorithm and a pyramid reconstruction algorithm.

[0091] It should be noted that the above-mentioned embodiments of the three-dimensional reconstruction method are also applicable to the three-dimensional reconstruction device 200 of the embodiments of the present application, and will not be described here.

[0092] Referring to Figure 17 The embodiment of the present application also provides a three-dimensional reconstruction system 300 based on a non-Lambert model. The three-dimensional reconstruction system 300 comprises one or more processors 310 and a memory 320. The memory 320 stores a computer program, and the computer program is executed by the processor 310 to implement the three-dimensional reconstruction method of any of the above embodiments.

[0093] For example, the computer program is executed by the processor 310 to implement the following three-dimensional reconstruction method: 010: image acquisition is performed on the object to be reconstructed 10 to obtain a plurality of groups of reconstruction images with different illumination angles; 020: the corresponding incident light vector is calculated according to the illumination angle of the reconstruction image; 030: the surface normal vector of the object to be reconstructed 10 is calculated according to the reflection formula of the non-Lambert model based on the incident light vector and the reconstruction image; 040: high reconstruction is performed on the object to be reconstructed 10 based on the surface normal vector to obtain a reconstruction result.

[0094] For another example, the computer program is executed by the processor 310 to implement the following three-dimensional reconstruction method: 011: control a plurality of light sources 20 located at different illumination angles to emit light in turn; 012: in the case that each light source 20 emits light, the image acquisition device 30 is controlled to perform image acquisition on the object to be reconstructed 10 to obtain a plurality of groups of reconstruction images; At this time, the corresponding incident light vector is calculated according to the illumination angle of the reconstruction image (i.e. 020), which comprises: 021: the corresponding incident light vector is calculated according to the device parameters of the image acquisition device 30 and the illumination angle of the reconstruction image.

[0095] It should be pointed out that the above-mentioned embodiments of the three-dimensional reconstruction method and the three-dimensional reconstruction device 200 are also applicable to the three-dimensional reconstruction system 300 of the embodiment of the present application, and will not be described here.

[0096] Referring to Figure 18 The embodiment of the present application also provides a computer readable storage medium 400, which stores a computer program 410. The program is executed by the processor 420 to implement the three-dimensional reconstruction method of any of the above embodiments.

[0097] For example, the program is executed by the processor 420 to implement the following three-dimensional reconstruction method: 010: image acquisition is performed on the object to be reconstructed 10 to obtain a plurality of groups of reconstruction images with different illumination angles; 020: calculating corresponding incident light vectors according to the light illumination angles of the reconstructed images; 030: calculating surface normal vectors of the object to be reconstructed 10 according to the reflection formula of the non-Lambertian model based on the incident light vectors and the reconstructed images; 040: performing height reconstruction on the object to be reconstructed 10 based on the surface normal vectors to obtain a reconstruction result.

[0098] For another example, when the program is executed by the processor 420, the following three-dimensional reconstruction method is implemented: 011: controlling multiple light sources 20 located at different light illumination angles to emit light in turn; 012: in the case that each light source 20 emits light, respectively controlling the image acquisition device 30 to perform image acquisition on the object to be reconstructed 10 to obtain multiple groups of reconstructed images; At this time, according to the light illumination angles of the reconstructed images, corresponding incident light vectors are calculated (i.e. 020), including: 021: calculating corresponding incident light vectors according to the device parameters of the image acquisition device 30 and the light illumination angles of the reconstructed images.

[0099] It should be noted that the above-mentioned implementation of the three-dimensional reconstruction method and the three-dimensional reconstruction device 200 is also applicable to the computer readable storage medium 400 of the present application, and will not be described here.

[0100] In summary, in the three-dimensional reconstruction method based on the non-Lambertian model, the three-dimensional reconstruction device 200, the three-dimensional reconstruction system 300 and the computer readable storage medium 400 of the present application, the three-dimensional reconstruction is performed based on the non-Lambertian model, multiple groups of reconstructed images with different light illumination angles are acquired, corresponding incident light vectors are calculated according to the light illumination angles of the reconstructed images, the surface normal vectors of the object to be reconstructed 10 are calculated according to the reflection formula of the non-Lambertian model, and the height reconstruction is performed on the object to be reconstructed 10 based on the surface normal vectors to obtain a reconstruction result. In this way, a three-dimensional reconstruction with high precision can be realized, and the reconstruction effect is good for objects with a large amount of highlight information, and the adaptability is strong.

[0101] In the description of the application, reference has been made to descriptive terms such as "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. Such terminology means that a particular feature, structure, material or characteristic being described is included in at least one embodiment or example of the application. The illustrative examples given are not necessarily mutually exclusive, and the illustrative features, structures, materials or characteristics can be combined in any suitable manner in any one or more embodiments or examples. Moreover, where the description states that a feature, structure, material or characteristic is included in at least one embodiment or example, it is understood that the feature, structure, material or characteristic is included in at least one embodiment or example of the application, unless the description states otherwise.

[0102] Any process or method described in flow charts or otherwise described herein can be understood as representing a module, segment, or portion of code that includes one or more executable instructions for implementing specific logical functions or steps, and the preferred embodiments of the application include additional or fewer steps, in different orders, including as appropriate, before, after, or in between illustrated steps, with the understanding that the application is not limited in this respect.

[0103] The logic and / or steps represented in flow charts or otherwise described herein, for example, can be embodied in non-transitory computer-readable storage medium that includes code that, when executed by a machine, causes the machine to perform such functions or steps, and the preferred embodiments of the application include additional or fewer steps, in different orders, including as appropriate, before, after, or in between illustrated steps, with the understanding that the application is not limited in this respect.

[0104] It should be understood that parts of the present application can be realized in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be realized as software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized in hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0105] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof. In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software function module. The integrated module, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0106] Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A 3D reconstruction method based on a non-Lambertian model, characterized in that: include: Perform image acquisition on the object to be reconstructed to obtain multiple sets of reconstructed images with different illumination angles; Calculating a corresponding incident light vector according to the illumination angle of the reconstructed image; Calculating a surface normal vector of the object to be reconstructed based on the incident light vector and the reconstructed image according to a reflection formula of the non-Lambertian model; The object to be reconstructed is height-reconstructed based on the surface normal vector to obtain a reconstruction result.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The image acquisition of the object to be reconstructed to obtain multiple groups of reconstructed images with different illumination angles includes: Control multiple light sources at different lighting angles to emit light in sequence; When each of the light sources emits light, the image acquisition device is controlled to acquire images of the object to be reconstructed, thereby obtaining a plurality of groups of reconstructed images; The calculating the corresponding incident light vector according to the illumination angle of the reconstructed image includes: The corresponding incident light vector is calculated according to the device parameters of the image acquisition device and the illumination angle of the reconstructed image.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The device parameters include a focal length and a pixel size, the illumination angle includes a rotation angle and a tilt angle, and calculating the corresponding incident light vector according to the device parameters of the image acquisition device and the illumination angle of the reconstructed image includes: The incident light vector of each pixel in each of the reconstructed images is calculated according to the focal length, the pixel size, the rotation angle and the tilt angle of the reconstructed image.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The three-dimensional reconstruction method further includes: Setting material parameters of the non-Lambertian model; The calculating the surface normal vector of the object to be reconstructed based on the incident light vector and the reconstructed image according to the reflection formula of the non-Lambertian model includes: Based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image and the material parameters, the surface normal vector of each corresponding position point in the object to be reconstructed is calculated according to the reflection formula of the non-Lambertian model.

5. The three-dimensional reconstruction method according to claim 4, characterized in that: The non-Lambertian model is a Cook-Torrance model, and the material parameters include any one or more of basic reflectivity, roughness, diffuse reflection color, diffuse reflection coefficient, and specular reflection coefficient.

6. The three-dimensional reconstruction method according to claim 4, characterized in that: The calculating, based on the incident light vector of each pixel point in each reconstructed image, the reconstructed image, and the material parameter, of the surface normal vector of each corresponding position point in the object to be reconstructed according to the reflection formula of the non-Lambertian model, includes: Setting the initial value of the normal vector to be solved in the reflection formula; Calculating a first brightness of each corresponding position point in the object to be reconstructed according to the reflection formula based on the incident light vector and the material parameter of each pixel point in each reconstructed image; Based on the first brightness and the reconstructed image, the normal vector to be solved is optimized and iterated according to a nonlinear least mean square algorithm to obtain the surface normal vector corresponding to each of the position points.

7. The three-dimensional reconstruction method according to claim 6, characterized in that: The optimizing and iterating the normal vector to be solved according to a nonlinear least mean square algorithm based on the first brightness and the reconstructed image to obtain the surface normal vector corresponding to each position point includes: Acquire a second brightness of each pixel in each of the reconstructed images; Calculating a brightness difference between the first brightness of each of the position points and the second brightness of the corresponding pixel point in each of the reconstructed images, to obtain a plurality of brightness differences corresponding to a plurality of groups of the reconstructed images; The normal vector to be solved is optimized and iterated according to the square sum of the multiple brightness differences to obtain the surface normal vector corresponding to each position point.

8. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of performing height reconstruction on the object to be reconstructed based on the surface normal vector to obtain a reconstruction result includes: Based on a height reconstruction algorithm, performing height reconstruction on the object to be reconstructed according to the surface normal vector to obtain the reconstruction result; The height reconstruction algorithm includes any one of a global iteration algorithm, a frequency domain discrete cosine transform algorithm, and a pyramid reconstruction algorithm.

9. A three-dimensional reconstruction device based on a non-Lambertian model, characterized in that: include: An image acquisition module is used to acquire images of the object to be reconstructed and obtain multiple sets of reconstructed images with different illumination angles; A first calculation module, configured to calculate a corresponding incident light vector according to the illumination angle of the reconstructed image; A second calculation module is configured to calculate a surface normal vector of the object to be reconstructed based on the incident light vector and the reflection formula of the non-Lambertian model; The height reconstruction module is used to perform height reconstruction on the object to be reconstructed based on the surface normal vector to obtain a reconstruction result.

10. A three-dimensional reconstruction system based on a non-Lambertian model, characterized in that: The three-dimensional reconstruction system includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the three-dimensional reconstruction method according to any one of claims 1 to 8 is implemented.

Images