Target surface normal reconstruction method and device based on physical constraint

Abstract

The invention discloses a target surface normal reconstruction method and device based on physical constraints, and belongs to the technical field of three-dimensional target reconstruction. The method comprises the steps of calculating target surface polarization characteristics based on test data of a polarization imaging measurement device; establishing an equation set of the relationship between the target surface normal and the depth; constructing zenith angle constraints based on the luminous flux idea, and correcting the relationship between the target surface normal and the depth by using the zenith angle constraints; constructing a loss function about depth by utilizing convexity prior constraint, and constraining the corrected equation set by utilizing the loss function; and solving the corrected equation set constrained by the loss function by using the target surface polarization characteristics to obtain zenith angle and depth estimation, and determining final target surface normal distribution by using the obtained zenith angle and depth estimation. According to the invention, more accurate normal distribution of the target surface can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional target reconstruction, and in particular to a method and device for reconstructing target surface normals based on physical constraints. Background Art

[0002] The reflected light from the surface of most objects generally contains three parts: specular reflected polarized light generated by direct reflection from the medium surface, diffuse reflected polarized light generated by the scattered and refracted light from the medium surface through the particles inside the object, and unpolarized light generated by scattering from tiny rough surface elements. By observing the polarization characteristics in the reflected light, the shape curvature of the object surface can be reflected, and the normal vector of the object surface can be calculated.

[0003] At present, the exploration of the three-dimensional structure of targets based on polarization information mainly uses the polarization information of reflected light to invert the structural shape of the target, and then completes the target's posture and position information analysis to further describe and realize the cognition and understanding of the target. However, when using the polarized light reflected from the target surface to reconstruct the surface normal, the degree of polarization does not change monotonically with the incident angle, resulting in ambiguity in the zenith angle of the normal, that is, the three-dimensional structure of the target is discontinuous or deformed. Summary of the invention

[0004] The present invention provides a method and device for reconstructing target surface normals based on physical constraints, which can solve the problem of discontinuity or deformation of target three-dimensional structure shape in related technologies. The technical solution is as follows:

[0005] On the one hand, a method for reconstructing a target surface normal based on physical constraints is provided, the method comprising:

[0006] Calculate the polarization characteristics of the target surface based on the test data of the polarization imaging measurement device;

[0007] Establish a set of equations for the relationship between the target surface normal and depth;

[0008] Based on the idea of ​​luminous flux, the zenith angle constraint is constructed to modify the relationship between the target surface normal and depth using the zenith angle constraint.

[0009] Constructing a loss function about depth by using a convexity prior constraint, so as to constrain the modified system of equations by using the loss function;

[0010] The target surface polarization characteristics are used to solve the modified equation group constrained by the loss function to obtain zenith angle and depth estimation, and the obtained zenith angle and depth estimation are used to determine the final target surface normal distribution.

[0011] On the other hand, a device for reconstructing target surface normal based on physical constraints is provided, the device comprising:

[0012] A calculation unit, used for calculating the polarization characteristics of the target surface based on the test data of the polarization imaging measurement device;

[0013] Establishing a unit for establishing a set of equations for the relationship between the target surface normal and the depth;

[0014] A correction unit, used to construct a zenith angle constraint based on the idea of ​​luminous flux, so as to correct the relationship between the normal and depth of the target surface by using the zenith angle constraint;

[0015] A constraint unit, used for constructing a loss function about depth by using a convexity prior constraint, so as to constrain the modified set of equations by using the loss function;

[0016] A determination unit is used to solve the modified equation group constrained by the loss function using the polarization characteristics of the target surface to obtain the zenith angle and depth estimation, and determine the final target surface normal distribution using the obtained zenith angle and depth estimation.

[0017] On the other hand, a computer device is provided, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of the above-mentioned target surface normal reconstruction method based on physical constraints.

[0018] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the target surface normal reconstruction method based on physical constraints are implemented.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the target surface normal reconstruction method based on physical constraints are implemented.

[0020] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0021] By introducing the zenith angle constraint to correct the equation group of the relationship between the target surface normal and the depth, one solution can be removed when there are two solutions for the zenith angle, so as to remove the ambiguity of the zenith angle and obtain the accurate zenith angle solution; by introducing the convexity prior constraint to construct the loss function about the depth, the loss function is used to constrain the corrected equation group, so that the observation vector approaches the implicit normal vector during the depth calculation, so as to solve the problem of shape discontinuity or deformation, so as to obtain a more accurate normal distribution of the target surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flow chart of a target surface normal reconstruction method based on physical constraints provided by an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of normal polarization coordinates of a micro-facet on a target surface provided by an embodiment of the present invention;

[0025] Figure 3 It is a structural diagram of a target surface normal reconstruction device based on physical constraints provided by an embodiment of the present invention;

[0026] Figure 4 It is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] Please refer to Figure 1 , an embodiment of the present invention provides a method for reconstructing a target surface normal based on physical constraints, the method comprising:

[0029] Step 100, calculating the polarization characteristics of the target surface based on the test data of the polarization imaging measurement device;

[0030] Step 102, establishing a set of equations for the relationship between the target surface normal and depth;

[0031] Step 104, constructing a zenith angle constraint based on the light flux concept, so as to modify the relationship between the target surface normal and the depth by using the zenith angle constraint;

[0032] Step 106, constructing a loss function about depth using a convexity prior constraint, so as to constrain the modified equation set using the loss function;

[0033] Step 108, using the target surface polarization characteristics to solve the modified equation group constrained by the loss function to obtain zenith angle and depth estimation, and using the obtained zenith angle and depth estimation to determine the final target surface normal distribution.

[0034] In an embodiment of the present invention, a zenith angle constraint is introduced to correct the set of equations for the relationship between the target surface normal and the depth. This allows one solution to be removed when there are two solutions for the zenith angle, thereby removing the ambiguity of the zenith angle and obtaining an accurate solution for the zenith angle. A convexity prior constraint is introduced to construct a loss function for depth, and the loss function is used to constrain the corrected set of equations so that the observed vector approaches the implicit normal vector during depth calculation, thereby solving the problem of shape discontinuity or deformation, thereby obtaining a more accurate normal distribution of the target surface.

[0035] Described below Figure 1 How the various steps are performed.

[0036] First, with respect to step 100 , the polarization characteristics of the target surface are calculated based on the test data of the polarization imaging measurement device.

[0037] In the embodiment of the present invention, the polarization characteristics of the target surface need to be calculated first, and the test data can be obtained based on the polarization imaging measurement device. The polarization imaging measurement device includes: a polarization analyzer, an imaging system, and a data acquisition and processing module. Specifically, the infrared radiation images I0, I1 of the target scene with polarization directions of 0°, 45°, 90°, and 135° are obtained by rotating the polarizer. 45 ,I 90 ,I 135 , and complete the infrared radiation image I of the target scene with polarization directions of 0°, 45°, 90° and 135° 0° ,I 45° ,I 90° ,I 135° After the corresponding registration, the Stokes parameter images S0, S1 and S2 are calculated according to the following formula.

[0038]

[0039] Among them, I θ is the registered radiation image when the polarizer is rotated by θ, where θ = 0°, 45°, 90°, 135°. RCP and I LCP They represent right-handed and left-handed circular polarization respectively. The circular polarization component of natural targets is small and can be ignored.

[0040] The polarization information of the target surface is obtained by calculating the Stokes parameter image. It is known that the degree of polarization represents a dimensionless number from 0 to 1, which is usually calculated by the Stokes parameter. The polarization characteristics of the target surface are calculated according to the following formula:

[0041]

[0042] Where p is the degree of polarization.

[0043] Then, for step 102, a set of equations for the relationship between the target surface normal and the depth is established.

[0044] Please refer to Figure 2 , is a schematic diagram of the normal polarization coordinates of the target surface micro-facet. Among them, θ is the zenith angle of the surface normal, which is used to characterize the angle between the observation direction and the normal direction; α is the azimuth angle, which is used to characterize the angle between the projection of the surface normal on the image plane and the coordinate axis. Given the polarization angle φ pol , then the light intensity I(φ pol ) can be expressed as:

[0045]

[0046] Where φ represents the phase angle of light, I max with I min are the upper and lower limits of the light intensity that the system can receive. The φ obtained from the above equation has an ambiguity of π. Based on the phase angle φ, the azimuth angle α of the surface normal has an ambiguity of π / 2, that is, when diffuse reflection accounts for the main component of the reflected light, α=φ; when specular reflection accounts for the main component, α=φ+π / 2.

[0047] The zenith angle θ of the surface normal is related to the degree of polarization p. The physical definition of the degree of polarization p is as follows:

[0048]

[0049] When diffuse reflection accounts for the main component, the relationship between the polarization degree, the zenith angle θ and the object's refractive index η is:

[0050]

[0051] When the refractive index is known, the zenith angle θ has a unique solution.

[0052] When specular reflection is the main component, the relationship between the degree of polarization and the zenith angle θ and the object's refractive index η is:

[0053]

[0054] At this time, there are two possible solutions for the zenith angle θ.

[0055] Based on the polarization characteristics of the target surface calculated based on the measured test data in step 100 and the relationship between the polarization degree and the zenith angle θ and the object refractive index η, the target surface normal map is roughly reconstructed in advance according to the specular reflection component and diffuse reflection component of the polarization image to obtain an approximate normal map, in which the zenith angle and azimuth angle may be ambiguous.

[0056] Since the normal vector distribution of the target surface can be regarded as the gradient of the depth distribution of the object in the three directions of x, y, and z, the problem of ambiguity in the zenith angle and azimuth angle can be transformed into a set of equations for establishing the relationship between the normal vector n of the target surface and the depth. Among them, the set of equations for the relationship between the normal vector n of the target surface and the depth z is:

[0057]

[0058] Next, for step 104, a zenith angle constraint is constructed based on the light flux concept, so as to utilize the zenith angle constraint to correct the relationship between the target surface normal and the depth.

[0059] On the basis of the approximate normal reconstruction in step 102, considering that there is a mapping relationship between the zenith angle θ and the refractive index η and the polarization degree p, that is, the relationship between the zenith angle and the normal vector is as follows:

[0060] n·v=cosθ

[0061] v=[0,0,1] T

[0062] Among them, v represents the observation direction. Since only diffuse reflection is considered, the polarization degree p of each pixel can be expressed as follows, including the refractive index η and the zenith angle θ of the object surface:

[0063]

[0064] According to the above formula, the expression of the zenith angle with respect to the refractive index η and the polarization degree p can be obtained by reverse calculation:

[0065]

[0066] According to the idea of ​​luminous flux, when the normal vector n is parallel to the illumination vector s, area dA receives the most light. As the angle θ between the normal vector n and the illumination vector s gradually increases, the amount of light received by area dA will become less and less. Since the materials at different positions of an object may be different, the albedo at different positions of the object is also different. Albedo is the ratio of the total radiation energy (M) emitted in all directions per unit time and per unit area to the total incident radiation energy (E). Therefore, according to Lambert's law, the following zenith angle constraint is obtained:

[0067] Iun =αn·s

[0068] Among them, I un is an unbiased image, α is the albedo, s is the illumination vector, and n is the normal vector. Substitute the equations for the relationship between the normal vector n and the depth z into n·v=cosθ and I un =αn·s, and replace the common As an intermediate equation, replace n·v=cosθ and I un =αn·s, etc., we can get the following modified equations:

[0069]

[0070] This constraint successfully combines the zenith angle θ, depth z, albedo α, illumination vector s, and unbiased image I un , and the observation direction v to simulate the luminous flux at various locations on the target surface under lighting conditions and achieve more precise constraints.

[0071] That is to say, according to the relationship between the zenith angle and the normal vector and the zenith angle constraint, the relationship between the target surface normal and the depth is corrected to obtain a corrected set of equations.

[0072] In this way, when there are two solutions for the zenith angle, the above-mentioned modified equation group can be used to remove the solution that does not satisfy the modified equation group from the two solutions, and the solution that satisfies the modified equation group is used as the determined zenith angle.

[0073] Further with respect to step 106, a loss function with respect to depth is constructed using a convexity prior constraint, so as to use the loss function to constrain the modified set of equations.

[0074] The convexity prior constraint is to ensure that the reconstructed target surface normal map has the correct convexity, prevent the reconstruction result from having too low a gradient, and reduce the situation where the azimuth is wrong. In addition to the azimuth calculated using the input data, the contour boundary of the target surface can also be considered to have an implicit implicit azimuth.

[0075] Therefore, by using the observed azimuth φ and the implicit azimuth φ inv Calculate the observed normal vector and the implicit normal vector, construct a loss function about the depth z, and use the least squares method to calculate the depth z so that the observed normal vector approaches the implicit normal vector.

[0076] n=[G x zcosθ,G y zcosθ,cosθ]

[0077] n inv =[sinφ inv 'sinθ,cosφinv 'sinθ,cosθ] T

[0078] D=[G x ,G y ]

[0079] Where n is the observed normal vector, n inv is the implicit normal vector, D represents the Sobel operator, G x and G y is the parameter of the Sobel operator, used to calculate the first-order differential at depth z.

[0080] Since we hope that the observed normal vector n can approach the implicit normal vector n inv , so the following loss function is adopted:

[0081]

[0082] Among them, ε is the loss function value, ω represents the weight distribution function of the implicit azimuth, which is an important parameter that allows the reconstructed image to correctly represent the concavity and convexity. Its expression is as follows:

[0083]

[0084] Among them, d b (u) represents the Euclidean distance between the pixel at position u and the nearest boundary pixel to u. max v∈F d b (v) represents the farthest distance between the pixel at v and the boundary pixel in the entire image, and m controls the decay speed of the weight along the boundary to the inside.

[0085] Finally, for step 108, the constrained modified equation group is solved using the target surface polarization characteristics to obtain zenith angle and depth estimation, and the obtained zenith angle and depth estimation are used to determine the final target surface normal distribution.

[0086] Specifically, in this step, the target surface polarization feature calculated in step 100 can be used to solve the modified equation group constrained by the loss function to obtain the value of the depth z, and the calculated depth z(x) can be used to update the albedo α(x) and zenith angle θ(x) of the image. The updated albedo α(x) and zenith angle θ(x) are then brought into the modified equation group constrained by the loss function to update the value of z(x), thereby obtaining the depth estimate z final Then use the Sobel operator D to calculate z final The gradient of is used to obtain the final target normal distribution.

[0087] Please refer to Figure 3The embodiment of the present invention provides a target surface normal reconstruction device based on physical constraints, the device comprising:

[0088] A calculation unit 300, used to calculate the polarization characteristics of the target surface based on the test data of the polarization imaging measurement device;

[0089] Establishing unit 302, used to establish a set of equations for the relationship between the target surface normal and the depth;

[0090] The correction unit 304 is used to construct a zenith angle constraint based on the light flux concept, so as to correct the relationship between the target surface normal and the depth by using the zenith angle constraint;

[0091] A constraint unit 306, configured to construct a loss function about depth using a convexity prior constraint, so as to constrain the modified set of equations using the loss function;

[0092] The determination unit 308 is used to solve the modified equation group constrained by the loss function using the polarization characteristics of the target surface to obtain the zenith angle and depth estimation, and determine the final target surface normal distribution using the obtained zenith angle and depth estimation.

[0093] In one embodiment of the present invention, the equation group of the relationship between the target surface normal and the depth is:

[0094]

[0095] Among them, n is the normal vector and z is the depth.

[0096] In one embodiment of the present invention, the zenith angle constraint is:

[0097] I un =αn·s

[0098] Among them, I un is the unbiased image, α is the albedo, and s is the illumination vector;

[0099] The correction unit is specifically used to correct the relationship between the target surface normal and the depth according to the relationship between the zenith angle and the normal vector and the zenith angle constraint, and obtain the following corrected equation group:

[0100]

[0101] Among them, S1, S2, S3 are Stokes parameters, and (v1, v2, v3) is the observation direction.

[0102] In one embodiment of the present invention, when the constraint unit constructs the loss function about the depth using the convexity prior constraint, it specifically includes: using the observed azimuth and the implicit azimuth, calculating the observed normal vector and the implicit normal vector, and constructing the following loss function about the depth so that the observed normal vector approaches the implicit normal vector:

[0103]

[0104] Among them, ε is the loss function value, ω represents the implicit azimuth φ inv The weight distribution function, D = [G x ,G y ] is the Sobel operator.

[0105] In one embodiment of the present invention, the determination unit is specifically used to: use the polarization characteristics of the target surface to solve the revised equation group constrained by the loss function to obtain an initial depth value; use the initial depth value to update the albedo and zenith angle, and substitute the updated albedo and zenith angle into the revised equation group constrained by the loss function to obtain an updated depth value; use the depth update value to obtain a depth estimate value; use the Sobel operator to calculate the gradient of the depth estimate to obtain the final target surface normal distribution.

[0106] It should be noted that the target surface normal reconstruction device based on physical constraints provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the target surface normal reconstruction device based on physical constraints provided in the above embodiment and the target surface normal reconstruction method embodiment based on physical constraints belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0107] The embodiment of the present application also provides a computer device, please refer to Figure 4 The computer device includes a processor and a memory, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the target surface normal reconstruction method based on physical constraints provided by the above-mentioned method embodiments.

[0108] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the target surface normal reconstruction method based on physical constraints provided in the above-mentioned method embodiments.

[0109] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes any of the physical constraint-based target surface normal reconstruction methods described in the above embodiments.

[0110] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0111] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0112] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0113] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A target surface normal reconstruction method based on physical constraints, characterized in that: The method comprises: Calculate the polarization characteristics of the target surface based on the test data of the polarization imaging measurement device; Establish a set of equations for the relationship between the target surface normal and depth; Based on the idea of ​​luminous flux, the zenith angle constraint is constructed to modify the relationship between the target surface normal and depth using the zenith angle constraint. Constructing a loss function about depth by using a convexity prior constraint, so as to constrain the modified system of equations by using the loss function; The target surface polarization characteristics are used to solve the modified equation group constrained by the loss function to obtain zenith angle and depth estimation, and the obtained zenith angle and depth estimation are used to determine the final target surface normal distribution.

2. The method according to claim 1, characterized in that The equation group of the target surface normal and depth relationship is: Among them, n is the normal vector and z is the depth.

3. The method according to claim 2, characterized in that The zenith angle constraint is: I un =αn·s Among them, I un is the unbiased image, α is the albedo, and s is the illumination vector; The method of using the zenith angle constraint to correct the relationship between the target surface normal and the depth includes: According to the relationship between the zenith angle and the normal vector and the zenith angle constraint, the relationship between the target surface normal and the depth is corrected to obtain the following corrected equation group: Among them, S1, S2, S3 are Stokes parameters, and (v1, v2, v3) is the observation direction.

4. The method according to claim 3, characterized in that The method of constructing a loss function about depth by using convexity prior constraints includes: Using the observed azimuth and implicit azimuth, the observed normal vector and implicit normal vector are calculated, and the following loss function about depth is constructed to make the observed normal vector approach the implicit normal vector: Among them, ε is the loss function value, ω represents the implicit azimuth φ inv The weight distribution function, D = [G x ,G y ] is the Sobel operator.

5. The method according to claim 4, characterized in that The method of solving the modified set of equations constrained by the loss function using the target surface polarization characteristics to obtain a zenith angle and a depth estimate, and determining a final target surface normal distribution using the obtained zenith angle and the depth estimate, includes: Solving the modified equations constrained by the loss function using the target surface polarization characteristics to obtain an initial depth value; The albedo and the zenith angle are updated using the initial depth value, and the updated albedo and zenith angle are substituted into the modified equation group constrained by the loss function to obtain the updated depth value; Obtain a depth estimate using the depth update value; The Sobel operator is used to calculate the gradient of the depth estimate and obtain the final target surface normal distribution.

6. A target surface normal reconstruction device based on physical constraints, characterized in that: The device comprises: A calculation unit, used for calculating the polarization characteristics of the target surface based on the test data of the polarization imaging measurement device; Establishing a unit for establishing a set of equations for the relationship between the target surface normal and the depth; A correction unit, used to construct a zenith angle constraint based on the idea of ​​luminous flux, so as to correct the relationship between the normal and depth of the target surface by using the zenith angle constraint; A constraint unit, used for constructing a loss function about depth by using a convexity prior constraint, so as to constrain the modified set of equations by using the loss function; A determination unit is used to solve the modified equation group constrained by the loss function using the polarization characteristics of the target surface to obtain the zenith angle and depth estimation, and determine the final target surface normal distribution using the obtained zenith angle and depth estimation.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images