Phase deflection and photometric stereo fused surface reconstruction method and system, computer readable storage medium and computer program product

Abstract

The invention relates to the technical field of image processing, and discloses a phase deflection and photometric stereo fused surface reconstruction method and system, a computer readable storage medium and a computer program product. The method comprises the following steps: respectively acquiring a plurality of fringe patterns with different phases in a first direction and a second direction of an object to be imaged; a reflection image of the object needing to be imaged after the binary images of the object needing to be imaged are projected in the multiple different directions is captured; a phase deflection method is used for solving a shape graph of the object, a captured reflection graph is used as photometric stereo input, an intensity matrix is constructed, and an average curvature graph of the object is calculated; and fusing the shape graph and the average curvature graph to obtain a final result graph. According to the method, the respective advantages of PMD and photometric stereo are comprehensively utilized, the adaptability to a complex surface is enhanced by dynamic weight, a finer effect can be achieved, and the multi-scene application capability of the prior art can also be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, a system, a computer-readable storage medium, and a computer program product for surface reconstruction by fusing phase deflectometry and photometric stereo. Background Art

[0002] Traditional PS (Photometric Stereo) technology is based on the Lambertian reflection model, estimating surface normal vectors through multiple images under different illumination conditions, and is applicable to diffuse reflection surfaces. However, this method is highly sensitive to light source calibration parameters (such as azimuth angle and incident intensity), and calibration errors directly affect the reconstruction accuracy. In addition, the actual object surface often contains a mixture of diffuse and specular reflection characteristics, resulting in distorted normal vector estimation and degraded reconstruction quality when traditional PS technology is used to process non-Lambertian surfaces.

[0003] PMD (Phase Measuring Deflectometry) projects structured fringes through a programmed light source, analyzing the phase change of the reflected fringes to measure the surface shape, and is particularly suitable for high-reflectivity specular surfaces (such as polished metals, optical components). However, when PMD is used to process diffuse reflection surfaces, due to the easy loss of phase information in low-reflectivity regions, the algorithm complexity is high and the computational resources consumed are large, making it difficult to ensure surface continuity and reconstruction accuracy.

[0004] It can be seen from this that a single reconstruction method is difficult to simultaneously meet the measurement requirements of diffuse and specular reflection surfaces. In the prior art, the fusion algorithms of multiple methods often rely on additional hardware or complex light source calibration, increasing the system cost and operation difficulty. Therefore, there is an urgent need in the art for a method that does not require calibration and can efficiently fuse the advantages of PMD and PS to achieve high-precision reconstruction of the entire domain of complex surfaces. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, a system, a computer-readable storage medium, and a computer program product for surface reconstruction by fusing phase deflectometry and photometric stereo, so as to solve or at least partially solve the technical problems mentioned in the above background art.

[0006] To achieve this purpose, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for surface reconstruction by fusing phase deflectometry and photometric stereo, including:

[0008] Collecting multiple fringe patterns with different phases in the first direction and the second direction of the object to be imaged; and capturing the reflected images after binary projections of the object to be imaged in multiple different directions;

[0009] Calculate the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio of all the captured fringe patterns using the phase deflection method;

[0010] Calculate the absolute phase maps of the object in the first direction and the second direction, and obtain the shape map of the object based on the absolute phase maps;

[0011] Use the captured reflection map as the input of photometric stereo to construct an intensity matrix;

[0012] Perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the three largest singular values, obtain the photometric stereo normal vector through normalization approximation, and calculate the average curvature map of the object;

[0013] Calculate the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, and fuse the shape map and the average curvature map to obtain the final result map.

[0014] Optionally, the capturing of the reflection maps after binary projections of the object to be imaged in multiple different directions specifically includes:

[0015] Simulate illuminations in multiple different directions of photometric stereo, and capture the reflection maps after binary projections of the object in the multiple different directions;

[0016] Among them, the multiple different directions include the four directions of up, down, left, and right of the object.

[0017] Optionally, in the first direction and the second direction of the object to be imaged, capturing fringe patterns with multiple different phases specifically includes:

[0018] Preset the shapes and periods of the grating fringes in the first direction and the second direction;

[0019] Project the preset grating fringes onto the object to be imaged respectively, and capture the fringe patterns with multiple different phases.

[0020] Optionally, denote the first direction as the x direction and the second direction as the y direction, and the luminance I of the captured fringe pattern n is expressed as:

[0021]

[0022] where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflection fringe phase to be demodulated;

[0023] The method for calculating the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio of all the captured fringe patterns using the phase deflection method is as follows:

[0024] Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G;

[0025] The calculation formulas for M, S, D, and G are respectively:

[0026]

[0027] D(x,y) = M(x,y) - S(x,y);

[0028]

[0029] Optionally, the multiple fringe maps with different phases include K first fringe maps with different phases in the first direction and K second fringe maps with different phases in the second direction;

[0030] Calculating the absolute phase maps of the object in the first direction and the second direction, and obtaining the shape map of the object according to the absolute phase maps, specifically includes:

[0031] Using the phase shift method, calculating the folded phase map in the first direction and the folded phase map in the second direction The method is:

[0032]

[0033] Performing phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is:

[0034]

[0035] According to the obtained absolute phases, obtaining the shape map P(x,y), the method is:

[0036]

[0037] where k is the period order corresponding to the folded phase; Φ x (x, y) is the absolute phase obtained by phase unwrapping, Φ y (x, y) is the absolute phase obtained by phase unwrapping.

[0038] Optionally, K is 4.

[0039] Optionally, the step of using the captured reflection map as the input of photometric stereo and constructing the intensity matrix specifically includes:

[0040] Flattening the captured reflection map to construct the intensity matrix M (f,p) ; The construction method of the intensity matrix is:

[0041] M (f，p) = UΣV T ;

[0042] Where f is the number of images of the captured reflection map, p is the number of pixel points of each captured reflection map, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

[0043] Optionally, performing SVD decomposition on the intensity matrix, selecting the right singular vectors corresponding to the largest 3 singular values, obtaining the photometric stereo normal vector through normalization approximation, and calculating the mean curvature map of the object, specifically including:

[0044] Performing SVD decomposition on the intensity matrix, and taking the right singular vectors corresponding to the largest 3 singular values to form matrix V' T (3×p) ;

[0045] For V' T (3×p) Performing normalization to obtain the photometric stereo normal vector N (3×p) , and the method is:

[0046]

[0047] According to the components (n (3×p) , n x , n y , n z ) of the photometric stereo normal vector N, calculating the gradient component p in the horizontal direction and the gradient component q in the vertical direction, and the method is:

[0048]

[0049] Using the gradient components p and q, calculating the mean curvature map H(x, y), and the method is:

[0050]

[0051] D = (1 + p 2 + q 2 );

[0052]

[0053] Where (r, c) is the projection coordinate of the gradient component, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0054] Optionally, calculate the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, and fuse the shape map and the mean curvature map to obtain the final result map, which specifically includes:

[0055] Calculate the dynamic fusion weight using the specular reflection map S(x, y) and the diffuse reflection map D(x, y), and the calculation method is:

[0056]

[0057] Fuse the shape map and the mean curvature map to obtain the final result map R(x, y), and the fusion method is:

[0058] R(x, y) = ω PMD P(x, y) + ω PS H(x, y).

[0059] Optionally, project the pre-set grating fringes onto the object to be imaged respectively, and collect the fringe maps with multiple different phases, specifically:

[0060] Use a display screen or a projector to project the pre-set grating fringes onto the object to be imaged;

[0061] Use an area array camera to collect the fringe maps with multiple different phases.

[0062] In a second aspect, the present invention provides a system for a surface reconstruction method that fuses phase deflection and photometric stereo, including:

[0063] A camera module, configured to respectively collect fringe maps with multiple different phases in a first direction and a second direction of the object to be imaged; and configured to capture the reflection map after binary map projection of the object to be imaged in multiple different directions;

[0064] A phase deflection processing module, electrically connected to the camera module, configured to calculate the average brightness map, specular reflection map, diffuse reflection map, and gloss ratio of all the collected fringe maps using the phase deflection method; and calculate the absolute phase maps of the object in the first direction and the second direction, and obtain the shape map of the object according to the absolute phase maps;

[0065] A photometric stereo processing module, electrically connected to the phase deflection processing module, configured to use the captured reflection map as the input of photometric stereo to construct an intensity matrix; and perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, obtain the photometric stereo normal vector through normalization approximation, and calculate the mean curvature map of the object;

[0066] A fusion module, electrically connected to the phase deflection processing module and the photometric stereo processing module, is configured to calculate a dynamic fusion weight based on the specular reflection map and the diffuse reflection map, and fuse the shape map and the mean curvature map to obtain a final result map.

[0067] Optionally, the camera module includes a programmable light source module, a display screen, and an area array camera;

[0068] The programmable light source module is configured to preset the shapes and periods of the grating stripes in the first direction and the second direction to be projected, and project periodic sine stripes onto the object to be imaged by using the display screen;

[0069] The area array camera is configured to collect multiple fringe maps with different phases of the object to be imaged in the first direction and the second direction.

[0070] Optionally, the programmable bar light source is further configured to simulate illuminations in multiple different directions of photometric stereo; the area array camera is further configured to capture the reflection maps of the object after binary map projections in the multiple different directions;

[0071] Wherein, the multiple different directions include the four directions of up, down, left, and right of the object.

[0072] Optionally, denote the first direction as the x direction and the second direction as the y direction, and the brightness I of the collected fringe map n is expressed as:

[0073]

[0074] Wherein, (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflection fringe phase to be demodulated;

[0075] The calculation methods of the average brightness map, the specular reflection map, the diffuse reflection map, and the gloss ratio are as follows:

[0076] Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G;

[0077] The calculation formulas of M, S, D, and G are respectively:

[0078]

[0079] D(x, y) = M(x, y) - S(x, y);

[0080]

[0081] Optionally, the multiple fringe patterns with different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction;

[0082] The phase deflection processing module is specifically configured to:

[0083] Using the phase shift method, calculate the folded phase map in the first direction and the folded phase map in the second direction The method is:

[0084]

[0085] Perform phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is:

[0086]

[0087] According to the obtained absolute phases, obtain the shape map P(x, y), and the method is:

[0088]

[0089] where k is the period order corresponding to the folded phase; Φ x (x, y) is the absolute phase obtained by phase unwrapping, Φ y (x, y) is the absolute phase obtained by phase unwrapping.

[0090] Optionally, K is 4.

[0091] Optionally, the photometric stereo processing module is specifically configured to flatten the captured reflection map and construct the intensity matrix M (f,p) ;

[0092] The construction method of the intensity matrix is:

[0093] M (f，p) = U∑V T ;

[0094] where f is the number of images of the captured reflection map, p is the number of pixel points of each captured reflection map, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

[0095] Optionally, the photometric stereo processing module is specifically configured to:

[0096] Perform SVD decomposition on the intensity matrix, and take the right singular vectors corresponding to the largest 3 singular values to form the matrix V' T (3×p);

[0097] Normalize V' T (3×p) to obtain the photometric stereo normal vector N (3×p) , and the method is as follows:

[0098]

[0099] According to the components (n (3×p) , n x , n y , n z ) of the photometric stereo normal vector N, calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction, and the method is as follows:

[0100]

[0101] Use the gradient components p and q to calculate the mean curvature map H(x, y), and the method is as follows:

[0102]

[0103] D = (1 + p 2 + q 2 );

[0104]

[0105] where (r, c) is the projection coordinate of the gradient component, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0106] Optionally, the fusion module is specifically used for:

[0107] Calculate the dynamic fusion weight by using the specular reflection map S(x, y) and the diffuse reflection map D(x, y), and the calculation method is as follows:

[0108]

[0109] Fuse the shape map and the mean curvature map to obtain the final result map R(x, y), and the fusion method is as follows:

[0110] R(x, y) = ω PMD P(x, y) + ω PS H(x, y).

[0111] Optionally, the display screen is an LCD.

[0112] In a third aspect, the present invention further provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method for surface reconstruction by fusing phase deflection and photometric stereo as described above.

[0113] In a fourth aspect, the present invention further provides a computer program product including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method for surface reconstruction by fusing phase deflection and photometric stereo as described above is implemented.

[0114] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0115] The present invention aims to provide a method for fusing PMD and PS without light source calibration. PMD calculates the shape map of the specular reflection part, and photometric stereo calculates the curvature map. The PMD is used to make up for the deficiency of photometric stereo in dealing with specular reflection. When performing image fusion, the weights are dynamically calculated through diffuse reflection and specular reflection components, comprehensively utilizing the respective advantages of PMD and photometric stereo. The dynamic weights enhance the adaptability to complex surfaces, achieving a more refined effect, improving the multi-scene application ability of the prior art, and enhancing the accuracy and applicability of three-dimensional reconstruction and defect detection of complex surfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0116] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0117] Figure 1 It is a flowchart of a method for surface reconstruction by fusing phase deflection and photometric stereo provided by an embodiment of the present invention.

[0118] Figure 2 It is a schematic diagram of the architecture of a system for a method for surface reconstruction by fusing phase deflection and photometric stereo provided by an embodiment of the present invention.

[0119] Figure 3-1 It is a grating fringe pattern in the first direction provided by an embodiment of the present invention.

[0120] Figure 3-2 It is a grating fringe pattern in the second direction provided by an embodiment of the present invention.

[0121] Figure 3-3 It is a binary fringe pattern provided by an embodiment of the present invention.

[0122] Figure 4An average brightness map of an object provided by an embodiment of the present invention.

[0123] Figure 5 A specular reflection map of an object provided by an embodiment of the present invention.

[0124] Figure 6 A diffuse reflection map of an object provided by an embodiment of the present invention.

[0125] Figure 7 A gloss ratio of an object provided by an embodiment of the present invention.

[0126] Figure 8 A shape map of an object provided by an embodiment of the present invention.

[0127] Figure 9 An average curvature map of an object provided by an embodiment of the present invention.

[0128] Figure 10 A final result map of an object provided by an embodiment of the present invention.

[0129] In the figure:

[0130] 10. Camera module; 20. Phase deflection processing module; 30. Photometric stereo processing module; 40. Fusion module. Detailed implementation manners

[0131] To make the object, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0132] Embodiment 1:

[0133] Please refer to Figure 1 , Figure 1 which is a flowchart of a method for surface reconstruction by fusing phase deflection and photometric stereo provided by an embodiment of the present invention. The method specifically includes:

[0134] Step 110: Collect a plurality of fringe patterns with different phases in the first direction and the second direction of the object to be imaged; and capture the reflection map after binary projection of the object to be imaged in a plurality of different directions.

[0135] Exemplarily, please refer to FIG. 3, Figure 3-1 which is a grating fringe pattern in the first direction provided by an embodiment of the present invention, Figure 3-2The grating fringe pattern in the second direction provided by the embodiment of the present invention Figure 3-3 The binary fringe pattern provided by the embodiment of the present invention

[0136] In this embodiment, taking the back panel of the mobile phone as an example, the back panel of the mobile phone is placed on the imaging platform, and light is applied to the object to be imaged, and images are collected in the first direction (horizontal x direction) and the second direction (vertical y direction) respectively

[0137] Specifically, as shown in Figure 3, the shapes and periods of the grating fringes in the first direction and the second direction are preset. For the convenience of distinction, the grating fringes in the set first direction are denoted as the first grating fringes, and the grating fringes in the set second direction are denoted as the second grating fringes

[0138] The preset grating fringes are projected onto the object to be imaged by using a display screen or a projector, and a planar array camera is used to collect fringe patterns with multiple different phases respectively; specifically including

[0139] The first grating fringes are projected onto the object to be imaged, and a planar array camera is used to collect multiple first fringe patterns with different phases respectively; the second grating fringes are projected onto the object to be imaged, and a planar array camera is used to collect multiple second fringe patterns with different phases respectively

[0140] In this embodiment, the total number of the collected first fringe patterns is equal to the total number of the second fringe patterns

[0141] More specifically, in this embodiment, the display screen is an LCD (Liquid Crystal Display)

[0142] Similarly, the preset binary fringe pattern is projected onto the object to be imaged by using a display screen or a projector, and the illuminations in multiple different directions of photometric stereo are simulated, and the reflection maps of the object after the binary maps are projected in multiple different directions are captured

[0143] It should be noted that the binary map is black and white (0 or 1), and there is no need to calibrate the position of the light source, and it can directly simulate the multi-directional illumination of PS; in this embodiment, the binary map includes binary maps in four directions: up (simulating full brightness at the top), down (simulating full brightness at the bottom), left (simulating full brightness on the left), and right (simulating full brightness on the right); the reflection maps captured by the camera can be sequentially denoted as I up 、I down 、I left 、I rightt 。

[0144] Step 120: Use the phase deflection method to calculate the average brightness map, specular reflection map, diffuse reflection map, and gloss ratio of all the collected fringe patterns

[0145] Specifically, let the first direction be Figure 3-1in the horizontal x direction, the second direction is Figure 3-2 in the vertical y direction, and the brightness I of the collected fringe pattern n is expressed as:

[0146]

[0147] where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation, is the reflection fringe phase to be demodulated;

[0148] Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G. The calculation method is:

[0149]

[0150] D(x, y) = M(x, y) - S(x, y);

[0151]

[0152] In this embodiment, according to the fringe pattern of the mobile phone backplane collected, the average brightness map, the specular reflection map, the diffuse reflection map, and the gloss ratio calculated are as Figures 4-7 shown;

[0153] where Figure 4 is the average brightness map of an object provided by an embodiment of the present invention, Figure 5 is the specular reflection map of an object provided by an embodiment of the present invention, Figure 6 is the diffuse reflection map of an object provided by an embodiment of the present invention, Figure 7 is the gloss ratio of an object provided by an embodiment of the present invention.

[0154] Step 130: Calculate the absolute phase maps of the object in the first direction and the second direction, and obtain the shape map of the object according to the absolute phase maps.

[0155] Further, let the total number of the first fringe pattern and the total number of the second fringe pattern collected both be K; K is a natural number greater than 1;

[0156] Step 130 specifically includes:

[0157] Step 131: Use the phase-shifting method to calculate the folded phase map in the first direction and the folded phase map in the second direction The method is:

[0158]

[0159] As an optional implementation manner, in this embodiment, K is taken as 4.

[0160] Step 132: Unwrap the two folded phase diagrams respectively to obtain the corresponding absolute phases. The method of phase unwrapping is as follows:

[0161]

[0162] Step 133: According to the obtained absolute phases, obtain the shape diagram P(x, y). The method is as follows:

[0163]

[0164] where k is the period order corresponding to the folded phase, k x (x, y) is the corresponding period order, k y (x, y) is the corresponding period order; Φ x (x, y) is the absolute phase obtained by phase unwrapping, Φ y (x, y) is the absolute phase obtained by phase unwrapping.

[0165] Exemplarily, please refer to Figure 8 , Figure 8 which is a shape diagram of an object provided by an embodiment of the present invention; that is; that is, according to Figures 4-7 the average luminance diagram, specular reflection diagram, diffuse reflection diagram, and gloss ratio shown, the calculated shape diagram is as Figure 8 shown.

[0166] Step 140: Use the captured reflection diagram as the input of photometric stereo to construct an intensity matrix.

[0167] The construction method of the intensity matrix M (f,p) is as follows:

[0168] M (f，p) = U∑V T ;

[0169] where f is the number of images of the captured reflection diagram (in this embodiment, f = 4, that is, 4 reflection diagrams captured under four simulated illumination directions on the front, back, left, and right of the mobile phone backplane), p is the number of pixel points of each captured reflection diagram, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

[0170] Step 150: Perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, approximately obtain the photometric stereo normal vector through normalization, and calculate the average curvature diagram of the object.

[0171] Step 150 specifically includes:

[0172] Perform SVD decomposition (Singular Value Decomposition) on the intensity matrix, and take the right singular vectors corresponding to the largest 3 singular values to form matrix V'. T (3×p) ;

[0173] For V' T (3×p) Perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is:

[0174]

[0175] According to the components (n (3×p) , n x , n y , n z ) of the photometric stereo normal vector N, calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction, and the method is:

[0176]

[0177] Use the gradient components p and q to calculate the mean curvature map H(x, y), and the method is:

[0178]

[0179] D = (1 + p 2 + q 2 );

[0180]

[0181] where (r, c) is the projection coordinate of the gradient component, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0182] Exemplarily, please refer to Figure 9 , Figure 9 which is the mean curvature map of an object provided by an embodiment of the present invention; that is, according to the intensity matrix constructed from the mean luminance map, specular reflection map, diffuse reflection map, and gloss ratio shown in Figures 4-7 , after performing SVD decomposition, normalization and other processing steps, the mean curvature map shown in Figure 9 is obtained.

[0183] Step 160, calculate the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, and fuse the shape map and the mean curvature map to obtain the final result map.

[0184] Specifically, step 160 includes:

[0185] Step 161: Calculate the dynamic fusion weight using the specular reflection map S(x, y) and the diffuse reflection map D(x, y). The calculation method is as follows:

[0186]

[0187] Step 162: Fuse the shape map and the mean curvature map to obtain the final result map R(x, y). The fusion method is as follows:

[0188] R(x, y) = ω PMD P(x, y) + ω PS H(x, y).

[0189] Exemplarily, as Figure 10 shown, Figure 10 is the final result map of an object provided by an embodiment of the present invention; that is, by fusing the shape map of Figure 8 and the mean curvature map of Figure 9 , the final result map as shown in Figure 10 is obtained.

[0190] This embodiment proposes a method for surface reconstruction that fuses phase deflection and photometric stereo. By utilizing the characteristics of PMD and PS, PMD calculates the shape map of the specular reflection part, and photometric stereo calculates the curvature map. PMD makes up for the deficiency of photometric stereo in dealing with specular reflection;

[0191] When performing image fusion, the weight is dynamically calculated through the diffuse reflection and specular reflection components, comprehensively utilizing the respective advantages of PMD and photometric stereo. The dynamic weight enhances the adaptability to complex surfaces, can achieve a more refined effect, and also improves the multi-scene application ability of the existing technology.

[0192] Embodiment 2:

[0193] Please refer to Figure 2 , Figure 2 which is the architecture schematic diagram of a system for surface reconstruction that fuses phase deflection and photometric stereo provided by an embodiment of the present invention. The system specifically includes:

[0194] The camera module 10 is used to collect multiple fringe patterns with different phases in the first direction and the second direction of the object to be imaged; and is used to capture the reflection map after the binary map projection of the object to be imaged in multiple different directions;

[0195] The phase deflection processing module 20 is electrically connected to the camera module 10 and is used to calculate the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio of all the collected fringe patterns using the phase deflection method; and calculate the absolute phase maps of the object in the first direction and the second direction, and obtain the shape map of the object based on the absolute phase maps.

[0196] The photometric stereo processing module 30 is electrically connected to the phase deflection processing module 20 and is used to use the reflection map as the input of photometric stereo to construct an intensity matrix; and perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, approximately obtain the photometric stereo normal vector through normalization, and calculate the average curvature map of the object.

[0197] The fusion module 40 is electrically connected to the phase deflection processing module 20 and the photometric stereo processing module 30 and is used to calculate the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, fuse the shape map and the average curvature map, and obtain the final result map.

[0198] Specifically, as shown in Figure 3, the camera module 10 includes a programmable strip light source, a display screen, and a matrix camera.

[0199] The programmable strip light source is used to preset the shape and period of the grating fringes in the first direction and the second direction to be projected on the display screen.

[0200] The display screen is used to project the preset grating fringes onto the object to be imaged.

[0201] The matrix camera is used to collect multiple fringe patterns with different phases of the object to be imaged in the first direction and the second direction, including: projecting the first grating fringe onto the object to be imaged, and using the matrix camera to collect multiple first fringe patterns with different phases respectively; projecting the second grating fringe onto the object to be imaged, and using the matrix camera to collect multiple second fringe patterns with different phases respectively; in this embodiment, the total number of the collected first fringe patterns is equal to the total number of the second fringe patterns.

[0202] In this embodiment, the display screen is an LCD (Liquid Crystal Display).

[0203] Similarly, use the display screen or the projector to project the preset binary fringe pattern onto the object to be imaged, simulate the illuminations in multiple different directions of photometric stereo, and capture the reflection map after the binary map projections of the object in multiple different directions.

[0204] It should be noted that the binary image is black and white (0 or 1), and the light source position does not need to be calibrated, and it can directly simulate the multi-directional illumination of PS; in this embodiment, the binary image includes binary images in four directions: up (simulating full brightness at the top), down (simulating full brightness at the bottom), left (simulating full brightness on the left), and right (simulating full brightness on the right); the reflected images captured by the camera can be sequentially denoted as I up , I down , I left , I rightt .

[0205] It should be noted that the camera module 10 in this embodiment adopts a coaxial system, and a programmable light source is used for lighting, so that the phase deflection and photometric stereo algorithm calculations can be completed simultaneously.

[0206] Exemplarily, as shown in Figure 3, let the first direction be the x direction in Figure 3, and the second direction be the y direction in Figure 3. The brightness I of the collected fringe pattern n is expressed as:

[0207]

[0208] where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflected fringe phase to be demodulated;

[0209] Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G. The calculation method is:

[0210]

[0211] D(x, y) = M(x, y) - S(x, y);

[0212]

[0213] As shown in Figure 3, when the object is a mobile phone, according to the collected fringe pattern, the calculated average brightness map, specular reflection map, diffuse reflection map and gloss ratio are as Figures 4-7 shown.

[0214] Specifically, the phase deflection processing module 20 is specifically used for:

[0215] Using the phase shift method, calculate the folded phase map in the first direction and the folded phase map in the second direction The method is:

[0216]

[0217] As a preferred implementation manner, in this embodiment, K is taken as 4.

[0218] Further, phase unwrapping is performed on the two folded phase diagrams respectively to obtain the corresponding absolute phases. The method of phase unwrapping is as follows:

[0219]

[0220] Further, according to the obtained absolute phases, the shape diagram P(x, y) is obtained. The method is as follows:

[0221]

[0222] where k is the period order corresponding to the folded phase; Φ x (x, y) is the absolute phase obtained by phase unwrapping, and Φ y (x, y) is the absolute phase obtained by phase unwrapping.

[0223] Exemplarily, please refer to Figure 8 , Figure 8 which is a shape diagram of an object provided by an embodiment of the present invention; that is, that is, according to Figures 4-7 the shown average luminance diagram, specular reflection diagram, diffuse reflection diagram, and gloss ratio, the calculated shape diagram is as Figure 8 shown.

[0224] Specifically, the photometric stereo processing module 30 is used to take the captured reflection diagram as the input of photometric stereo and construct an intensity matrix;

[0225] The method for constructing the intensity matrix M is as follows:

[0226] M (f，p) = UΣV T ;

[0227] where f is the number of images of the captured reflection diagram (in this embodiment, f = 4, that is, 4 reflection diagrams captured under four simulated illumination directions on the back of the mobile phone, namely up, down, left, and right), p is the number of pixel points of each captured reflection diagram, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are left singular vectors; V is the right singular value matrix, and its column vectors are right singular vectors.

[0228] Further, the photometric stereo processing module 30 is specifically used for:

[0229] Performing SVD decomposition on the intensity matrix, and taking the right singular vectors corresponding to the largest 3 singular values to form a matrix V' T (3×p) ;

[0230] Normalizing V' T (3×p) to obtain the photometric stereo normal vector N(3×p) , the method is as follows:

[0231]

[0232] Furthermore, the photometric stereo processing module 30 is specifically further configured to:

[0233] According to the components (n (3×p) , n x , n y ) of the photometric stereo normal vector N z , calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction. The method is as follows:

[0234]

[0235] Use the gradient components p and q to calculate the mean curvature map H(x, y). The method is as follows:

[0236]

[0237] D = (1 + p 2 + q 2 );

[0238]

[0239] where (r, c) are the projection coordinates of the gradient components, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0240] Exemplarily, please refer to Figure 9 , Figure 9 which is the mean curvature map of an object provided by an embodiment of the present invention; that is, according to the intensity matrix constructed from the mean brightness map, specular reflection map, diffuse reflection map, and gloss ratio shown in Figures 4-7 , after performing SVD decomposition, normalization and other processing steps, the mean curvature map shown in Figure 9 is obtained.

[0241] Specifically, the fusion module 40 is specifically configured to:

[0242] Set a fusion coefficient, and perform image fusion processing according to the shape map P(x, y) and the mean curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is as follows:

[0243] Calculate the dynamic fusion weight using the specular reflection map S(x, y) and the diffuse reflection map D(x, y). The calculation method is as follows:

[0244]

[0245] Fusing the shape map and the mean curvature map to obtain the final result map R(x, y), and the fusion method is as follows:

[0246] R(x, y) = ω PMD P(x, y) + ω PS H(x, y).

[0247] Exemplarily, as Figure 10 shown, Figure 10 is a surface reconstruction result map of an object provided by an embodiment of the present invention; that is, by fusing the shape map of Figure 8 and the mean curvature map of Figure 9 , the surface reconstruction result map as shown in Figure 10 is obtained.

[0248] In summary, in this embodiment, the programmable light source of PMD is used for lighting, so that the image light source information as the input of photometric stereo is controllable, which is an optimization of the unknown light source information of traditional photometric stereo;

[0249] The gloss ratio, diffuse reflection map, specular reflection map, and average luminance map calculated by PMD are used to simulate the gray-scale differences of the image in different states as the input of photometric stereo, and the curvature map is calculated. PMD makes up for the deficiency of photometric stereo in dealing with specular reflection;

[0250] When performing image fusion, according to the actual situation, by adjusting the fusion coefficient, the respective advantages of PMD and photometric stereo can be comprehensively utilized, a more refined effect can be achieved, the multi-scene application ability of the existing technology can also be improved, and global high-precision reconstruction can be realized.

[0251] Embodiment 3:

[0252] This embodiment also provides a computer-readable storage medium, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement a method for surface reconstruction by fusing phase deflection and photometric stereo as described in Embodiment 1.

[0253] Since Embodiment 1 has elaborated in detail on the method for surface reconstruction by fusing phase deflection and photometric stereo, it will not be elaborated in this embodiment.

[0254] Embodiment 4:

[0255] The present invention also provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, a method for surface reconstruction by fusing phase deflection and photometric stereo as described in Embodiment 1 is implemented.

[0256] Since the method of surface reconstruction by fusing phase deflection and photometric stereo has been described in detail in the first embodiment, it will not be elaborated again in this embodiment.

[0257] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware or by a program instructing the relevant hardware. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, etc. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks and other various media that can store program codes.

[0258] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for surface reconstruction integrating phase deflectometry and photometric stereo, characterized in that Including: Collecting fringe patterns with multiple different phases in the first direction and the second direction of the object to be imaged respectively; And capturing the reflection images after binary image projections of the object to be imaged in multiple different directions; Using the phase deflection method to calculate the average luminance map, specular reflection map, diffuse reflection map and gloss ratio of all the collected fringe patterns; Calculating the absolute phase maps of the object in the first direction and the second direction, and obtaining the shape map of the object according to the absolute phase maps; Taking the captured reflection images as the input of photometric stereo, and constructing an intensity matrix; Performing SVD decomposition on the intensity matrix, selecting the right singular vectors corresponding to the largest 3 singular values, approximately obtaining the photometric stereo normal vector through normalization, and calculating the average curvature map of the object; Calculating the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, and fusing the shape map and the average curvature map to obtain the final result map.

2. A method for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 1, wherein The capturing the reflection images after binary image projections of the object to be imaged in multiple different directions specifically includes: Simulating illuminations in multiple different directions of photometric stereo, and capturing the reflection images after binary image projections of the object in the multiple different directions; Wherein, the multiple different directions include the four directions of up, down, left and right of the object.

3. A method for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 2, characterized in that, The collecting fringe patterns with multiple different phases in the first direction and the second direction of the object to be imaged respectively specifically includes: Presetting the shapes and periods of the grating fringes in the first direction and the second direction; Projecting the preset grating fringes onto the object to be imaged respectively, and collecting the fringe patterns with multiple different phases.

4. A method for surface reconstruction that integrates phase deflectometry and photometric stereo according to claim 3, characterized in that, Denote the first direction as the x - direction and the second direction as the y - direction, and the luminance I of the acquired fringe pattern n is expressed as: where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, and B(x, y) is the fringe modulation degree, which is the phase of the reflected fringe to be demodulated; The using the phase deflection method to calculate the average luminance map, specular reflection map, diffuse reflection map and gloss ratio of all the collected fringe patterns, the specific method is: Let the average intensity map be represented as M, the specular reflection map be represented as S, the diffuse reflection map be represented as D, and the gloss ratio be represented as G; The calculation formulas of M, S, D and G are respectively: D(x,y) = M(x,y) - S(x,y); 5. A method for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 4, wherein The fringe patterns with multiple different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction; The calculating the absolute phase maps of the object in the first direction and the second direction, and obtaining the shape map of the object according to the absolute phase maps specifically includes: Using the phase-shifting method, the folded phase map in the first direction is calculated and the folded phase map in the second direction The method is as follows: Performing phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is: According to the obtained absolute phases, obtaining the shape map P(x,y), the method is: where k is the periodic order corresponding to the folded phase; Φ x (x, y) is the absolute phase obtained by phase unwrapping, Φ y (x, y) is the absolute phase obtained by phase unwrapping.

6. A method for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 5, characterized in that, K is 4.

7. A method for surface reconstruction integrating phase deflectometry and photometric stereo according to claim 5, characterized in that, The taking the captured reflection images as the input of photometric stereo, and constructing an intensity matrix specifically includes: Flatten the captured reflection map to construct the intensity matrix M (f,p) ; The method for constructing the intensity matrix is as follows: M (f，p) = U∑V T ; Wherein, f is the number of images of the captured reflection images, p is the number of pixel points of each captured reflection image, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are left singular vectors; V is the right singular value matrix, and its column vectors are right singular vectors.

8. A method for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 7, characterized in that, The performing SVD decomposition on the intensity matrix, selecting the right singular vectors corresponding to the largest 3 singular values, approximately obtaining the photometric stereo normal vector through normalization, and calculating the average curvature map of the object specifically includes: Perform SVD decomposition on the intensity matrix, and form matrix V' by taking the right singular vectors corresponding to the largest 3 singular values T (3×p) ; For V' T (3×p) Normalize it to obtain the photometric stereo normal vector N (3×p) , and the method is as follows: According to the photometric stereo normal vector N (3×p) component (n x , n y , n z ), calculate the horizontal gradient component p and the vertical gradient component q by the method: Using the gradient components p and q, calculate the mean curvature map H(x, y) by the method of: where (r, c) are the projection coordinates of the gradient component, is expressed as the gradient in the first direction, is expressed as the gradient in the second direction, and f is p or q.

9. A method for surface reconstruction by fusing phase deflectometry and photometric stereo according to claim 8, characterized in that, Calculating the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, and fusing the shape map and the mean curvature map to obtain the final result map, specifically including: Calculate the dynamic fusion weight using the specular reflection map S(x, y) and the diffuse reflection map D(x, y), and the calculation method is: Fuse the shape map and the mean curvature map to obtain the final result map R(x, y), and the fusion method is: R(x,y) = ω PMD P(x, y) + ω PS H(x, y).

10. A method for surface reconstruction integrating phase deflectometry and photometric stereo according to claim 3, characterized in that Project the pre-set grating fringes onto the object to be imaged respectively, and collect the fringe maps with multiple different phases, specifically: Use a display screen or a projector to project the pre-set grating fringes onto the object to be imaged; Use an area array camera to collect the fringe maps with multiple different phases.

11. A system for a surface reconstruction method that fuses phase deflectometry and photometric stereo, characterized in that, Including: A camera module, configured to collect fringe maps with multiple different phases in the first direction and the second direction of the object to be imaged respectively; And configured to capture the reflection map after the binary map projection of the object to be imaged in multiple different directions; A phase deflection processing module, electrically connected to the camera module, configured to use the phase deflection method to calculate the average luminance map, the specular reflection map, the diffuse reflection map and the gloss ratio of all the collected fringe maps; And calculate the absolute phase map of the object in the first direction and the second direction, and obtain the shape map of the object according to the absolute phase map; A photometric stereo processing module, electrically connected to the phase deflection processing module, configured to use the captured reflection map as the input of photometric stereo to construct an intensity matrix; And configured to perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, obtain the photometric stereo normal vector through normalization approximation, and calculate the mean curvature map of the object; A fusion module, electrically connected to the phase deflection processing module and the photometric stereo processing module, configured to calculate the dynamic fusion weight according to the specular reflection map and the diffuse reflection map, and fuse the shape map and the mean curvature map to obtain the final result map.

12. A system for a surface reconstruction method that fuses phase deflectometry and photometric stereo according to claim 11, characterized in that, The camera module includes a programmable light source module, a display screen and an area array camera; The programmable light source module is configured to preset the shape and period of the grating fringes in the first direction and the second direction to be projected, and use the display screen to project periodic sine fringes onto the object to be imaged; The area array camera is configured to collect fringe maps with multiple different phases of the object to be imaged in the first direction and the second direction.

13. The system of a surface reconstruction method integrating phase deflection and photometric stereo according to claim 12, characterized in that, The programmable strip light source is also configured to simulate illuminations in multiple different directions of photometric stereo; the area array camera is also configured to capture the reflection map after the binary map projection of the object in the multiple different directions; Wherein, the multiple different directions include the four directions of up, down, left and right of the object.

14. A surface reconstruction system integrating phase deflectometry and photometric stereo according to claim 13, characterized in that, Let the first direction be the x - direction and the second direction be the y - direction. The luminance I of the acquired fringe pattern n is expressed as: Among them, (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, and B(x, y) is the fringe modulation degree, which is the phase of the reflected fringe to be demodulated; The calculation methods of the average luminance map, the specular reflection map, the diffuse reflection map and the gloss ratio are: Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G; The calculation formulas of M, S, D and G are respectively: D(x, y) = M(x, y) - S(x, y); 15. A system for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 14, wherein, The multiple fringe patterns with different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction; The phase deflection processing module is specifically configured to: Using the phase shift method, the folded phase diagram in the first direction is calculated and the folded phase diagram in the second direction The method is as follows: Perform phase unwrapping on the two folded phase diagrams respectively to obtain the corresponding absolute phases: The method of phase unwrapping is: According to the obtained absolute phases, obtain the shape diagram P(x, y), and the method is: where k is the periodic order corresponding to the folded phase; Φ x (x, y) is the absolute phase obtained by phase unwrapping, Φ y (x, y) is the absolute phase obtained by phase unwrapping.

16. A surface reconstruction system integrating phase deflectometry and photometric stereo according to claim 15, characterized in that, K is 4.

17. A surface reconstruction system integrating phase deflectometry and photometric stereo according to claim 15, characterized in that The photometric stereo processing module is specifically configured to flatten the captured reflection map and construct an intensity matrix M (f,p) ; The method for constructing the intensity matrix is: Where f is the number of images of the captured reflection diagram, p is the number of pixel points of each captured reflection diagram, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are left singular vectors; V is the right singular value matrix, and its column vectors are right singular vectors.

18. A system for surface reconstruction that fuses phase deflectometry and photometric stereo according to claim 17, wherein, The photometric stereo processing module is specifically configured to: Perform SVD decomposition on the intensity matrix, and form matrix V' with the right singular vectors corresponding to the largest 3 singular values T (3×p) ; For V' T (3×p) Normalize it to obtain the photometric stereo normal vector N (3×p) , and the method is as follows: According to the photometric stereo normal vector N (3×p) components (n x , n y , n z ), calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction. The method is as follows: Use the gradient components p and q to calculate the mean curvature diagram H(x, y), and the method is: D = (1 + p 2 + q 2 ); where (r, c) are the projection coordinates of the gradient component, is expressed as the gradient in the first direction, is expressed as the gradient in the second direction, and f is p or q.

19. A surface reconstruction system integrating phase deflectometry and photometric stereo according to claim 18, characterized in that, The fusion module is specifically configured to: Calculate the dynamic fusion weight by using the specular reflection diagram S(x, y) and the diffuse reflection diagram D(x, y), and the calculation method is: Fuse the shape diagram and the mean curvature diagram to obtain the final result diagram R(x, y), and the fusion method is: R(x,y) = ω PMD P(x, y) + ω PS H(x, y).

20. A surface reconstruction system integrating phase deflectometry and photometric stereo according to claim 12, characterized in that The display screen is an LCD.

21. A computer-readable storage medium storing at least one instruction, characterized in that, The instructions are loaded and executed by the processor to implement a method for surface reconstruction by fusing phase deflection and photometric stereo as described in any one of claims 1-10.

22. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, a method for surface reconstruction by fusing phase deflection and photometric stereo as described in any one of claims 1-10 is implemented.

Images