Surface reconstruction method and system based on phase deflection and photometric stereo fusion, computer readable storage medium and computer program product

Abstract

The invention relates to the technical field of image processing and machine vision, and discloses a surface reconstruction method and system based on phase deflection and photometric stereo fusion, a computer readable storage medium and a computer program product. The method comprises the following steps: solving a shape graph of an object by using a phase deflection method, and calculating an average curvature graph of the object according to a photometric three-dimensional normal vector; and fusing the shape graph and the average curvature graph to obtain a surface reconstruction result graph of the object. According to the method, the gray scale difference of the gloss ratio, the diffuse reflection image, the mirror reflection image and the average brightness image obtained through calculation with the phase deflection method in different states is simulated and serves as input of the photometric stereo, the curvature image is calculated, the defects of the photometric stereo during mirror reflection processing are overcome, the finer effect can be achieved, and the image quality is improved. And global high-precision reconstruction can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and machine vision, and in particular, to a method, a system, a computer-readable storage medium, and a computer program product for surface reconstruction based on the fusion of phase deflectometry and photometric stereo. Background Art

[0002] Traditional photometric stereo vision methods are based on the Lambertian reflection model. Although they can analyze the micro-texture features of the object surface, their reconstruction quality strongly depends on the light source calibration parameters. Specifically, measurement errors in parameters such as the light source azimuth angle and incident intensity will directly lead to deviations in the calculation results. In addition, the actual object surface usually exhibits non-Lambertian reflection characteristics that include diffuse reflection and specular reflection components, resulting in distortion phenomena when estimating the surface normal vector by traditional methods and significantly reducing the surface reconstruction accuracy.

[0003] For the phase deflectometry measurement technology, it is widely applicable to the detection of high-reflectivity mirror surfaces, and high-precision light source control can be achieved through a programmable light source (programmable light source array). However, this technology has problems such as high algorithm complexity and large consumption of computing resources when dealing with diffuse reflection surfaces, and phase information loss is prone to occur in low-reflectivity regions, resulting in the destruction of surface continuity.

[0004] Therefore, when there are both diffuse reflection and specular reflection regions on the surface of the object to be measured, the existing single-mode measurement technology cannot achieve high-precision reconstruction of the entire region. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, a system, a computer-readable storage medium, and a computer program product for surface reconstruction based on the fusion of phase deflectometry and photometric stereo, so as to solve or at least partially solve the technical problems mentioned in the above background art.

[0006] To achieve this purpose, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for surface reconstruction based on the fusion of phase deflectometry and photometric stereo, including: Collect a plurality of fringe patterns with different phases in the first direction and the second direction of the object; Use the phase deflectometry method to calculate the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio of all the collected fringe patterns; Calculate the absolute phase maps of the object in the first direction and the second direction respectively, and obtain the shape map of the object according to the absolute phase maps; Use the average luminance map, the specular reflection map, the diffuse reflection map, and the gloss ratio as the input of photometric stereo to construct an intensity matrix; Perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, and approximately obtain the photometric stereo normal vector through normalization; Calculate the average curvature map of the object based on the photometric stereo normal vector; Fuse the shape map and the average curvature map to obtain the surface reconstruction result map of the object.

[0007] Optionally, collecting multiple fringe patterns with different phases in the first direction and the second direction of the object specifically includes: Preset the shapes and periods of the grating fringes in the first direction and the second direction on the display screen; Project the preset grating fringes onto the object to be imaged respectively, and use an area array camera to collect the multiple fringe patterns with different phases.

[0008] Optionally, denote the first direction as the x direction, the second direction as the y direction, and the brightness I of the collected fringe pattern n is expressed as: ; where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflected fringe phase to be demodulated; Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G. The calculation methods are: ; ; ; .

[0009] Optionally, the multiple fringe patterns with different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction; The method of respectively calculating the absolute phase maps of the object in the first direction and the second direction and obtaining the shape map of the object based on the absolute phase maps specifically includes: Using the phase shift method, calculate the wrapped phase map in the first direction and the wrapped phase map in the second direction , and the method is: , ; Perform phase unwrapping on the two wrapped phase maps respectively to obtain the corresponding absolute phases. The method of phase unwrapping is: , ; Based on the obtained absolute phase, the shape map P(x, y) is obtained by the following method: ; where k is the period order corresponding to the folded phase; is the absolute phase obtained by phase unwrapping, is the absolute phase obtained by phase unwrapping.

[0010] Optionally, K is 4.

[0011] Optionally, the method for constructing the intensity matrix M is: ; where, f is the number of images, p is the number of pixel points in each image, and Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

[0012] Optionally, performing SVD decomposition on the intensity matrix, selecting the right singular vectors corresponding to the largest 3 singular values, and approximately obtaining the photometric stereo normal vector through normalization, specifically including: Performing SVD decomposition on the intensity matrix, and taking the right singular vectors corresponding to the largest 3 singular values to form a matrix V' T (3×p) ; For V' T (3×p) perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is: .

[0013] Optionally, calculating the mean curvature map of the object to be imaged according to the photometric stereo normal vector, specifically including: According to the components (n (3×p) , n x , n y , n z ) of the photometric stereo normal vector N , ; Using the gradient components p and q, calculate the mean curvature map H(x, y), and the method is: ; ; ; ; ; where (r, c) is the projection coordinate of the gradient component. is expressed as the gradient in the first direction. is expressed as the gradient in the second direction, and f is p or q.

[0014] Optionally, fusing the shape map and the mean curvature map to obtain the surface reconstruction result map of the object, specifically: Set the fusion coefficient , and perform image fusion processing on the shape map P(x, y) and the mean curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is: .

[0015] Optionally, the display screen is an LCD.

[0016] In a second aspect, the present invention provides a surface reconstruction system based on phase deflectometry and photometric stereo fusion, including: A fringe pattern acquisition module, configured to respectively acquire a plurality of fringe patterns with different phases in the first direction and the second direction of the object; A phase deflectometry processing module, electrically connected to the fringe pattern acquisition module, configured to use the phase deflectometry method to calculate the average luminance map, specular reflection map, diffuse reflection map and gloss ratio of all the acquired fringe patterns; and further configured to respectively calculate the absolute phase maps of the object in the first direction and the second direction, and obtain the shape map of the object according to the absolute phase maps; A photometric stereo processing module, electrically connected to the phase deflectometry processing module, configured to use the average luminance map, the specular reflection map, the diffuse reflection map and the gloss ratio as the input of photometric stereo to construct an intensity matrix; and further configured to perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, and approximately obtain the photometric stereo normal vector through normalization; and further configured to calculate the average curvature map of the object according to the photometric stereo normal vector; A fusion module, electrically connected to the photometric stereo processing module, configured to fuse the shape map and the average curvature map to obtain the surface reconstruction result map of the object.

[0017] Optionally, the fringe pattern acquisition module includes a programmable bar light source, a display screen and a area array camera; The programmable strip light source is used to preset the shape and period of grating stripes in the first direction and the second direction to be projected on the display screen; The display screen is used to project the preset grating stripes onto the object to be imaged; The area array camera is used to collect multiple fringe patterns with different phases of the object to be imaged in the first direction and the second direction.

[0018] Optionally, let the first direction be the x direction and the second direction be the y direction. The brightness I of the collected fringe pattern n is expressed as: ; where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation, is the reflected fringe phase to be demodulated; Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G. The calculation method is: ; ; ; .

[0019] Optionally, the multiple fringe patterns with different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction; The phase deflection processing module is specifically used for: Using the phase shift method, calculate the folded phase map in the first direction and the folded phase map in the second direction , the method is: , ; Perform phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is: , ; According to the obtained absolute phases, obtain the shape map P(x, y), the method is: ; where k is the period order corresponding to the folded phase; is the absolute phase obtained by phase unwrapping, is the absolute phase obtained by phase unwrapping.

[0020] Optionally, K is 4.

[0021] Optionally, the method for constructing the intensity matrix M is as follows: ; where f is the number of images, p is the number of pixel points in each image, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are left singular vectors; V is the right singular value matrix, and its column vectors are right singular vectors.

[0022] Optionally, the photometric stereo processing module is specifically configured to: Perform SVD decomposition on the intensity matrix, and take the right singular vectors corresponding to the largest 3 singular values to form a matrix V' T (3×p) ; For V' T (3×p) perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is: .

[0023] Optionally, the photometric stereo processing module is specifically configured to: According to the components (n (3×p) , n x , n y , n z ) of the photometric stereo normal vector N, calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction, and the method is: , ; Use the gradient components p and q to calculate the mean curvature map H(x, y), and the method is: ; ; ; ; ; where (r, c) is the projection coordinate of the gradient component, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0024] Optionally, the fusion module is specifically configured to: Set the fusion coefficient Perform image fusion processing based on the shape map P(x, y) and the mean curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is as follows: .

[0025] Optionally, the display screen is an LCD.

[0026] In a third aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method for surface reconstruction based on phase deflection and photometric stereo fusion as described above.

[0027] In a fourth aspect, the present invention also provides a computer program product including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method for surface reconstruction based on phase deflection and photometric stereo fusion as described above is implemented.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows: The method for surface reconstruction based on phase deflection and photometric stereo fusion provided by the present invention simulates the gray-scale differences of the image in different states by using the gloss ratio, diffuse reflection map, specular reflection map, and average luminance map calculated by the phase deflection method, and uses them as the input of photometric stereo to calculate the curvature map, making up for the deficiency of photometric stereo in dealing with specular reflection, achieving a more delicate effect, and enabling full-domain high-precision reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0030] Figure 1 It is a flowchart of a method for surface reconstruction based on phase deflection and photometric stereo fusion provided by an embodiment of the present invention.

[0031] Figure 2 It is a schematic architecture diagram of a system for surface reconstruction based on phase deflection and photometric stereo fusion provided by an embodiment of the present invention.

[0032] Figure 3 It is a schematic structural diagram of a fringe pattern acquisition module provided by an embodiment of the present invention.

[0033] Figure 4 It is an average luminance map of an object provided by an embodiment of the present invention.

[0034] Figure 5 This is a specular reflection diagram of an object provided by an embodiment of the present invention.

[0035] Figure 6 This is a diffuse reflection diagram of an object provided by an embodiment of the present invention.

[0036] Figure 7 This is a gloss ratio of an object provided by an embodiment of the present invention.

[0037] Figure 8 This is a shape diagram of an object provided by an embodiment of the present invention.

[0038] Figure 9 This is an average curvature diagram of an object provided by an embodiment of the present invention.

[0039] Figure 10 This is a surface reconstruction result diagram of an object provided by an embodiment of the present invention.

[0040] In the figure: 10. Fringe pattern acquisition module; 20. Phase deflection processing module; 30. Photometric stereo processing module; 40. Fusion module. Specific implementation manner

[0041] In order to make the object, features, and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] Embodiment 1: Please refer to Figure 1 , Figure 1 This is a flowchart of a method for surface reconstruction based on the fusion of phase deflection and photometric stereo provided by an embodiment of the present invention. The method specifically includes: Step 110: Acquire fringe patterns with multiple different phases in the first direction and the second direction of the object.

[0043] As an optional implementation manner, please refer to Figure 3 , Figure 3 This is a schematic structural diagram of the fringe pattern acquisition module provided by an embodiment of the present invention; Specifically, as shown in Figure 3As shown, the shapes and periods of the grating fringes in the first direction and the second direction are preset on the display screen. For the convenience of distinction, the grating fringes in the set first direction are denoted as the first grating fringes, and the grating fringes in the set second direction are denoted as the second grating fringes; Project the grating fringes onto the object to be imaged, and use an area array camera to collect multiple fringe patterns with different phases respectively; specifically including: Project the first grating fringes onto the object to be imaged, and use an area array camera to collect multiple first fringe patterns with different phases respectively; project the second grating fringes onto the object to be imaged, and use an area array camera to collect multiple second fringe patterns with different phases respectively; In this embodiment, the total number of the collected first fringe patterns is equal to the total number of the second fringe patterns.

[0044] In this embodiment, the display screen is an LCD (Liquid Crystal Display).

[0045] Step 120: Use the phase deflectometry method to calculate the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio of all the collected fringe patterns.

[0046] Exemplarily, as Figure 3 shown, assume the first direction is the Figure 3 x direction in Figure 3 and the second direction is the n y direction in. The luminance I of the collected fringe pattern is expressed as: where, (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflection fringe phase to be demodulated; Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G. The calculation method is: ; ; ; .

[0047] Exemplarily, as Figure 3 shown, when the object is a mobile phone, according to the collected fringe patterns, the calculated average luminance map, specular reflection map, diffuse reflection map, and gloss ratio are as Figures 4 - 7 shown; where, Figure 4 is the average luminance map of an object provided by the embodiment of the present invention, Figure 5A specular reflection map of an object provided by an embodiment of the present invention Figure 6 A diffuse reflection map of an object provided by an embodiment of the present invention Figure 7 A gloss ratio of an object provided by an embodiment of the present invention

[0048] Step 130: Calculate the absolute phase maps of the object in the first direction and the second direction respectively, and obtain the shape map of the object according to the absolute phase maps

[0049] Further, let the total number of the first stripe patterns and the total number of the second stripe patterns collected both be K; K is a natural number greater than 1 Step 130 specifically includes Step 131: Use the phase-shifting method to calculate the folded phase map in the first direction and the folded phase map in the second direction , and the method is , ; As an optional implementation manner, in this embodiment, K is taken as 4

[0050] Step 132: Perform phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is , ; Step 133: Obtain the shape map P(x, y) according to the obtained absolute phases, and the method is ; where k is the period order corresponding to the folded phase i.e the corresponding period order i.e the corresponding period order; is the absolute phase obtained by phase unwrapping is the absolute phase obtained by phase unwrapping

[0051] Exemplarily, please refer to Figure 8 , Figure 8 A shape map of an object provided by an embodiment of the present invention; that is; that is, according to Figures 4 - 7 the average luminance map, specular reflection map, diffuse reflection map and gloss ratio shown, the calculated shape map is as shown in Figure 8 ;

[0052] Step 140: Use the average luminance map, specular reflection map, diffuse reflection map and gloss ratio as the inputs of photometric stereo to construct an intensity matrix

[0053] The construction method of the intensity matrix M is as follows: ; Among them, f is the number of images, p is the number of pixel points in each image, Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

[0054] Step 150: Perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, and approximately obtain the photometric stereo normal vector through normalization.

[0055] Step 150 specifically includes: Perform SVD decomposition (Singular Value Decomposition) on the intensity matrix, and take the right singular vectors corresponding to the largest 3 singular values to form the matrix V' T (3×p) ; For V' T (3×p) Perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is: .

[0056] Step 160: Calculate the average curvature map of the object according to the photometric stereo normal vector.

[0057] Step 160 is specifically implemented as: , ; Use the gradient components p and q to calculate the average curvature map H(x, y), and the method is: ; ; ; ; ; Among them, (r, c) is the projection coordinate of the gradient component, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0058] Exemplarily, please refer to Figure 9 , Figure 9An average curvature map of an object provided by an embodiment of the present invention; that is, an intensity matrix constructed according to the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio shown in Figures 4 - 7 After performing SVD decomposition, normalization and other processing steps, the average curvature map shown in Figure 9 is obtained.

[0059] Step 170: Fuse the shape map and the average curvature map to obtain a surface reconstruction result map of the object.

[0060] Step 170 specifically includes: Set a fusion coefficient , and perform image fusion processing according to the shape map P(x, y) and the average curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is: .

[0061] Exemplarily, as shown in Figure 10 : Figure 10 A surface reconstruction result map of an object provided by an embodiment of the present invention; that is, after fusing the shape map of Figure 8 and the average curvature map of Figure 9 , the surface reconstruction result map shown in Figure 10 is obtained.

[0062] A surface reconstruction method based on phase deflectometry and photometric stereo fusion provided by this embodiment has at least the following beneficial effects compared with the prior art: 1. The gloss ratio, diffuse reflection map, specular reflection map, and average luminance map simulated images calculated by using PMD (Phase Measuring Deflectometry) to calculate the gray-scale differences in different states are used as the input of photometric stereo to calculate the curvature map, and PMD is used to make up for the deficiency of photometric stereo in dealing with specular reflection; 2. When performing image fusion, according to the actual situation, by adjusting the fusion coefficient, the respective advantages of PMD and photometric stereo can be comprehensively utilized, a more refined effect can be achieved, and the multi-scene application ability of the prior art can also be improved, and full-domain high-precision reconstruction can be realized.

[0063] Embodiment 2: Please refer to Figure 2 , Figure 2 which is the architecture schematic diagram of a surface reconstruction system based on phase deflectometry and photometric stereo fusion provided by an embodiment of the present invention. The system specifically includes: A fringe pattern acquisition module 10, configured to acquire a plurality of fringe patterns with different phases in the first direction and the second direction of the object respectively; The phase deflection processing module 20 is electrically connected to the fringe pattern acquisition module 10 and is used to calculate the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio of all the acquired fringe patterns using the phase deflection method; it is also used to calculate the absolute phase maps of the object in the first direction and the second direction respectively, and obtain the shape map of the object based on the absolute phase maps; The photometric stereo processing module 30 is electrically connected to the phase deflection processing module 20 and is used to use the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio as the inputs of photometric stereo to construct an intensity matrix; it is also used to perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, and obtain the photometric stereo normal vector through normalization approximation; it is also used to calculate the average curvature map of the object based on the photometric stereo normal vector; The fusion module 40 is electrically connected to the photometric stereo processing module 30 and is used to fuse the shape map and the average curvature map to obtain the surface reconstruction result map of the object.

[0064] Specifically, as Figure 3 shown, the fringe pattern acquisition module 10 includes a programmable bar light source, a display screen, and a area array camera; The programmable bar light source is used to preset the shape and period of the grating fringes in the first direction and the second direction to be projected on the display screen; The display screen is used to project the preset grating fringes onto the object to be imaged; The area array camera is used to acquire multiple fringe patterns with different phases of the object to be imaged in the first direction and the second direction, including: projecting the first grating fringe onto the object to be imaged, and using the area array camera to acquire multiple first fringe patterns with different phases respectively; projecting the second grating fringe onto the object to be imaged, and using the area array camera to acquire multiple second fringe patterns with different phases respectively; in this embodiment, the total number of the acquired first fringe patterns is equal to the total number of the second fringe patterns.

[0065] In this embodiment, the display screen is an LCD (Liquid Crystal Display).

[0066] Exemplarily, as Figure 3 shown, let the first direction be Figure 3 the x direction in Figure 3 and the second direction be n the y direction in, and the luminance I of the acquired fringe pattern is expressed as: where, (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflection fringe phase to be demodulated; Let the average intensity map be denoted as M, the specular reflection map as S, the diffuse reflection map as D, and the gloss ratio as G. The calculation method is as follows: ; ; ; .

[0067] As Figure 3 shown, when the object is a mobile phone, according to the collected fringe pattern, the calculated average brightness map, specular reflection map, diffuse reflection map, and gloss ratio are as Figures 4 - 7 shown.

[0068] Specifically, the phase deflection processing module 20 is specifically used for: Using the phase shift method, calculate the folded phase map in the first direction and the folded phase map in the second direction , and the method is: , ; As a preferred implementation, in this embodiment, K is taken as 4.

[0069] Furthermore, perform phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases. The method of phase unwrapping is: , ; Furthermore, according to the obtained absolute phases, obtain the shape map P(x,y), and the method is: ; where k is the periodic order corresponding to the folded phase; is the absolute phase obtained by phase unwrapping, is the absolute phase obtained by phase unwrapping.

[0070] Exemplarily, please refer to Figure 8 , Figure 8 which is a shape map of an object provided by an embodiment of the present invention; that is; that is, according to Figures 4 - 7 the shown average brightness map, specular reflection map, diffuse reflection map, and gloss ratio, the calculated shape map is as Figure 8 shown.

[0071] Specifically, the construction method of the intensity matrix M is: ; where f is the number of images, pis the number of pixels in each image, and Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

[0072] Further, the photometric stereo processing module 30 is specifically configured to: perform SVD decomposition on the intensity matrix, and take the right singular vectors corresponding to the largest 3 singular values to form a matrix V ' T (3×p) ; For V' T (3×p) perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is: .

[0073] Further, the photometric stereo processing module 30 is specifically further configured to: According to the components (n (3×p) , n x , n y ) of the photometric stereo normal vector N z , calculate the horizontal gradient component p and the vertical gradient component q, and the method is: , ; Use the gradient components p and q to calculate the mean curvature map H(x, y), and the method is: ; ; ; ; ; where (r, c) is the projection coordinate of the gradient component, represents the gradient in the first direction, represents the gradient in the second direction, and f is p or q.

[0074] Exemplarily, please refer to Figure 9 , Figure 9 is the mean curvature map of an object provided by an embodiment of the present invention; that is, according to Figures 4 - 7 the intensity matrix constructed by the average luminance map, specular reflection map, diffuse reflection map, and gloss ratio shown, after performing SVD decomposition, normalization and other processing steps, the mean curvature map as shown in Figure 9 is obtained.

[0075] Specifically, the fusion module 40 is specifically configured to: Set the fusion coefficient , and perform image fusion processing on the shape map P(x, y) and the mean curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is as follows: .

[0076] Exemplarily, as Figure 10 shown, Figure 10 is the surface reconstruction result map of an object provided by an embodiment of the present invention; that is, by Figure 8 shape map and Figure 9 mean curvature map are fused, the surface reconstruction result map as shown in Figure 10 is obtained.

[0077] The present invention proposes a surface reconstruction method based on phase deflection and photometric stereo fusion. Compared with the prior art, it has at least the following beneficial effects: 1. Using the programmable light source of PMD for lighting enables the controllability of the image light source information as the input of photometric stereo, which is an optimization of the unknown light source information in traditional photometric stereo. 2. Using the gloss ratio, diffuse reflection map, specular reflection map, and average luminance map calculated by PMD to simulate the gray-scale differences of images in different states as the input of photometric stereo, calculating the curvature map, and using PMD to make up for the deficiency of photometric stereo in dealing with specular reflection. 3. When performing image fusion, according to the actual situation, by adjusting the fusion coefficient, the respective advantages of PMD and photometric stereo can be comprehensively utilized, achieving a more refined effect, improving the multi-scene application ability of the prior art, and enabling high-precision reconstruction in the entire domain.

[0078] Embodiment 3: This embodiment also provides a computer-readable storage medium, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement a surface reconstruction method based on phase deflection and photometric stereo fusion as described in Embodiment 1.

[0079] Since Embodiment 1 has elaborated in detail on the surface reconstruction method based on phase deflection and photometric stereo fusion, it will not be repeated in this embodiment.

[0080] Embodiment 4: The present invention also provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, a surface reconstruction method based on phase deflection and photometric stereo fusion as described in Embodiment 1 is implemented.

[0081] Since the method of surface reconstruction based on the fusion of phase deflectometry and photometric stereo has been described in detail in the first embodiment, it will not be elaborated again in this embodiment.

[0082] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware or by a program instructing relevant hardware. The described program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, or an optical disc, etc. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage media include: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

[0083] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for surface reconstruction based on the fusion of phase deflectometry and photometric stereo, characterized in that Comprising: Collecting fringe patterns with multiple different phases respectively in the first direction and the second direction of the object; Calculating the average luminance map, specular reflection map, diffuse reflection map and gloss ratio of all the collected fringe patterns by using the phase deflection method; Calculating the absolute phase maps of the object in the first direction and the second direction respectively, and obtaining the shape map of the object according to the absolute phase maps; Taking the average luminance map, the specular reflection map, the diffuse reflection map and the gloss ratio as the inputs of photometric stereo, and constructing an intensity matrix; Performing SVD decomposition on the intensity matrix, selecting the right singular vectors corresponding to the largest 3 singular values, and approximately obtaining the photometric stereo normal vector through normalization; Calculating the average curvature map of the object according to the photometric stereo normal vector; Fusing the shape map and the average curvature map to obtain the surface reconstruction result map of the object.

2. The method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 1, wherein, The collecting fringe patterns with multiple different phases respectively in the first direction and the second direction of the object specifically includes: Presetting the shapes and periods of the grating fringes in the first direction and the second direction on the display screen; Projecting the preset grating fringes onto the object to be imaged respectively, and using a area array camera to collect the fringe patterns with multiple different phases.

3. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 1, characterized in that Denoting the first direction as the x direction and the second direction as the y direction, the luminance In of the collected fringe pattern is expressed as: ； where (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, and B(x, y) is the fringe modulation degree, which is the phase of the reflected fringe to be demodulated; Letting the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G, the calculation methods are: ； ； ； 。 4. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 3, characterized in that, The fringe patterns with multiple different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction; The calculating the absolute phase maps of the object in the first direction and the second direction respectively, and obtaining the shape map of the object according to the absolute phase maps specifically includes: Using the phase-shifting method, the folded phase map in the first direction is calculated and the folded phase map in the second direction , and the method is as follows: ， ； Performing phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is: ， ； Obtaining the shape map P(x, y) according to the obtained absolute phases, the method is: ； where k is the periodic order corresponding to the folded phase; is the absolute phase obtained by phase unwrapping, is the absolute phase obtained by phase unwrapping.

5. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 4, characterized in that, K is 4.

6. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 4, characterized in that, The construction method of the intensity matrix M is: ； Among them, f is the number of images, p is the number of pixel points in each image, and Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are the left singular vectors; V is the right singular value matrix, and its column vectors are the right singular vectors.

7. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 6, characterized in that The performing SVD decomposition on the intensity matrix, selecting the right singular vectors corresponding to the largest 3 singular values, and approximately obtaining the photometric stereo normal vector through normalization specifically includes: Perform SVD decomposition on the intensity matrix, and form a matrix with the right singular vectors corresponding to the largest 3 singular values V' T (3×p) ; For V' T (3×p) perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is as follows: 。 8. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 7, characterized in that, The calculating the average curvature map of the object to be imaged according to the photometric stereo normal vector specifically includes: According to the photometric stereo normal vector N (3×p) components (n x , n y , n z ), calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction. The method is as follows: ， ； Calculating the average curvature map H(x, y) by using the gradient components p and q, the method is: ； ； ； ； ； where (r, c) are the projection coordinates of the gradient component is expressed as the gradient in the first direction is expressed as the gradient in the second direction, and f is p or q 9. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 8, characterized in that, The fusing the shape map and the average curvature map to obtain the surface reconstruction result map of the object is specifically: Set the fusion coefficient , perform image fusion processing based on the shape map P(x, y) and the mean curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is as follows: 。 10. A method for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 2, characterized in that, The display screen is an LCD.

11. A surface reconstruction system based on the fusion of phase deflection and photometric stereo, characterized in that, Comprising: A fringe pattern acquisition module, configured to collect fringe patterns with multiple different phases respectively in the first direction and the second direction of the object; A phase deflection processing module, electrically connected to the fringe pattern acquisition module, configured to calculate the average luminance map, specular reflection map, diffuse reflection map and gloss ratio of all the collected fringe patterns by using the phase deflection method; and is also configured to calculate the absolute phase maps of the object in the first direction and the second direction respectively, and obtain the shape map of the object according to the absolute phase maps; The photometric stereo processing module is electrically connected to the phase deflectometry processing module, and is used to take the average luminance map, the specular reflection map, the diffuse reflection map, and the gloss ratio as the inputs of photometric stereo, and construct an intensity matrix; It is also used to perform SVD decomposition on the intensity matrix, select the right singular vectors corresponding to the largest 3 singular values, and approximately obtain the photometric stereo normal vector through normalization; it is also used to calculate the average curvature map of the object according to the photometric stereo normal vector; The fusion module is electrically connected to the photometric stereo processing module, and is used to fuse the shape map and the average curvature map to obtain the surface reconstruction result map of the object.

12. A surface reconstruction system based on the fusion of phase deflection and photometric stereo according to claim 11, characterized in that, The fringe pattern acquisition module includes a programmable strip light source, a display screen, and a matrix camera; The programmable strip light source is used to preset the shapes and periods of the grating fringes in the first direction and the second direction to be projected on the display screen; The display screen is used to project the preset grating fringes onto the object to be imaged; The matrix camera is used to collect multiple fringe patterns with different phases of the object to be imaged in the first direction and the second direction.

13. A system for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 11, characterized in that, Denote the first direction as the x - direction and the second direction as the y - direction. The luminance I of the acquired fringe pattern n is expressed as: ； Among them, (x, y) are the pixel coordinates of the image, A(x, y) represents the background light intensity distribution, B(x, y) is the fringe modulation degree, is the reflection fringe phase to be demodulated; Let the average intensity map be denoted as M, the specular reflection map be denoted as S, the diffuse reflection map be denoted as D, and the gloss ratio be denoted as G. The calculation method is as follows: ； ； ； 。 14. A system for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 13, wherein The multiple fringe patterns with different phases include K first fringe patterns with different phases in the first direction and K second fringe patterns with different phases in the second direction; The phase deflectometry processing module is specifically used for: Using the phase shift method, the folded phase diagram in the first direction is calculated and the folded phase diagram in the second direction , and the method is as follows: ， ； Performing phase unwrapping on the two folded phase maps respectively to obtain the corresponding absolute phases: The method of phase unwrapping is as follows: ， ； According to the obtained absolute phases, the shape map P(x, y) is obtained. The method is as follows: ； where k is the period level corresponding to the folded phase; is the absolute phase obtained by phase unwrapping, is the absolute phase obtained by phase unwrapping.

15. A surface reconstruction system based on the fusion of phase deflection and photometric stereo according to claim 14, characterized in that, K is 4.

16. A system for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 14, characterized in that, The construction method of the intensity matrix M is as follows: ； Among them, f is the number of images, p is the number of pixel points of each image, and Σ is the singular value matrix; U is the left singular value matrix, and its column vectors are left singular vectors; V is the right singular value matrix, and its column vectors are right singular vectors.

17. A surface reconstruction system based on the fusion of phase deflection and photometric stereo according to claim 16, characterized in that, The photometric stereo processing module is specifically used for: Perform SVD decomposition on the intensity matrix, and form a matrix with the right singular vectors corresponding to the largest 3 singular values V' T (3×p) ; For V' T (3×p) perform normalization to obtain the photometric stereo normal vector N (3×p) , and the method is as follows: 。 18. A surface reconstruction system based on the fusion of phase deflection and photometric stereo according to claim 17, characterized in that, The photometric stereo processing module is specifically used for: According to the photometric stereo normal vector N (3×p) 's components (n x , n y , n z ), calculate the gradient component p in the horizontal direction and the gradient component q in the vertical direction. The method is as follows: ， ； Using the gradient components p and q, calculate the average curvature map H(x, y). The method is as follows: ； ； ； ； ； where (r, c) are the projection coordinates of the gradient component, is expressed as the gradient in the first direction, is expressed as the gradient in the second direction, and f is p or q.

19. A system for surface reconstruction based on the fusion of phase deflection and photometric stereo according to claim 18, characterized in that, The fusion module is specifically used for: Set the fusion coefficient , perform image fusion processing according to the shape map P(x, y) and the mean curvature map H(x, y) to obtain the surface reconstruction result map R(x, y). The method is as follows: 。 20. A surface reconstruction system based on the fusion of phase deflection and photometric stereo according to claim 12, characterized in that, The display screen is an LCD.

21. A computer-readable storage medium storing at least one instruction, characterized in that, Instructions are loaded and executed by a processor to implement a surface reconstruction method based on the fusion of phase deflectometry and photometric stereo as described in any one of claims 1-10.

22. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instructions are executed by a processor, a surface reconstruction method based on the fusion of phase deflectometry and photometric stereo as described in any one of claims 1-10 is implemented.

Images