Streak analysis method based on deep learning of physical information

Abstract

This invention discloses a deep learning-based fringe analysis method based on physical information, aiming to provide an efficient and accurate 3D contour analysis method. It includes projecting a single high-frequency fringe pattern onto the test scene using a projector, and acquiring a single frame of fringe image using a camera. An initial low-precision wrapping phase is obtained using a deep learning-enhanced Fourier transform contour measurement method, resulting in an initial phase map of the test scene. A lightweight DNN is used for phase analysis, and the Fourier loss function and phase loss function are calculated and backpropagated to output reliable, high-quality phase results for different types of samples. Combining a phase unfolding algorithm and a 3D reconstruction algorithm, the 3D contour of the test scene is reconstructed based on the relationship between the image coordinate system and the camera coordinate system. This invention improves the accuracy and efficiency of fringe analysis and has broad application prospects, especially suitable for precision industrial inspection and 3D reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of optical measurement technology, specifically a fringe analysis method based on deep learning of physical information. Background Technology

[0002] Optical metrology is a general metrology technique that utilizes light as the information carrier for non-contact measurements and is fundamental to manufacturing, basic research, and engineering applications. With the invention of lasers and CCDs, many optical metrology methods and instruments have been applied to state-of-the-art manufacturing processes, precision positioning, and quality assessment due to their capabilities in accuracy, sensitivity, repeatability, and speed. In optical metrology, the desired physical properties of an object (profile, distance, deformation, etc.) are encoded into observed measurements (e.g., deformed fringe / speckle images). The success of traditional optical metrology methods largely depends on the forward model of image formation and its inverse solution; therefore, they are called model-driven or physics-driven methods. However, these image-forming models are merely approximate estimates of actual phenomena governed by physical laws and are subject to interference from experimental environments (such as motion, vibration, and nonlinearity) and object surface characteristics (such as shine and translucency), leading to problems such as weak performance of simple models or the inability of complex models to be accurately reverse-engineered under poor measurement conditions.

[0003] With the explosive growth of available data and computing resources, deep learning, as a "data-driven" machine learning technique, has achieved impressive success in many fields, such as computer vision and computational imaging. Deep learning has permeated almost all aspects of optometry and provided solutions to many challenging problems, such as fringe denoising, fringe analysis, and digital holographic reconstruction. Phase retrieval from fringe images is a fundamental task and a typical example of the many applications of deep learning in optometry.

[0004] In recent years, high-speed 3D shape measurement has been widely applied in various fields, such as industrial quality inspection and 3D face recognition. Among many excellent methods, fringe projection profilometry (FPP), based on structured light and triangulation principles, has proven to be one of the high-performance techniques for measuring high-speed motion and rapid deformation due to its inherent non-contact, full-field, accurate, and efficient characteristics. Mainstream FPP methods can be broadly categorized into two fringe analysis algorithms: Fourier transform profilometry (FTP) and phase-shifting profilometry (PSP). FTP is well-suited for dynamic 3D acquisition, obtaining a phase map from a single fringe pattern; however, this method suffers from spectral overlap, limiting its measurement quality. Compared to FTP, PSP is more robust, achieving pixel-level phase measurement results with higher resolution and accuracy; however, it typically requires multiple fringe patterns to reconstruct the 3D shape of an object. Deep learning-based fringe analysis methods effectively address the shortcomings of PSP, obtaining high-precision wrapped phase through single-frame fringe analysis. However, deep learning-based fringe analysis networks suffer from poor generalization performance and unstable output quality, which can easily lead to uncertain errors in the 3D measurement process. Therefore, for deep learning-based fringe analysis methods, there is currently a lack of a more robust deep learning fringe analysis method to achieve high-precision 3D measurement. Summary of the Invention

[0005] To address the aforementioned technical deficiencies in existing technologies, this invention proposes a deep learning-based stripe analysis method based on physical information. By integrating a lightweight DNN with a learning-enhanced Fourier transform contour measurement module, the method overcomes the defect of poor generalization performance, thereby reconstructing a high-precision three-dimensional contour of the scene under test.

[0006] The technical solution to achieve the objective of this invention is: a stripe analysis method based on deep learning of physical information, comprising the following steps:

[0007] Step 1: Project a single high-frequency stripe pattern onto the scene under test, and the camera acquires a single frame of high-frequency stripe image. The initial wrapping phase is obtained by using the Fourier transform contour measurement method based on deep learning enhancement, and the initial phase map of the scene under test is obtained.

[0008] Step 2: Input the initial phase map and high-frequency fringe map into the lightweight network to predict and wrap the phase numerator and denominator to obtain the true phase map;

[0009] Step 3: Calculate the Fourier loss function values ​​of the initial wrapped phase map obtained in Step 1 and the true phase map obtained in Step 2, and use the phase loss function to calculate the phase loss value.

[0010] Step 4: Update the initial phase map to the true phase map obtained in step 2. Repeat steps 2 and 3 to iterate the network until the Fourier loss function value and the phase loss function value of the network converge, and obtain the numerator and denominator of the output of the lightweight DNN neural network.

[0011] Step 5: Obtain a high-precision wrap-around phase map based on the numerator and denominator output by the lightweight DNN neural network;

[0012] Step 6: Using phase unfolding algorithm and 3D reconstruction algorithm, reconstruct the 3D contour of the scene under test based on the relationship between the image coordinate system and the camera coordinate system.

[0013] Preferably, the specific method for obtaining the initial wrapping phase using the deep learning-enhanced Fourier transform contour measurement method is as follows:

[0014] Obtain the Fourier spectrum F of the high-frequency stripe image I in ;

[0015] After performing a Fourier transform and spectral centering on the input tensor, a learnable filter ω1 of size K1×H1×W1 is used to weaken the input spectrum F. in The zeroth order of center C1 is given by the following formula:

[0016]

[0017] Where F1 represents the spectrum obtained by weakening the zeroth order using the Fourier transform profilometry method, F in The Fourier spectrum of the high-frequency stripe image I is represented by o, which represents the Hadamard product.

[0018] First-order spectral information is extracted using a filter ω2 with a size of K2×H2×w2:

[0019]

[0020] The center of ω2 is set as C2, which is estimated by the N-step phase shift method.

[0021] Preferably, the lightweight network includes a context path, a spatial path, and a feature fusion module. The context path collects edge and phase features with a large receptive field by rapidly downsampling and encoding global context information, guiding the learning of refined high-level features. The spatial path captures and encodes spatial information, outputting low-level features. The features from the context path and the spatial path are connected by the feature fusion module, and the output is wrapped with phase numerator and denominator by upsampling prediction.

[0022] Preferably, the specific formula for calculating the Fourier loss value using the Fourier loss function in step 3 is as follows:

[0023]

[0024] The specific formula for calculating the phase loss value using the phase loss function is as follows:

[0025]

[0026] Where Y is the input fringe phase pattern, Y GT It is the true phase map obtained by the 12-step phase shift method, Y LeFTP It is a phase map obtained by Fourier transform profilometry, F Y , and It's Y, Y GT and Y LeFTP The 2D Discrete Fourier Transform output by the Deep Learning-Enhanced Fourier Transform Contour Measurement Module.

[0027] Preferably, in step 3, a high-precision encapsulated phase map φ(x,y) is obtained using the numerator M(x,y) and denominator D(x,y) output by a lightweight DNN neural network:

[0028]

[0029] Preferably, the wrapped phase output by the network is subjected to stereo phase unwrapping, and the three-dimensional contour of the scene under test is reconstructed according to the relationship between the image coordinate system and the camera coordinate system, so as to obtain the three-dimensional coordinate values ​​of the scene in the camera coordinate system (X). c ,Y c Z c ):

[0030]

[0031] Where norm(g) represents the modulo operation, T represents the translation vector between the left and right cameras, u0 and v0 are the principal point coordinate parameters of the cameras, and f x and f y Let d(i,j) be the focal length parameters of the camera in the x and y directions, and d(i,j) be the subpixel disparity value at point (i,j).

[0032] Compared with existing technologies, this invention offers significant advantages: It proposes a more robust, generalizable, and high-precision deep learning-based fringe analysis method. This method projects a single-frame fringe pattern onto the test scene, and a camera acquires the single-frame fringe image. Then, a lightweight deep learning-based DNN network outputs reliable, high-quality phase results. Phase unfolding and 3D reconstruction algorithms are then used to reconstruct the 3D contour of the test scene based on the relationship between the image coordinate system and the camera coordinate system. This invention proposes a deep learning-enhanced Fourier transform contour measurement method to obtain an initial low-precision wrapping phase, thus acquiring an initial wrapping phase map of the test scene. This provides guidance for the fringe map input to the lightweight network, resulting in high-precision phase. Furthermore, a lightweight DNN network is proposed, comprising a context path and a spatial path. The context path collects edge and phase features with a large receptive field by rapidly downsampling and encoding global contextual information, guiding the learning of refined high-level features. The spatial path captures spatial information encoding rich details and outputs low-level features.

[0033] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0034] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0035] Figure 1 This is a flowchart of a stripe analysis method based on physical information and deep learning.

[0036] Figure 2 This is a schematic diagram of a network model for a stripe analysis method based on physical information deep learning.

[0037] Figure 3 Comparison results of various single-frame stripe analysis methods for the David model. Detailed Implementation

[0038] It is readily understood that, based on the technical solution of this invention, various embodiments of the invention can be conceived by those skilled in the art without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention. Rather, these embodiments are provided to enable those skilled in the art to gain a more thorough understanding of the invention. Preferred embodiments of the invention are described below in conjunction with the accompanying drawings, which form part of this application and, together with the examples of the invention, serve to illustrate the innovative concept of the invention.

[0039] The present invention is based on a deep learning-based stripe analysis method using physical information, comprising the following five steps:

[0040] Step 1: Project a single high-frequency stripe pattern onto the scene under test using a projector to encode the image. Acquire a single frame of high-frequency stripe image using a camera. Utilize a deep learning-enhanced Fourier transform profilometry method to obtain the Fourier spectrum F of the high-frequency stripe image I. in After performing a Fourier transform and spectral centering on the input tensor, a learnable filter ω1 of size K1×H1×W1 is used to weaken the input spectrum F. in The zeroth order of the center C1, where K1 represents the zeroth order of the spectrum, and H1 and W1 represent the length and width of the extracted spectrum:

[0041]

[0042] Where o represents the Hadamard product.

[0043] Use another filter ω2 with a size of K2×H2×w2 to extract first-order spectral information, where K2 represents the first-order spectral order, and H2 and W2 represent the length and width of the extracted spectrum:

[0044]

[0045] Here, the center of ω2 is set as C2, which is estimated by N-step phase shift.

[0046] Due to the asymmetry of the spectrum, the initial phase can be recovered from the equation.

[0047] Step 2: Combine the initial phase pattern F2 obtained in Step 1 with the input high-frequency fringe pattern F in This guides a lightweight DNN to output reliable, high-quality phase results for different types of samples.

[0048] Specifically, the lightweight network includes a context path, a spatial path, and a feature fusion module. The context path collects edge and phase features with a large receptive field by rapidly downsampling and encoding global context information, guiding the learning of refined high-level features. The spatial path captures spatial information encoding rich details and outputs low-level features. Features from the context path and the spatial path are concatenated by the feature fusion module and upsampled to predict the output numerator M(x,y) and denominator D(x,y).

[0049] Step 3: Calculate the Fourier loss function values ​​of the initial wrapped phase map obtained in Step 1 and the true phase map obtained in Step 12 using the Fourier loss function; calculate the phase loss value using the phase loss function.

[0050] Specifically, the Fourier loss value is as follows:

[0051]

[0052] Specifically, the formula for calculating the phase loss value using the phase loss function is as follows:

[0053]

[0054] Where Y is the input fringe phase pattern, Y GT It is the true phase map obtained by the 12-step phase shift method, Y LeFTP It is a phase map obtained by Fourier transform profilometry, F Y , and It's Y, Y GT and Y LeFTP The 2D Discrete Fourier Transform output by the Deep Learning-Enhanced Fourier Transform Contour Measurement Module.

[0055] Step 4: Repeat steps 2 and 3 to iterate the network until the Fourier loss function and phase loss function values ​​of the network converge, and obtain the numerator and denominator of the output of the lightweight DNN neural network.

[0056] Step 5: Obtain the high-precision wrapper phase map φ(x,y) based on the numerator M(x,y) and denominator D(x,y) output by the lightweight DNN neural network. The specific formula is as follows:

[0057]

[0058] Step 6: Perform stereo phase unwrapping on the wrapped phase output by the network, and reconstruct the 3D contour of the scene under test based on the relationship between the image coordinate system and the camera coordinate system, to obtain the 3D coordinate values ​​of the scene in the camera coordinate system (X). c ,Y c Zc ):

[0059]

[0060] Where norm(g) represents the modulo operation, T represents the translation vector between the left and right cameras, u0 and v0 are the principal point coordinate parameters of the cameras, and f x and f y Let d(i,j) be the focal length parameters of the camera in the x and y directions, and d(i,j) be the subpixel disparity value at point (i,j).

[0061] This invention proposes a novel LeFTP method for phase retrieval, from... Figure 2 As can be seen in (c), this method utilizes a learned enhancement filter to adaptively extract the spectrum, improving upon the poor generalization of deep learning and successfully reducing the MAE error by approximately 18%. Furthermore, in Figure 2 In (a), LeFTP is combined with a lightweight DNN to further optimize phase quality. The network includes a context path and a spatial path. The context path collects edge and phase features with a large receptive field by rapidly downsampling and encoding global context information, guiding the learning of refined high-level features. The spatial path captures spatial information encoding rich details and outputs low-level features. (Comparison) Figure 3 As shown in (a)-(e), compared to U-Net, FTP, LeFTP, and Net head+LeFTP, the David model of this invention has a phase error of only 0.0741 rad, representing a significant improvement in accuracy. Furthermore, phase accuracy is significantly improved on complex surfaces. Additionally, the network's generalization ability is greatly enhanced under the guidance of the Fourier module's features. With the help of a lightweight DNN network, the model's parameter count is only 1.5G, and the network speed is also significantly improved, reaching 53.23 FPS.

[0062] This method significantly improves the analysis performance of single-frame images and possesses strong practicality and universality, making it widely applicable to various image processing tasks. Experimental results demonstrate that the LeFTP and DNN-integrated approach is innovative and superior, providing new ideas and methods for single-frame fringe analysis. The fringe analysis accuracy achieved using this method is approximately 0.05 rad, thus realizing high-precision three-dimensional measurement.

[0063] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0064] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single instance or with reference to a single figure. However, the present invention should not be construed as including all features included in the exemplary embodiments as essential technical features of the claims of this patent.

[0065] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

Claims

1. A stripe analysis method based on deep learning of physical information, characterized in that, Includes the following steps: Step 1: Project a single high-frequency stripe pattern onto the scene under test, and the camera acquires a single frame of high-frequency stripe image. The initial wrapping phase is obtained by using the Fourier transform contour measurement method based on deep learning enhancement, and the initial phase map of the scene under test is obtained. Step 2: Input the initial phase map and high-frequency fringe map into the lightweight network to predict and wrap the phase numerator and denominator to obtain the true phase map; Step 3: Calculate the Fourier loss function values ​​of the initial wrapped phase map obtained in Step 1 and the true phase map obtained in Step 2, and use the phase loss function to calculate the phase loss value. Step 4: Update the initial phase map to the true phase map obtained in step 2. Repeat steps 2 and 3 to iterate the network until the Fourier loss function value and the phase loss function value of the network converge, and obtain the numerator and denominator of the output of the lightweight DNN neural network. Step 5: Obtain a high-precision wrap-around phase map based on the numerator and denominator output by the lightweight DNN neural network; Step 6: Using phase unfolding algorithm and 3D reconstruction algorithm, reconstruct the 3D contour of the scene under test based on the relationship between the image coordinate system and the camera coordinate system.

2. The stripe analysis method based on physical information deep learning according to claim 1, characterized in that, The specific method for obtaining the initial wrapping phase using the deep learning-enhanced Fourier transform contour measurement method is as follows: Acquiring high-frequency stripe images Fourier spectrum ; After performing Fourier transform and spectral centering on the input tensor, the following is adopted: Learnable filters of size To weaken the input spectrum center The zeroth order is given by the following formula: in, This indicates the spectrum obtained by weakening the zeroth order using the Fourier transform profilometry method. Represents high-frequency stripe images Fourier spectrum Represents the Hadamard product; Use filters Extracting first-order spectral information from a value of K2×H2×w2: in, The center is set by The step phase shift method estimates .

3. The stripe analysis method based on deep learning of physical information according to claim 1, characterized in that, The lightweight network includes a context path, a spatial path, and a feature fusion module. The context path collects edge and phase features with a large receptive field by rapidly downsampling and encoding global context information, guiding the learning of refined high-level features. The spatial path captures and encodes spatial information and outputs low-level features. Features from the context path and spatial path are concatenated by the feature fusion module and the phase numerator and denominator are wrapped by upsampling prediction output.

4. The stripe analysis method based on deep learning of physical information according to claim 1, characterized in that, The specific formula for calculating the Fourier loss value using the Fourier loss function in step 3 is as follows: The specific formula for calculating the phase loss value using the phase loss function is as follows: in, It is the input fringe phase pattern. This is the true phase map obtained by the 12-step phase shift method. It is a phase map obtained by Fourier transform profilometry. , and yes , and The 2D Discrete Fourier Transform output by the Deep Learning-Enhanced Fourier Transform Contour Measurement Module.

5. The stripe analysis method based on deep learning of physical information according to claim 1, characterized in that, In step 3, the molecules output by a lightweight DNN neural network are used. and denominator Obtain a high-precision encapsulated phase map : 。 6. The stripe analysis method based on deep learning of physical information according to claim 1, characterized in that, The wrapped phase output by the network is subjected to stereo phase unwrapping, and the 3D contour of the scene under test is reconstructed based on the relationship between the image coordinate system and the camera coordinate system, thus obtaining the 3D coordinate values ​​of the scene in the camera coordinate system. : in, This represents the modulo operation. This represents the translation vector between the left and right cameras. and The principal point coordinates of the camera. and These are the focal length parameters of the camera in the x and y directions. For point The subpixel disparity value at that location.

Images