A three-dimensional reconstruction method, device and medium based on multi-frequency fringe synchronous optimization

Abstract

The application belongs to the technical field of computer vision and optical measurement, and discloses a three-dimensional reconstruction method based on multi-frequency stripe synchronous optimization, equipment and medium, which comprises the following steps: based on a monocular structured light system, a plurality of groups of sinusoidal stripe images with different frequencies are projected onto the surface of an object to be measured by a projector, and a plurality of phase steps are performed respectively, and corresponding image sequences are collected by using a camera; the relative phase value of a pixel point is obtained by using a structured light phase shift method; initial absolute phase values and modulation intensities are obtained based on the structured light phase shift method and a multi-frequency heterodyne method; and the absolute phase and modulation intensity parameters of the multi-frequency stripe are jointly optimized by using a Gauss-Newton iteration method. By using the above method, the anti-noise capability and reconstruction precision are improved, the parameter adjustment and threshold recovery mechanism are introduced in the optimization process, the convergence stability and abnormal value elimination are ensured, and the method is suitable for high-precision structured light application scenarios such as industrial vision measurement and complex surface detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and optical measurement, and particularly relates to a three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization, a device and a medium. BACKGROUND

[0002] As a high-precision optical three-dimensional measurement method, multi-frequency phase shift structured light technology is widely used in industrial vision detection, complex surface reconstruction and other fields due to its high resolution, fast speed and other advantages. The technology projects a series of sinusoidal fringe patterns with different frequencies onto the measured object, uses a monocular camera to capture the deformed fringe images, recovers the absolute phase distribution through phase calculation and unwrapping process, and finally realizes three-dimensional point cloud reconstruction.

[0003] However, there are many error sources in the actual measurement process, including nonlinear response of the projector and the camera, environmental light interference, image sensor noise and uneven surface reflection characteristics, which seriously affect the phase calculation accuracy. The traditional phase calculation method is usually based on the ideal sinusoidal fringe assumption, and the wrapped phase is calculated independently for each frequency and then demodulated by heterodyne, which has poor robustness to the above error sources. Although some methods try to compensate through pre-calibration or filtering, they often rely on high-precision hardware or introduce additional operation complexity, and fail to fully exploit the internal constraints between multi-frequency images, resulting in insufficient phase accuracy and stability in noisy environments.

[0004] In addition, existing phase optimization methods mostly focus on error correction of a single frequency, or only optimize phase information while ignoring the joint action of amplitude parameters, and fail to achieve global optimization of multi-frequency observation information at the pixel level. SUMMARY

[0005] The purpose of the present application is to provide a three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization, a device and a medium, aiming to improve the phase accuracy and three-dimensional reconstruction stability in multi-frequency phase shift structured light systems.

[0006] To achieve the above purpose, the present application provides a three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization, comprising the following steps:

[0007] Step S1, based on a monocular structured light system, a projector projects a plurality of sets of sinusoidal fringe images with different frequencies onto the surface of the measured object, and a plurality of phase steps are performed respectively, and a camera is used to capture the corresponding image sequence;

[0008] Step S2, using structured light phase shift method, the relative phase value of the pixel point is obtained;

[0009] Step S3, based on structured light phase shift method and multi-frequency heterodyne method, the initial absolute phase value and the modulation intensity are obtained;

[0010] Step S4, the absolute phase and the modulation intensity parameter of the multi-frequency fringe are jointly optimized by Gauss-Newton iteration method.

[0011] Preferably, the monocular structured light system comprises a structured light projector, a monocular structured light camera and a control device connected to the projector and the monocular camera.

[0012] The structured light projector is configured to project the coded image, and the monocular structured light camera is configured to capture the coded image.

[0013] Preferably, in step S1, first, the structured light projector projects seventeen images, one all-white image, four four-step phase shift images with a wave number of N1, four four-step phase shift images with a wave number of N2, and eight eight-step phase shift images with a wave number of N3; then, the monocular structured light camera captures images using hard triggering while the structured light projector projects images.

[0014] Preferably, in step S2, the structured light phase shift method is used to obtain the relative phase value of the pixel point, and the specific process is as follows:

[0015] The standard N-step phase shift model is as follows:

[0016] ;

[0017] wherein, represents a two-dimensional image pixel point; represents background light intensity; represents modulation intensity; is the number of phase shift steps, ; is the total number of phase shift steps; represents the intensity of the pixel of the step image; is a cosine function; is the relative phase value of the pixel point, and is as follows:

[0018] ;

[0019] wherein, is a tangent function; is a sine function.

[0020] Preferably, in step S3, based on the structured light phase shift method and the multi-frequency heterodyne method, the initial absolute phase value and the modulation intensity are obtained, and the specific process is as follows:

[0021] Step S31, based on the relative phase obtained by the structured light phase shift method, four-step, four-step and eight-step phase shift calculation is performed on each group of frequency fringe images to obtain the wrapped relative phase under each frequency wherein represents frequency;

[0022] Step S32, the initial absolute phase is obtained by recursive solution using multi-frequency heterodyne method, as follows:

[0023] ;

[0024] wherein, is the initial absolute phase value; represents rounding operation; , and represents the equivalent phase obtained by subtracting the relative phase at different frequencies; , and are the stripe periods corresponding to different frequencies;

[0025] Step S33, the modulation intensity at different frequencies is obtained using structured light phase shifting method, as follows:

[0026] ;

[0027] wherein, is the phase shift step number; is the total phase shift step number; represents the intensity of the pixel in the i-th image.

[0028] Preferably, in step S4, the initial absolute phase and modulation intensity parameters of the multi-frequency stripe are jointly optimized by Gauss-Newton iterative method, and the specific process is as follows:

[0029] Step S41, the gray value and the initial absolute phase value of the current image pixel in different frequencies are extracted;

[0030] Step S42, the joint residual sum of squares is constructed as the optimization objective function , as follows:

[0031] ;

[0032] wherein, is the initial absolute phase value to be optimized; , and are the modulation intensities at different frequencies, respectively; , and are the actually measured gray values at different wave numbers, respectively; , and are the phase offsets at different wave numbers, respectively; is the background light intensity; ​

[0033] where the residual is defined as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] where, , and represent the individual intensity residuals under different wave numbers and different phase offsets, respectively;

[0038] Therefore, the total objective function is represented as:

[0039] ;

[0040] Step S43, partial derivatives are solved for parameters , , and respectively, each row of the Jacobian matrix corresponds to the partial derivative of each parameter, and the joint equation is constructed as follows:

[0041] ;

[0042] where, represents the Jacobian matrix; represents the parameter update amount, represents the initial absolute phase update amount; , and are the different frequency modulation intensity update amounts, respectively; represents the joint residual vector, and each component is the difference between the model gray value and the actual gray value under different frequency and phase offset conditions; represents the transpose;

[0043] Step S44, a regularization parameter is introduced on the diagonal to construct an augmented matrix as follows:

[0044] ;

[0045] where, is an identity matrix; is a regularization parameter; by Cholesky decomposition or SVD decomposition, the augmented matrix constructed is solved to obtain the parameter update amount ;

[0046] Step S45: Use adjustment factors to gradually update the initial absolute phase and modulation intensity, and perform error control.

[0047] Preferably, in step S45, when the nth iteration calculation is completed, the parameter update amount is... The values ​​are compared with a preset standard to determine whether to output the absolute phase and modulation intensity after this iteration.

[0048] First, set the initial absolute phase. The preset standard value is Modulation intensity , and The preset standard value is ;in ;like , , , Stop the iteration and calculate the absolute phase value after the current iteration. ;

[0049] Then, update the absolute phase value. With the initial absolute phase value absolute value of the difference , compared with the preset standard value If a comparison is made, Then output the updated absolute phase value. Otherwise, do not trust the optimization and output the initial absolute phase value. ;

[0050] If the number of iterations reaches the preset maximum number of iterations At that time, parameter update amount If the preset criteria are not met, the iteration stops and the initial absolute phase value is output. .

[0051] A computer device includes a memory and a processor, the memory being used to store instructions and the processor being used to execute the instructions to implement a three-dimensional reconstruction method based on multi-frequency stripe synchronization optimization as described above.

[0052] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a three-dimensional reconstruction method based on multi-frequency stripe synchronization optimization as described above.

[0053] Therefore, the present invention employs the above-mentioned three-dimensional reconstruction method, equipment, and medium based on multi-frequency fringe synchronous optimization, and the beneficial effects are as follows:

[0054] (1) The absolute phase and modulation degree parameters of multiple frequencies are jointly modeled and optimized, the defects that the traditional single frequency optimization is easily affected by noise and local distortion are overcome, the optimization result is more in line with the real optical imaging rule, and the reconstruction accuracy of the system is significantly improved.

[0055] (2) The regularization and step factor control are introduced in the iteration process, the divergence and oscillation problems in the traditional nonlinear optimization are effectively avoided, the optimization process is more stable and reliable, and the robustness under the complex imaging conditions such as high reflection, low reflection and multiple exposure is ensured.

[0056] (3) By uniformly constraining the multi-frequency phase residual through Gauss-Newton iteration, the convergence speed can be greatly improved while ensuring the calculation accuracy, which is suitable for fast processing of large-scale point cloud data, and meets the dual requirements of efficiency and accuracy in industrial detection, surface defect detection and other application scenarios.

[0057] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 It is a flow chart of a three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization. DETAILED DESCRIPTION

[0059] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples.

[0060] As shown in Figure 1 A three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization comprises the following steps:

[0061] Step S1, based on a monocular structured light system, a plurality of groups of sinusoidal fringe images with different frequencies are projected onto the surface of the object to be measured by a projector, and a plurality of phase steps are performed respectively, and a camera is used to collect corresponding image sequences;

[0062] Step S2, using structured light phase shift method, the relative phase value of the pixel point is obtained;

[0063] Step S3, based on structured light phase shift method and multi-frequency heterodyne method, the initial absolute phase value and modulation intensity are obtained;

[0064] Step S4, the absolute phase and modulation intensity parameters of the multi-frequency fringe are jointly optimized by Gauss-Newton iteration method.

[0065] EMBODIMENT

[0066] This invention provides a three-dimensional reconstruction method based on multi-frequency fringe synchronization optimization, which can improve the phase accuracy and three-dimensional reconstruction stability of multi-frequency phase-shift structured light systems. The method specifically includes the following steps:

[0067] Step S1: Based on a monocular structured light system, project multiple sets of sinusoidal fringe images of different frequencies onto the surface of the object under test using a projector, perform multiple phase steps, and use a camera to acquire the corresponding image sequences.

[0068] A monocular structured light system includes: a structured light projector, a monocular structured light camera, and a control device connecting the projector and the monocular camera. The structured light projector is used to project coded images; the monocular structured light camera is used to acquire coded images.

[0069] First, a structured light projector projects seventeen images: one pure white image, four four-step phase-shifted images with wavenumber N1, four four-step phase-shifted images with wavenumber N2, and eight eight-step phase-shifted images with wavenumber N3. Then, while the structured light projector projects the images, a monocular structured light camera uses hard triggering to acquire the images.

[0070] Step S2: Use the structured light phase shift method to obtain the relative phase value of the pixel.

[0071] The standard N-step phase shift model is shown below:

[0072] ;

[0073] in, Represents pixels in a two-dimensional image; Indicates the intensity of an image pixel; Indicates background light intensity; Indicates modulation intensity; For the number of phase shift steps, ; This represents the total number of phase shift steps; Indicates the first Step image pixel intensity; It is a cosine function; The relative phase values ​​of the pixels are shown below:

[0074] ;

[0075] in, It is the tangent function; It is a sine function.

[0076] Because the formula contains , , Since there are three unknowns, at least three phase shift diagrams are needed to solve the problem. This invention uses four-step and eight-step phase shift diagrams.

[0077] Step S3, based on the structured light phase shift method and the multi-frequency heterodyne method, obtain the initial absolute phase value and the modulation intensity.

[0078] Step S31, based on the relative phase obtained by the structured light phase shift method, four-step, four-step and eight-step phase shift calculation is performed on each group of frequency stripe images respectively to obtain the wrapped relative phase under each frequency. , wherein represents the frequency.

[0079] Step S32, using the multi-frequency heterodyne method, the initial absolute phase is obtained by recursive calculation as follows:

[0080] ;

[0081] wherein, is the initial absolute phase value; represents the rounding operation; , and represent the equivalent phase obtained by subtracting the relative phase under different frequencies; , and are the stripe periods corresponding to different frequencies.

[0082] Step S33, using the structured light phase shift method, the modulation intensity of different frequencies is obtained as follows:

[0083] ;

[0084] wherein, is the number of phase shifts; is the total number of phase shifts; represents the intensity of the pixel of the step image.

[0085] Step S4, by Gauss-Newton iterative method, the initial absolute phase and the modulation intensity parameters of the multi-frequency stripe are jointly optimized.

[0086] Step S41, extract the gray value and the initial absolute phase value of the current image pixel in different frequencies;

[0087] Step S42, construct the joint residual sum of squares as the optimization objective function as follows:

[0088] ;

[0089] wherein, is the initial absolute phase value to be optimized, and the phase values of different frequencies have been converted into the same standard phase value; , and are the modulation intensities at different frequencies, respectively; , and are the actually measured gray values at different wave numbers, respectively; , and are the phase shift amounts at different wave numbers, respectively; is the background light intensity.

[0090] wherein the residual error is defined as follows:

[0091] ;

[0092] ;

[0093] ;

[0094] wherein, , and are the single brightness residual errors under different wave numbers and different phase shift conditions, respectively.

[0095] Therefore, the total target function is expressed as:

[0096] ;

[0097] Step S43, partial derivatives of the parameters , , and are calculated, respectively, and each row of the Jacobian matrix corresponds to the partial derivative of each residual error with respect to each parameter, and the joint equation is constructed as follows:

[0098] ;

[0099] wherein, is the Jacobian matrix; is the parameter update amount, is the initial absolute phase update amount; , and are the modulation intensity update amounts at different frequencies, respectively; is the joint residual error vector, and each component is the difference between the model gray value and the actual gray value under different frequency and phase shift conditions; is the transpose.

[0100] Step S44, considering that ill-conditioned or irreversible problems may occur, the regularization parameter is introduced on the diagonal line Construct the augmented matrix as follows:

[0101] ;

[0102] in, It is the identity matrix. The range of values ​​is The parameters are adaptively adjusted based on the noise level of the experimental data. The constructed augmented matrix is ​​solved using Cholesky decomposition or SVD decomposition to obtain the parameter update values. .

[0103] Step S45: Use adjustment factors to gradually update the initial absolute phase and modulation intensity, and perform error control.

[0104] When the nth iteration is completed, update the parameter amount. The values ​​are compared with a preset standard to determine whether to output the absolute phase and modulation intensity after the current iteration.

[0105] First, set the initial absolute phase. The preset standard value is Modulation intensity , and The preset standard value is ;in .like , , , Stop the iteration and calculate the absolute phase value after the current iteration. .

[0106] Then, update the absolute phase value. With the initial absolute phase value absolute value of the difference , compared with the preset standard value If a comparison is made, Then output the updated absolute phase value. Otherwise, do not trust the optimization and output the initial absolute phase value. .

[0107] If the number of iterations reaches the preset maximum number of iterations At that time, parameter update amount If the preset criteria are not met, the iteration stops and the initial absolute phase value is output. .

[0108] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0109] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0110] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways to obtain the electronic program, and then storing it in the computer memory.

[0111] It is worth noting that the contents not elaborated in detail in the present application are all prior art and are well known to those skilled in the art.

[0112] Therefore, the application adopts the above-mentioned three-dimensional reconstruction method, device and medium based on multi-frequency fringe synchronous optimization, overcomes the defects that the traditional single frequency optimization is easily affected by noise and local distortion by jointly modeling and optimizing the absolute phase and modulation parameter of multiple frequencies, makes the optimization result more in line with the real optical imaging law, thereby significantly improving the reconstruction accuracy of the system; the regularization and step factor control are introduced in the iteration process, effectively avoiding the divergence and oscillation problems in the traditional nonlinear optimization, making the optimization process more stable and reliable, and ensuring the robustness under complex imaging conditions such as high reflection, low reflection and multiple exposure; by uniformly constraining the multi-frequency phase residual through Gauss-Newton iteration, the application can greatly improve the convergence speed while ensuring the calculation accuracy, is suitable for fast processing of large-scale point cloud data, and meets the dual requirements of efficiency and accuracy in application scenarios such as industrial detection and surface defect detection.

[0113] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization, characterized in that, Includes the following steps: Step S1: Based on a monocular structured light system, project multiple sets of sinusoidal fringe images of different frequencies onto the surface of the object under test using a projector, and perform multiple phase steps respectively, and use a camera to acquire the corresponding image sequences. Step S2: Use the structured light phase-shifting method to obtain the relative phase value of the pixel; Step S3: Based on the structured light phase shift method and multi-frequency heterodyne method, obtain the initial absolute phase value and modulation intensity; Step S4: The absolute phase and modulation intensity parameters of the multi-frequency fringes are jointly optimized using the Gauss-Newton iterative method. The specific process is as follows: Step S41: Extract the grayscale values ​​and initial absolute phase values ​​of the current image pixels at different frequencies; Step S42: Construct the joint sum of squared residuals as the optimization objective function. As shown below: ； in, The initial absolute phase value to be optimized; , and These represent the modulation intensity at different frequencies; , and These are the actual gray values ​​measured at different wavenumbers; , and These represent the phase shift at different wave numbers; Background light intensity; The residual is defined as follows: ； ； ； in, , and These represent individual brightness residuals under different wavenumbers and different phase shifts, respectively. Therefore, the overall objective function Represented as: ； Step S43: For each parameter... , , and Find the partial derivatives and the Jacobian matrix. Each row corresponds to a partial derivative of the residual with respect to each parameter. The joint equations are constructed as follows: ； in, Represents the Jacobian matrix; Indicates the parameter update amount. This represents the initial absolute phase update amount; , and These represent the update amounts of modulation intensity at different frequencies; This represents the joint residual vector, where each component is the difference between the model gray value and the actual gray value under different frequency and phase shift conditions. Indicates transpose; Step S44: Introduce regularization parameters on the diagonal. Construct the augmented matrix as follows: ； in, It is the identity matrix; The parameters are regularization parameters; the constructed augmented matrix is ​​solved using Cholesky decomposition or SVD decomposition to obtain the parameter update values. ; Step S45: Use adjustment factors to gradually update the initial absolute phase and modulation intensity, and perform error control; When the nth iteration is completed, update the parameter amount. The values ​​are compared with a preset standard to determine whether to output the absolute phase and modulation intensity after this iteration. First, set the initial absolute phase. The preset standard value is Modulation intensity , and The preset standard value is ;in ;like , , , Stop the iteration and calculate the absolute phase value after the current iteration. ; Then, update the absolute phase value. With the initial absolute phase value absolute value of the difference , compared with the preset standard value If a comparison is made, Then output the updated absolute phase value. Otherwise, do not trust the optimization and output the initial absolute phase value. ; If the number of iterations reaches the preset maximum number of iterations At that time, parameter update amount If the preset criteria are still not met, the iteration stops and the initial absolute phase value is output. .

2. The three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization according to claim 1, characterized in that, A monocular structured light system includes: a structured light projector, a monocular structured light camera, and a control device connecting the projector and the monocular camera; The structured light projector is used to project coded images; the monocular structured light camera is used to acquire coded images.

3. The three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization according to claim 1, characterized in that, In step S1, firstly, a structured light projector projects seventeen images: one pure white image, four four-step phase-shifted images with wavenumber N1, four four-step phase-shifted images with wavenumber N2, and eight eight-step phase-shifted images with wavenumber N3. Then, while the structured light projector projects the images, a monocular structured light camera performs image acquisition using hard triggering.

4. The three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization according to claim 1, characterized in that, In step S2, the relative phase value of the pixel is obtained using the structured light phase-shifting method. The specific process is as follows: The standard N-step phase shift model is shown below: ； in, Represents pixels in a two-dimensional image; Indicates background light intensity; Indicates modulation intensity; For the number of phase shift steps, ; This represents the total number of phase shift steps; Indicates the first Step image pixel intensity; It is a cosine function; The relative phase values ​​of the pixels are shown below: ； in, It is the tangent function; It is a sine function.

5. The three-dimensional reconstruction method based on multi-frequency fringe synchronous optimization according to claim 1, characterized in that, In step S3, the initial absolute phase value and modulation intensity are obtained based on the structured light phase shift method and the multi-frequency heterodyne method. The specific process is as follows: Step S31: Based on the relative phase obtained by the structured light phase-shifting method, perform four-step, four-step, and eight-step phase-shifting calculations on the fringe images of each frequency group to obtain the relative phase of the wrapping at each frequency. ,in Indicates frequency; Step S32: Using the multi-frequency heterodyne method, the initial absolute phase is obtained through recursive calculation, as shown below: ； in, This is the initial absolute phase value; This indicates the rounding operation; , and This represents the equivalent phase obtained by subtracting the relative phases at different frequencies. , and The stripe period corresponding to different frequencies; Step S33: Using the structured light phase-shifting method, the modulation intensity at different frequencies is obtained, as shown below: ； in, This represents the number of phase shift steps. This represents the total number of phase shift steps; Indicates the first Step image pixel intensity.

6. A computer device, characterized in that, It includes a memory and a processor, the memory being used to store instructions, and the processor being used to execute the instructions to implement a three-dimensional reconstruction method based on multi-frequency stripe synchronization optimization as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a three-dimensional reconstruction method based on multi-frequency stripe synchronous optimization as described in any one of claims 1 to 5.

Images