Multi-camera external parameter calibration method based on moire fringe and depth information

Abstract

The invention relates to a multi-camera external parameter calibration method based on moire fringes and depth information, which is characterized in that the calibration method is applied to a calibration system, the calibration system comprises a rack, an equipment module and a calibration target module, the calibration target module comprises a multi-view camera, a camera clamping mechanism and a camera adjusting mechanism, and the multi-view camera clamping mechanism is connected with the equipment module. The camera adjusting mechanism is installed at the bottom of the rack, the camera clamping mechanism is installed at the upper end of the camera adjusting mechanism, the multi-view camera is installed on the camera clamping mechanism, and the calibration target module comprises a plurality of targets, a target supporting mechanism, a target adjusting mechanism, a light source and a light source adjusting mechanism. The light source adjusting mechanism, the light source and the target adjusting mechanism are installed on the top of the machine frame from top to bottom, the target supporting mechanism is installed at the lower end of the target adjusting mechanism, the multiple targets are pasted in the target supporting mechanism, and the multi-view camera right faces the multiple targets.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual positioning, and in particular relates to a multi-camera extrinsic parameter calibration method based on moiré fringes and depth information. Background Art

[0002] In moiré fringe correction, the relative position relationship of multi-cameras affects the conversion of parallax information and depth information. In an ideal multi-camera ranging system, the positions of multiple cameras are at the same horizontal position, and the images taken are consistent in the vertical direction without deviation. Therefore, only the position deviation in the horizontal direction needs to be calculated. However, in the actual installation process of the multi-camera system, there will be human errors, etc. The positions of the two cameras cannot be completely consistent, so that the two images will be in different horizontal planes, which will also produce parallax in the vertical direction. It can be seen that the current camera calibration method still has certain challenges in calibration accuracy and efficiency. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a multi-camera extrinsic parameter calibration method based on moiré fringes and depth information, the purpose of which is to realize multi-camera automatic calibration and improve the multi-camera calibration accuracy.

[0004] To achieve the above object, the technical solution adopted by the present invention is:

[0005] A multi-camera extrinsic parameter calibration method based on moiré fringes and depth information, the calibration method is applied to a calibration system, the calibration system comprises a frame, an equipment module and a calibration target module, the module comprises a multi-eye camera, a camera clamping mechanism and a camera adjustment mechanism, the camera adjustment mechanism is installed at the bottom of the frame, the camera clamping mechanism is installed at the upper end of the camera adjustment mechanism, the multi-eye camera is installed on the camera clamping mechanism, the calibration target module comprises several targets, a target support mechanism, a target adjustment mechanism, a light source and a light source adjustment mechanism, the light source adjustment mechanism, the light source and the target adjustment mechanism are installed on the top of the frame from top to bottom, the target support mechanism is installed at the lower end of the target adjustment mechanism, several targets are posted in the target support mechanism, and the multi-eye camera is facing the several targets.

[0006] Adjust the camera adjustment mechanism to place the multi-eye camera at the specified test position, adjust the target adjustment mechanism to place the target at the specified test distance, adjust to the specified distance, set the light source parameters, and provide appropriate lighting adjustment.

[0007] First, determine the relative positions (center position, rotation angle and tilt) of the first distance calibration pattern and the second distance calibration pattern, and then determine the relative positions (center position, rotation angle and tilt) of the third distance calibration pattern in turn. Then, place the first camera module on the platform, align the center of the multi-layer calibration pattern formed by several targets, and place other cameras to be tested on the same platform. Finally, take pictures with the first camera and other camera modules at the same time to test the external parameters of other cameras relative to the first camera.

[0008] Among the calibration patterns obtained at three different distances, target one is a dot pattern, target two is a stripe pattern with outer dots and inner straight lines, and target three is a straight line stripe pattern.

[0009] Compared with the prior art, the advantages of the present invention are: it realizes the rapid calibration of the external parameters of multiple cameras relative to the main camera, and at the same time improves the accuracy of the external parameter calibration; on the basis of the moiré fringe measurement principle, the Hanning window spectrum correction algorithm is used to perform frequency domain processing on the moiré fringe interference signal to suppress signal spectrum leakage, thereby improving the measurement accuracy of the moiré fringe when the width is relatively small, thereby obtaining accurate fringe density. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 It is a schematic diagram of the calibration system of the present invention;

[0012] Figure 2 Schematic diagram of the relative positions of the multi-eye camera and several targets of the present invention;

[0013] Figure 3 A picture taken by camera 1 of the present invention;

[0014] Figure 4 The picture is taken by the second camera of the present invention;

[0015] Figure 5 A schematic diagram showing the formation principle of moiré fringes of the present invention;

[0016] Figure 6 It is a principle flow chart of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0018] like Figure 1 As shown: a multi-camera extrinsic parameter calibration method based on moiré fringes and depth information, the calibration method is applied to a calibration system, the calibration system includes a frame, an equipment module and a calibration target module, the module includes a multi-eye camera, a camera clamping mechanism and a camera adjustment mechanism, the camera adjustment mechanism is installed at the bottom of the frame, the camera clamping mechanism is installed at the upper end of the camera adjustment mechanism, the multi-eye camera is installed on the camera clamping mechanism, the calibration target module includes several targets, a target support mechanism, a target adjustment mechanism, a light source and a light source adjustment mechanism, the light source adjustment mechanism, the light source and the target adjustment mechanism are installed on the top of the frame from top to bottom, the target support mechanism is installed at the lower end of the target adjustment mechanism, several targets are posted in the target support mechanism, and the multi-eye camera is facing the several targets.

[0019] Adjust the camera adjustment mechanism to place the multi-eye camera at the specified test position, adjust the target adjustment mechanism to place the target at the specified test distance, adjust to the specified distance, set the light source parameters, and provide appropriate lighting adjustment.

[0020] like Figure 2 As shown: first determine the relative positions of the first distance calibration pattern and the second distance calibration pattern, that is, the center position, rotation angle and tilt angle, then determine the relative positions of the third distance calibration pattern in turn, which are also the center position, rotation angle and tilt angle, then place the first camera module on the platform, align the center of the multi-layer calibration pattern formed by several targets, and place other cameras to be tested on the same platform. Finally, the first camera and other camera modules take pictures at the same time to test the external parameters of other cameras relative to the first camera.

[0021] Among the three calibration patterns at different distances, target one is a dot pattern, target two is a stripe pattern with dots on the outside and straight lines on the inside, and target three is a straight line stripe pattern.

[0022] In the above environment, a camera is used to shoot Figure 3 The picture shown is taken by camera 2. Figure 4 As shown in the picture, the superposition of two linear targets and the resulting moiré fringes are established on the same coordinate system, as Figure 5 The schematic diagram of the moiré fringe formation is shown.

[0023] Assuming that target H1 is parallel to the y-axis and the target constant is d1, the angle between target H2 and the y-axis is θ, and the target constant is d2, the angle between the moiré fringe and the y-axis is φ, and the width of the moiré fringe is ω, we can obtain:

[0024]

[0025] When the target constants d of the two target pieces 1 =d 2 =d, then:

[0026]

[0027] Since the essence of moiré fringes is the result of interference and diffraction, and carries information such as frequency and phase, FFT can be used to process moiré fringes to obtain information such as frequency. Among them, FFT is an efficient algorithm of DFT, called fast Fourier transform.

[0028] Assume that the moiré fringe is a signal sequence x(n) sampled at N points, and perform Fourier transform g(n) on it:

[0029] g(n)=F(x(n)),x(n)=e (j2πfn / N)

[0030] Where: f is the frequency; j is the imaginary part; N is the length of the discrete signal; n is the index of the signal sequence;

[0031] On this basis, the signal is truncated by adding a Hanning window, and then the truncated signal is transformed by FFT to obtain the signal spectrum. The time domain expression of the Hanning window can be expressed as:

[0032] w hn (n) = 0.5-0.5cos(2πn / N)

[0033] Where: w hn (n)(n=0,2,...,N-1) is the Hanning window; N is the width of the window function, that is, the length of the intercepted signal. The window length normalized spectrum modulus function is:

[0034]

[0035] Where: y(x) is the spectrum function.

[0036] The signal sequence x(n) collected by the moiré fringes is windowed by Hanning and then FFT transformed:

[0037] G (i) =F(x(n)×w hn )

[0038] The square of the main lobe mode function of the spectrum of the Hanning window harmonic signal is:

[0039]

[0040] Where: P(f) is the spectrum function; f0 is the frequency of the extracted time-domain signal; A is the amplitude of the extracted time-domain signal. According to the energy centroid distribution characteristic of the Hanning window function, it can be obtained that:

[0041]

[0042] Where: P k+i is the amplitude corresponding to the spectral line; n is the spectral line number; k is the spectral line number at the maximum amplitude. Then the center of the main lobe can be expressed as:

[0043]

[0044] Where is the theoretical value of the power spectrum at the peak of the main lobe. In practical applications, the higher the value of n, the higher the correction accuracy, but the larger the frequency interval between two adjacent spectral peaks is required. Then the correction formulas for frequency and phase angle can be expressed as:

[0045]

[0046] Where: fc is the corrected frequency; fs is the sampling frequency of the signal; N is the number of spectral points; Ik is the imaginary part of the spectral line value at the maximum; Rk is the real part of the spectral line value at the maximum. According to Parseval's theorem, the signal amplitude can be corrected:

[0047]

[0048] Where Kt is the Hanning window energy recovery coefficient.

[0049] With the sampling frequency constant, the width and inclination angle of the Moiré fringe and the information of the sampling area of the CMOS of the area array camera are converted into the number of cycles by using the Fourier transform method of the interference fringe. The number of cycles is corrected by using HnWECM, and the center of the main lobe of f0 is regarded as the theoretical number of cycles; at the same time, the inclination direction of the Moiré fringe is judged by the corrected phase, so as to distinguish whether the rotation angle is an obtuse angle or an acute angle, that is, the angle Tz rotating along the optical axis is obtained; the Moiré fringe displacement calculation is actually the Moiré signal phase difference calculation, and the phase difference is obtained to obtain the displacements in the X and Y directions.

[0050] Based on the above algorithm, a double-target Moiré fringe angle and displacement measurement system scheme can be constructed, and its principle flow chart is as Figure 6 shown.

[0051] According to the dot pictures of target 1 and target 2, the relationship between two cameras is constructed.

[0052] The world coordinate system is constructed by the target, and the world coordinate system is transformed with the camera coordinate system by the external parameter matrix. Suppose the world coordinate of a point is (x w ,y w ,z w ), the coordinates of the corresponding point in the camera coordinate system are (x c ,y c ,z c ),but

[0053]

[0054] Where R is the rotation matrix and T is the translation matrix.

[0055] For the pixel coordinate system and the camera coordinate system, they rely on the internal parameter matrix conversion. Let a point in the pixel coordinate system be (μ, v), and the coordinates of the corresponding point in the camera coordinate system be (x c ,y c ,z c ),but

[0056]

[0057] Where A is the internal parameter matrix and s is the scaling factor.

[0058] For multi-target positioning, we can obtain the positional relationship between the right camera and the left camera, and then obtain the relationship between the right camera coordinate system and the left camera coordinate system. Suppose the coordinates of a point in the left camera coordinate system are (x l ,y l ,z l ), the corresponding coordinate in the right camera coordinate system is (x r ,y r ,z r ),but

[0059]

[0060] Where R and T are the rotation matrix and translation matrix of the right camera relative to the left camera obtained by multi-target positioning.

[0061] Substitute Tz, X, and Y between the two cameras obtained by using moiré fringes into the external parameter matrix, then the multi-camera calibration is to solve the position relationship between the cameras:

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera extrinsic calibration method based on moiré fringes and depth information, characterized in that: The calibration method is applied to a calibration system, which includes a frame, an equipment module and a calibration target module, wherein the module includes a multi-eye camera, a camera clamping mechanism and a camera adjustment mechanism, wherein the camera adjustment mechanism is installed at the bottom of the frame, the camera clamping mechanism is installed at the upper end of the camera adjustment mechanism, and the multi-eye camera is installed on the camera clamping mechanism. The calibration target module includes several targets, a target support mechanism, a target adjustment mechanism, a light source and a light source adjustment mechanism, wherein the light source adjustment mechanism, the light source and the target adjustment mechanism are installed at the top of the frame from top to bottom, the target support mechanism is installed at the lower end of the target adjustment mechanism, and several targets are posted in the target support mechanism, and the multi-eye camera is facing the several targets. The multi-eye camera includes a first camera, a second camera and a third camera, and the plurality of targets include a target 1 of a first distance calibration pattern, a target 2 of a second distance calibration pattern and a target 3 of a third distance calibration pattern; The calibration method comprises the following steps: Step 1: determine the relative position of the first distance calibration pattern and the second distance calibration pattern, and then determine the relative position of the third distance calibration pattern, the relative position being the center position, the rotation angle and the tilt angle; Step 2: Place the first camera module on the platform, align the center of the multi-layer calibration pattern formed by several targets, and place the second camera and the third camera on the same platform; Step 3: The first camera, the second camera, and the third camera take pictures at the same time to test the external parameters of the other cameras relative to the first camera; Among the calibration patterns obtained at three different distances through steps 1 to 3, target 1 is a dot pattern, target 2 is a stripe pattern H1 with outer dots and inner straight lines, and target 3 is a straight stripe pattern H2; Step 4, superimpose the two linear targets H1 and H2 and establish them on the same coordinate system as the formed moiré fringes; Through the above steps, assuming that the target H1 is parallel to the y-axis and the target constant is d1, the angle between the target H2 and the y-axis is θ, and the target constant is d2, the angle between the moiré fringe and the y-axis is φ, and the width of the moiré fringe is ω, we can get: When the target constants of the two target pieces are d1=d2=d, then: Since the essence of moiré fringes is the result of interference and diffraction, and carries information such as frequency and phase, FFT can be used to process moiré fringes to obtain information such as frequency; Assume that the moiré fringe is a signal sequence x(n) sampled at N points, and perform Fourier transform g(n) on it: g(n)=F(x(n)),x(n)=e (j2πfn / N) ; Where: f is the frequency; j is the imaginary part; N is the length of the discrete signal; n is the index of the signal sequence; On this basis, the signal is truncated by adding a Hanning window, and then the truncated signal is transformed by FFT to obtain the signal spectrum. The time domain expression of the Hanning window can be expressed as: w hn (n)=0.5-0.5cos(2πn / N); Where: w hn (n)(n=0,2,...,N-1) is the Hanning window; N is the width of the window function, that is, the length of the intercepted signal; the window length normalized spectrum modulus function is: Where: y(x) is the spectrum function; The signal sequence x(n) collected by the moiré fringes is windowed by Hanning and then FFT transformed: G (i) =F(x(n)×w hn ); The square of the main lobe mode function of the spectrum of the Hanning window harmonic signal is: Where: P(f) is the spectrum function; f0 is the frequency of the intercepted time domain signal; A is the amplitude of the intercepted time domain signal; According to the energy center of gravity distribution characteristics of the Hanning window function, it can be concluded that: Where: P k+i is the amplitude corresponding to the spectral line; n is the spectral line number; k is the spectral line number with the maximum amplitude; the center of the main lobe can be expressed as: In the formula is the theoretical value of the power spectrum at the main lobe peak. In practical applications, the higher the value of n, the higher the correction accuracy. However, the larger the frequency interval between two adjacent spectrum peaks is required, the correction formula of frequency and phase angle can be expressed as: Where: fc is the corrected frequency; fs is the sampling frequency of the signal; N is the number of spectrum points; Ik is the imaginary part of the maximum spectrum line value; Rk is the real part of the maximum spectrum line value; According to Parseval's theorem, the signal amplitude can be corrected: Where: Kt is the Hanning window energy recovery coefficient; The sampling frequency is constant, and the interference fringe Fourier transform method is used to convert the width and inclination of the moiré fringes and the information of the sampling area of ​​the area array camera CMOS into the number of cycles. The number of cycles is corrected using HnWECM, and the main lobe center of f0 is regarded as the theoretical number of cycles. At the same time, the inclination direction of the moiré fringes is determined by the corrected phase, so as to distinguish whether the angle is an obtuse angle or an acute angle, that is, to obtain the angle Tz of rotation along the optical axis. The moiré fringe displacement solution is actually the moiré signal phase difference solution, and the phase difference is obtained to obtain the displacement in the X and Y directions.

2. A multi-camera extrinsic calibration method based on moiré fringes and depth information as claimed in claim 1, characterized in that: It also includes a dual-target moiré fringe angle and displacement measurement system solution; According to the dot images of target one and target two, the relationship between the two cameras is established; The world coordinate system is constructed by the target, and the world coordinate system is transformed with the camera coordinate system by the external parameter matrix. Suppose the world coordinate of a point is (x w ,y w ,z w ), the coordinates of the corresponding point in the camera coordinate system are (x c ,y c ,z c ),but: Wherein R is the rotation matrix, T is the translation matrix, and both the rotation matrix and the translation matrix are the external parameter matrices; For the pixel coordinate system and the camera coordinate system, they rely on the internal parameter matrix conversion. Let a point in the pixel coordinate system be (μ, v), and the coordinates of the corresponding point in the camera coordinate system be (x c ,y c ,z c ),but: Where A is the internal parameter matrix, s is the scaling factor; For multi-target positioning, we can obtain the positional relationship between the right camera and the left camera, and then obtain the relationship between the right camera coordinate system and the left camera coordinate system. Suppose the coordinates of a point in the left camera coordinate system are (x l ,y l ,z l ), the corresponding coordinate in the right camera coordinate system is (x r ,y r ,z r ),but: Where R and T are the rotation matrix and translation matrix of the right camera relative to the left camera obtained by multi-target positioning; Substitute Tz, X, and Y between the two cameras obtained by using moiré fringes into the external parameter matrix, then the multi-camera calibration is to solve the position relationship between the cameras:

Images