A multi-camera extrinsic parameter calibration method based on moire fringe and depth information

Abstract

The present application relates to a kind of multi-camera extrinsic calibration method based on moire fringe and depth information, it is characterized in that: the calibration method is applied to calibration system, the calibration system includes rack, equipment module and calibration target module, the module includes multi-view camera, camera clamping mechanism and camera adjusting mechanism, camera adjusting mechanism is installed at the bottom of rack, camera clamping mechanism is installed on the upper end of camera adjusting mechanism, multi-view camera is installed on camera clamping mechanism, the calibration target module includes several targets, target support mechanism, target adjusting mechanism, light source and light source adjusting mechanism, light source adjusting mechanism, light source and target adjusting mechanism are installed from top to bottom on the top of rack, target support mechanism is installed on the lower end of target adjusting mechanism, several targets are pasted in target support mechanism, the multi-view camera is opposite several targets.

Description

Technical Field

[0001] This invention belongs to the field of visual positioning technology, specifically relating to a multi-camera extrinsic calibration method based on moiré fringes and depth information. Background Technology

[0002] In moiré fringe correction, the relative positions of multiple cameras affect the conversion of parallax and depth information. In an ideal multi-view ranging system, multiple cameras are positioned at the same horizontal level, and the captured images are consistent vertically without deviation. Therefore, only the horizontal positional deviation needs to be calculated. However, in the actual installation of multi-view systems, human error and other factors can prevent two cameras from being perfectly aligned horizontally, resulting in two images on different horizontal planes and thus parallax in the vertical direction. Therefore, current camera calibration methods still face challenges in terms of both accuracy and efficiency. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a multi-camera extrinsic parameter calibration method based on moiré fringes and depth information, aiming to achieve automated multi-camera calibration and improve the accuracy of multi-camera calibration.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multi-camera extrinsic calibration method based on moiré fringes and depth information is disclosed. The calibration method is applied to a calibration system comprising a frame, an equipment module, and a calibration target module. Each module includes a multi-view camera, a camera clamping mechanism, and a camera adjustment mechanism. The camera adjustment mechanism is mounted at the bottom of the frame, and the camera clamping mechanism is mounted at the top of the camera adjustment mechanism. The multi-view camera is mounted on the camera clamping mechanism. The calibration target module includes several targets, a target support mechanism, a target adjustment mechanism, a light source, and a light source adjustment mechanism. The light source adjustment mechanism, the light source, and the target adjustment mechanism are mounted from top to bottom on the top of the frame. The target support mechanism is mounted at the bottom of the target adjustment mechanism, and several targets are attached to the target support mechanism. The multi-view camera faces the several targets.

[0005] Adjust the camera adjustment mechanism to position the multi-view camera at the designated test location, adjust the target adjustment mechanism to position the target at the designated test distance, adjust to the designated distance, set the light source parameters, and provide appropriate lighting adjustment.

[0006] First, determine the relative positions (center position, rotation angle, and tilt) of the first and second distance calibration patterns. Then, determine the relative positions (center position, rotation angle, and tilt) of the third distance calibration pattern. Next, place the first camera module on the platform and align the center of the multi-layer calibration pattern formed by several targets. Place the other cameras to be tested on the same platform. Finally, take pictures simultaneously with the first camera and other camera modules to test the extrinsic parameters of the other cameras relative to the first camera.

[0007] Among the calibration patterns obtained at three different distances, target one is a dot pattern, target two is a stripe pattern with outer dots and inner straight lines, and target three is a stripe pattern with straight lines.

[0008] Compared with the prior art, the advantages of this invention are: it realizes rapid calibration of the extrinsic parameters of multiple cameras relative to the main camera, while improving the accuracy of extrinsic parameter calibration; based on the moiré fringe measurement principle, it uses the Hanning window spectrum correction algorithm to perform frequency domain processing on the moiré fringe interference signal to suppress signal spectrum leakage, improve the measurement accuracy of moiré fringes when the width is small, and thus obtain accurate fringe density. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the calibration system of the present invention; Figure 2 This is a schematic diagram showing the relative positions of the multi-view camera and several targets of the present invention; Figure 3 The image is taken by the camera of this invention; Figure 4 These are images captured by the second camera of this invention; Figure 5 This is a schematic diagram illustrating the principle of moiré stripe formation in this invention. Figure 6 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0012] like Figure 1 The diagram illustrates a multi-camera extrinsic calibration method based on moiré fringes and depth information. This calibration method is applied to a calibration system comprising a frame, an equipment module, and a calibration target module. The module includes a multi-view camera, a camera clamping mechanism, and a camera adjustment mechanism. The camera adjustment mechanism is mounted at the bottom of the frame, and the camera clamping mechanism is mounted at the top of the camera adjustment mechanism. The multi-view camera is mounted on the camera clamping mechanism. The calibration target module includes several targets, a target support mechanism, a target adjustment mechanism, a light source, and a light source adjustment mechanism. The light source adjustment mechanism, the light source, and the target adjustment mechanism are mounted from top to bottom on the top of the frame. The target support mechanism is mounted at the bottom of the target adjustment mechanism, and several targets are attached to the target support mechanism. The multi-view camera faces the several targets.

[0013] Adjust the camera adjustment mechanism to position the multi-view camera at the designated test location, adjust the target adjustment mechanism to position the target at the designated test distance, adjust to the designated distance, set the light source parameters, and provide appropriate lighting adjustment.

[0014] like Figure 2 As shown: First, determine the relative positions of the first and second distance calibration patterns, namely the center position, rotation angle, and tilt angle. Then, determine the relative position of the third distance calibration pattern in sequence, also with the center position, rotation angle, and tilt angle. Next, place the first camera module on the platform and align the center of the multi-layer calibration pattern formed by several targets. Place the other cameras to be tested on the same platform. Finally, take pictures simultaneously with the first camera and other camera modules to test the extrinsic parameters of the other cameras relative to the first camera.

[0015] Among the three calibration patterns at different distances, target one is a dot pattern, target two is a stripe pattern with outer dots and inner straight lines, and target three is a stripe pattern with straight lines.

[0016] In the above environment, a photo taken by the camera is as follows: Figure 3 The image shown was captured by camera two. Figure 4 The image shown illustrates how the two linear targets are superimposed and the resulting moiré fringes are placed on the same coordinate system, as shown below. Figure 5 The diagram shown illustrates the principle of moiré fringe formation.

[0017] Assuming target H1 is parallel to the y-axis and has a target constant of d1, target H2 has an angle of θ with the y-axis and a target constant of d2, the moiré fringe has an angle of φ with the y-axis, and the width of the moiré fringe is ω, then we can obtain:

[0018]

[0019] When the target constants of the two target plates Then:

[0020]

[0021] Since moiré fringes are essentially the result of interference and diffraction, carrying information such as frequency and phase, FFT can be used to process moiré fringes and obtain information such as frequency; FFT is an efficient algorithm of DFT, called Fast Fourier Transform. Suppose that the moiré fringe is a signal sequence x(n) sampled at N points, and its Fourier transform g(n) is performed:

[0022] In the formula: f is the frequency; j is the imaginary part; N is the length of the discrete signal; n is the index of the signal sequence; Based on this, the signal is truncated using a Hanning window, and then the truncated signal is subjected to an FFT transform to obtain the signal's spectrum. The time-domain expression of the Hanning window can be expressed as:

[0023] In the formula: is the Hanning window; N is the width of the window function, i.e., the length of the truncated signal. The normalized spectral modulus function for the window length is:

[0024] In the formula: y(x) is the spectrum function.

[0025] The signal sequence x(n) acquired from moiré fringes is then subjected to FFT transformation after Hanning windowing:

[0026] The square of the main lobe mode function of the spectral spectrum of the Hanning window harmonic signal is:

[0027] In the formula: P(f) is the spectrum function; f0 is the frequency of the extracted time-domain signal; A is the amplitude of the extracted time-domain signal. Based on the energy centroid distribution characteristics of the Hanning window function, we can derive:

[0028] In the formula: Let n be the amplitude corresponding to the spectral line; n be the spectral line number; and k be the spectral line number where the amplitude is maximum. Then the center of the main lobe can be represented as:

[0029] In the formula This is the theoretical value of the power spectrum at the main lobe peak. In practical applications, the higher the value of n, the higher the correction accuracy, but a larger frequency interval between two adjacent spectral peaks is required. Therefore, the correction formula for frequency and phase angle can be expressed as:

[0030] In the formula: fc is the corrected frequency; fs is the signal sampling frequency; Ik is the imaginary part of the spectral value at the maximum; Rk is the real part of the spectral value at the maximum. By Passevar's theorem, the signal amplitude can be corrected:

[0031] In the formula, Kt is the energy recovery coefficient of the Hanning window.

[0032] With a constant sampling frequency, the width and tilt angle of the moiré fringes, as well as the sampling area information of the area array camera CMOS, are converted into the number of periods using the Fourier transform method of interference fringes. The number of periods is then corrected using HnWECM, with the center of the main lobe at f0 taken as the theoretical number of periods. Simultaneously, the tilt direction of the moiré fringes is determined by the corrected phase, thereby distinguishing whether the angle is obtuse or acute, i.e., obtaining the angle Tz of rotation along the optical axis. The moiré fringe displacement calculation is actually the moiré signal phase difference calculation, and the phase difference is obtained to obtain the displacement in the X and Y directions.

[0033] Based on the above algorithm, a dual-target moiré fringe angle and displacement measurement system can be constructed, and its principle flowchart is as follows. Figure 6 As shown.

[0034] Based on the dot images of target 1 and target 2, construct the relationship between each pair of cameras.

[0035] A world coordinate system is constructed using a target, and its transformation with the camera coordinate system relies on an extrinsic parameter matrix. Let the world coordinates of a point be... The coordinates of the corresponding point in the camera coordinate system are ,but

[0036] Where R is the rotation matrix and T is the translation matrix.

[0037] For pixel coordinates and camera coordinates, the transformation between them relies on an intrinsic parameter matrix. Let a point be in the pixel coordinate system... The coordinates of the corresponding point in the camera coordinate system are ,but

[0038] Where B is the intrinsic parameter matrix and s is the scaling factor.

[0039] For multi-camera positioning, we can obtain the positional relationship between the right and left cameras, and thus the relationship between the right camera coordinate system and the left camera coordinate system. Let the coordinates of a point in the left camera coordinate system be... The coordinates corresponding to the right camera coordinate system are ,but

[0040] Where R and T are the rotation and translation matrices of the right camera relative to the left camera obtained from multi-target calibration, respectively.

[0041] Substituting the Tz, X, and Y values ​​obtained from pairwise camera positions using moiré fringes into the extrinsic parameter matrix, the multi-camera calibration essentially involves solving for the positional relationships between the cameras. .

[0042] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera extrinsic parameter calibration method based on moiré fringes and depth information, characterized in that: The calibration method is applied to a calibration system, which includes a frame, an equipment module, and a calibration target module. The module includes a multi-view camera, a camera clamping mechanism, and a camera adjustment mechanism. The camera adjustment mechanism is installed at the bottom of the frame, the camera clamping mechanism is installed at the top of the camera adjustment mechanism, and the multi-view camera is installed on the camera clamping mechanism. The calibration target module includes several targets, a target support mechanism, a target adjustment mechanism, a light source, and a light source adjustment mechanism. The light source adjustment mechanism, the light source, and the target adjustment mechanism are installed from top to bottom on the top of the frame. The target support mechanism is installed at the bottom of the target adjustment mechanism, and several targets are attached to the target support mechanism. The multi-view camera faces the several targets. The multi-view camera includes a first camera, a second camera, and a third camera, and the multiple targets include target one with a first distance calibration pattern, target two with a second distance calibration pattern, and target three with a third distance calibration pattern; The calibration method includes the following steps: Step 1: Determine the relative positions of the first and second distance calibration patterns, and then determine the relative position of the third distance calibration pattern. The relative positions are the center position, rotation angle, and tilt angle. Step 2: Place the first camera module on the platform and align the center of the multi-layer calibration pattern formed by several targets. Place the second and third cameras on the same platform. Step 3: The first, second, and third cameras take pictures simultaneously, and the external parameters of the other cameras are tested relative to the first camera. Among the calibration patterns obtained at three different distances through steps one to three, target one is a dot pattern, target two is a striped pattern H1 with outer dots and inner straight lines, and target three is a straight striped pattern H2. Step 4: Superimpose the two linear targets H1 and H2 and establish them on the same coordinate system as the resulting moiré fringes; Through the above steps, assuming that target H1 is parallel to the y-axis and has a target constant of d1, target H2 has an angle of θ with the y-axis and a target constant of d2, the moiré fringe has an angle of φ with the y-axis, and the width of the moiré fringe is ω, then we can obtain: ； ； When the target constants of the two target plates Then: ； ； Since moiré fringes are essentially the result of interference and diffraction, carrying information such as frequency and phase, FFT can be used to process moiré fringes and obtain information such as frequency. Suppose that the moiré fringe is an N-point sampled signal sequence x(n), and its Fourier transform g(n) is performed: ； In the formula: f is the frequency; j is the imaginary part; N is the length of the discrete signal; n is the index of the signal sequence; Based on this, the signal is truncated using a Hanning window, and then the truncated signal is subjected to an FFT transform to obtain the signal's spectrum. The time-domain expression of the Hanning window can be expressed as: ； In the formula: For the Hanning window; N is the width of the window function, i.e., the length of the truncated signal; where the normalized spectral modulus function of the window length is: ； In the formula: y(x) is the spectrum function; The signal sequence x(n) acquired from moiré fringes is then subjected to FFT transformation after Hanning windowing: ； The square of the main lobe mode function of the spectral spectrum of the Hanning window harmonic signal is: ； In the formula: P(f) is the spectrum function; f0 is the frequency of the extracted time-domain signal; A is the amplitude of the extracted time-domain signal; Based on the energy centroid distribution characteristics of the Hanning window function, we can conclude that: ； In the formula: Let n be the amplitude corresponding to the spectral line; n be the spectral line number; k be the spectral line number where the amplitude is maximum; then the center of the main lobe can be represented as: In the formula This is the theoretical value of the power spectrum at the main lobe peak. In practical applications, the higher the value of n, the higher the correction accuracy, but a larger frequency interval between two adjacent spectral peaks is required. Therefore, the correction formula for frequency and phase angle can be expressed as: ； In the formula: fc is the corrected frequency; fs is the signal sampling frequency; Ik is the imaginary part of the spectral value at the maximum; Rk is the real part of the spectral value at the maximum; by Passevar's theorem, the signal amplitude can be corrected: In the formula: Kt is the energy recovery coefficient of the Hanning window; With a constant sampling frequency, the width and tilt angle of the moiré fringes, as well as the sampling area information of the area array camera CMOS, are converted into the number of periods using the Fourier transform method of interference fringes. The number of periods is then corrected using HnWECM, with the center of the main lobe at f0 taken as the theoretical number of periods. Simultaneously, the tilt direction of the moiré fringes is determined by the corrected phase, thereby distinguishing whether the angle is obtuse or acute, i.e., obtaining the angle Tz of rotation along the optical axis. The moiré fringe displacement calculation is actually the moiré signal phase difference calculation, and the phase difference is obtained to obtain the displacement in the X and Y directions.

2. The multi-camera extrinsic parameter calibration method based on moiré fringes and depth information as described in claim 1, characterized in that: It also includes a dual-target moiré fringe angle and displacement measurement system scheme; Based on the dot images of target one and target two, construct the relationship between each pair of cameras; A world coordinate system is constructed using a target, and its transformation with the camera coordinate system relies on an extrinsic parameter matrix. Let the world coordinates of a point be... The coordinates of the corresponding point in the camera coordinate system are ,but: ； Where R is the rotation matrix and T is the translation matrix, both of which are extrinsic parameter matrices; For pixel coordinates and camera coordinates, the transformation between them relies on an intrinsic parameter matrix. Let a point be in the pixel coordinate system... The coordinates of the corresponding point in the camera coordinate system are ,but: ； Where B is the intrinsic parameter matrix and s is the scaling factor; For multi-camera positioning, we can obtain the positional relationship between the right and left cameras, and thus the relationship between the right camera coordinate system and the left camera coordinate system. Let the coordinates of a point in the left camera coordinate system be... The coordinates corresponding to the right camera coordinate system are ,but: ； Where R and T are the rotation and translation matrices of the right camera relative to the left camera obtained by multi-target calibration, respectively. Substituting the Tz, X, and Y values ​​obtained from pairwise camera positions using moiré fringes into the extrinsic parameter matrix, the multi-camera calibration essentially involves solving for the positional relationships between the cameras. .

Images