Asynchronous binocular camera synchronous frame obtaining and positioning method based on binocular vision and video frame insertion

Abstract

The invention relates to the field of video frame insertion and binocular vision, in particular to an asynchronous binocular camera synchronous frame obtaining and positioning method based on binocular vision and video frame insertion, which comprises the following steps: S1, calibrating a binocular camera, and obtaining rough delay time of the binocular camera through testing; s2, acquiring asynchronous left and right camera images under the same clock, taking the left camera image as a reference view, and selecting two continuous frames of right camera images after rough delay time; s3, performing frame insertion on the two right camera images by using a frame insertion algorithm to obtain a series of right camera images; s4, obtaining an image with the highest synchronization degree with the left camera image in the series of images according to an optimal frame selection method; and S5, measuring the three-dimensional coordinates of the target point for the obtained synchronous left and right images by using a binocular vision method. The method overcomes the problems of high hardware synchronization cost and complex system in the existing binocular camera synchronization technology.

Description

Technical Field

[0001] The present invention relates to the field of video frame insertion and binocular vision, and in particular to a method for acquiring and positioning synchronous frames of an asynchronous binocular camera based on binocular vision and video frame insertion. Background Art

[0002] Binocular vision is a computer vision technology based on the principle of binocular vision, and the video interpolation method is a computer vision algorithm. Currently, the existing binocular camera synchronization methods mostly send trigger signals to the left and right cameras at the same time through external hardware triggers to synchronize the exposure and acquisition of the two cameras. Although this method can ensure the synchronization of the left and right camera images, it has high requirements for the camera trigger interface (needs to support external trigger mode) and external hardware will increase the complexity of the system and increase the system cost. In addition, although methods such as timestamp correction, frame discarding and resampling are relatively simple and convenient, they may cause information loss and are not suitable for accurate depth calculation in dynamic scenes. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and solve the problems in the existing binocular camera synchronization technology, such as high hardware synchronization cost, complex system, and difficulty in meeting real-time requirements of software methods. A method for synchronous frame acquisition and positioning of asynchronous binocular cameras based on binocular vision and video interpolation is proposed.

[0004] The object of the present invention is achieved by the following measures: a method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion, comprising the following steps:

[0005] S1. Calibrate the binocular camera and obtain the rough delay time of the binocular camera through testing;

[0006] S2, collecting asynchronous left and right camera images under the same clock, taking the left camera image as the reference view, and selecting two consecutive frames of right camera images after a rough delay time;

[0007] S3, interpolating the two right camera images using an interpolation algorithm to obtain a series of right camera images;

[0008] S4, obtaining the image with the highest degree of synchronization with the left camera image in the series of images according to the best frame selection method;

[0009] S5. Using a binocular vision method to measure the three-dimensional coordinates of the target point on the obtained synchronous left and right images.

[0010] Preferably, in the step S1, the internal and external parameters of the binocular camera are first obtained by using the Zhang Zhengyou calibration method; and through testing, the target object information is captured at the same global time, wherein the left and right cameras have a problem of image asynchrony, that is, a time delay problem, and the rough delay time of the left and right cameras is obtained by artificial methods according to the timestamp of each frame.

[0011] Preferably, in the step S3, after obtaining the rough delay time of the left and right cameras, the binocular image is divided into left and right images, the left image is used as a reference view, and two consecutive right image frames after the rough delay time are used as input, and the two input right image frames are interpolated using the RIFE video interpolation algorithm.

[0012] Preferably, the RIFE video interpolation algorithm used is mainly composed of IFNet and FusionNet. Given two frames of input I 0 ,I 1 , the purpose of the video interpolation algorithm is to generate an intermediate frame from t→(0,1) The RIFE video interpolation algorithm directly estimates the intermediate flow F by inputting the input frame into IFNet. t→0 , the network structure of IFNet is as follows Figure 2 As shown in the figure, the resolution is gradually improved in a "coarse to fine" manner. Multiple IFBlocks are used in conjunction with upsampling and downsampling modules to first calculate the coarse optical flow at a low resolution, then calculate the fine optical flow at a high resolution and use the linear motion assumption to approximate F t→1 :

[0013]

[0014] Then by backwardwarping the input frame, we get two rough results:

[0015] Then use FusionNet to backwardwarp the input frame to get two rough results The adjustment and fusion methods are as follows:

[0016]

[0017] Where M is the soft fusion map used to fuse two warp frames, and D is the reconstruction residual term used to refine image details.

[0018] Preferably, in step S4, to obtain the depth information of the synchronized binocular image, the binocular image needs to be epipolar corrected according to the internal and external parameters of the binocular camera obtained in step S1, so that any point P(X, Y) in the camera field of view is at position P in the left and right camera images. 1 (X 1,Y 1 ),P 2 (X 2 ,Y 2 ) are on the same horizontal line, the left camera view as a reference and the series of right images obtained by the frame complement algorithm are combined into a binocular image one by one, and the two-dimensional coordinates of the same point in the left and right images are detected using the detection algorithm. The left image is P L (X L ,Y L ), the right image is P R1 (X R1 ,Y R1 ),P R2 (X R2 ,Y R2 )…P Rn (X Rn ,Y Rn ), which is the same as the left image point P L Y coordinate value L The closest frame is the best frame that matches the left image the most.

[0019] Preferably, in step S5, the left and right camera intrinsic parameters and the rotation and translation matrix obtained in step S1 and the matching left target point position coordinates P obtained in step S4 are used. L (X L ,Y L ) and the right camera target point position coordinates P R (X R ,Y R ), B is the baseline length of the binocular camera (the distance between the optical centers of the left and right cameras), f is the focal length of the left and right cameras (the focal lengths of the left and right cameras are equal), and L is the width of the image captured by the left and right cameras. The parallax of point P in the left and right cameras is defined as:

[0020] d=|X L -X R |

[0021] Two imaging points P L and P R The distance between them is:

[0022]

[0023] According to the principle of similar triangles, we can conclude that:

[0024]

[0025] Then the distance Z from point P to the projection center plane can be obtained as:

[0026]

[0027] Taking the optical center of the left camera as the coordinate origin, according to the principle of similar triangles, we can get:

[0028]

[0029] From this, the three-dimensional coordinates of any point P in the field of view of the binocular camera can be calculated:

[0030]

[0031] The present invention uses a binocular system composed of two left and right cameras, captures two images (left and right) of the same scene through the binocular camera, determines the two-dimensional coordinates of the same point in the left and right camera images through camera calibration, epipolar correction, and parallax matching, and then uses the parallax and camera parameters to calculate the three-dimensional information of the target point. A method of increasing the frame rate by generating new transition frames between original video frames. A deep neural network is an artificial neural network containing multiple layers of neurons. It extracts complex features from large-scale data by imitating the structure and function of the human brain. The features extracted by the deep learning algorithm are more robust than traditional image processing and machine learning methods, and can well meet the requirements of accuracy and real-time performance in the target extraction process. RIFE is a video interpolation algorithm combined with a deep neural network. The deep neural network predicts the motion information (optical flow) between the input frames and the pixel distribution of the intermediate frames, and generates transition frames. It mainly includes two modules, IFNet and FusionNet. IFNet is used to extract the motion information (optical flow) between the given two frames of input I 0 ,I 1 Directly estimate the intermediate flow F t→0 , and then use the linear motion assumption to approximate F t→1 , and then get two rough results by backwardwarping the input frame Then use FusionNet to backward warp the input frame to get two rough results The RIFE video interpolation method can generate high-quality intermediate frames with excellent computational efficiency.

[0032] Beneficial effects of the invention: The invention overcomes the problems of high hardware synchronization cost, complex system, and difficulty in meeting real-time requirements in the existing binocular camera synchronization technology. The invention uses video interpolation technology to obtain intermediate frames, combines the best frame selection method to obtain the rightmost camera image with the highest matching degree, and uses the image to perform three-dimensional measurement. The process is convenient and fast, and has high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion;

[0034] Figure 2 It is a schematic diagram of the experimental system of the synchronous frame acquisition and positioning method of the asynchronous binocular camera based on binocular vision and video interpolation;

[0035] Figure 3 The schematic diagram of the IFNet network structure of the RIFE video interpolation algorithm based on binocular vision and video interpolation is a synchronous frame acquisition and positioning method for asynchronous binocular cameras;

[0036] Figure 4 It is a schematic diagram of binocular vision epipolar correction of a method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video interpolation;

[0037] Figure 5 The schematic diagram of binocular ranging for the synchronous frame acquisition and positioning method of an asynchronous binocular camera based on binocular vision and video interpolation; DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1: Figure 1 As shown, a method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion includes the following steps:

[0040] S1. Calibrate the binocular camera and obtain the rough delay time of the binocular camera through testing;

[0041] First, a binocular camera is used to shoot a chessboard calibration plate of appropriate size. During this process, the calibration plate should occupy more than 1 / 3 of the screen, and the chessboard should be kept as complete and clear as possible. Then, the internal and external parameters of the binocular camera are obtained using Zhang Zhengyou's calibration method. Through testing, the target object information is captured at the same global time. There is a problem of asynchronous picture between the left and right cameras, that is, a time delay problem. The rough delay time of the left and right cameras is obtained manually according to the timestamp of each frame.

[0042] S2, collecting asynchronous left and right camera images under the same clock, taking the left camera image as the reference view, and selecting two consecutive frames of right camera images after a rough delay time;

[0043] S3, interpolating the two right camera images using an interpolation algorithm to obtain a series of right camera images;

[0044] After obtaining the rough delay time of the left and right cameras, the binocular image is divided into left and right images. The left image is used as the reference view, and the two consecutive right images after the rough delay time are used as input. The RIFE video interpolation algorithm is used to interpolate the two input right images.

[0045] The RIFE video interpolation algorithm used is mainly composed of IFNet and FusionNet. Given two frames of input I 0 ,I 1 , the purpose of the video interpolation algorithm is to generate an intermediate frame from t→(0,1) The RIFE video interpolation algorithm directly estimates the intermediate flow F by inputting the input frame into IFNet. t→0 , the network structure of IFNet is as follows Figure 3 As shown in the figure, the resolution is gradually improved in a "coarse to fine" manner. Multiple IFBlocks are used in conjunction with upsampling and downsampling modules to first calculate the coarse optical flow at a low resolution, then calculate the fine optical flow at a high resolution and use the linear motion assumption to approximate F t→1 :

[0046]

[0047] Then by backwardwarping the input frame, we get two rough results:

[0048] Then use FusionNet to backwardwarp the input frame to get two rough results The adjustment and fusion methods are as follows:

[0049]

[0050] Where M is the soft fusion map used to fuse two warp frames, and D is the reconstruction residual term used to refine image details.

[0051] S4, obtaining the image with the highest degree of synchronization with the left camera image in the series of images according to the best frame selection method;

[0052] To obtain the depth information of the synchronized binocular image, it is necessary to perform epipolar correction on the binocular image according to the internal and external parameters of the binocular camera obtained in step S1. The result of epipolar correction is as follows: Figure 4 As shown, any point P(X,Y) under the camera field of view is located at position P in the left and right camera images. 1 (X 1 ,Y 1 ),P 2 (X 2 ,Y 2) are on the same horizontal line. The left camera view as a reference and the series of right images obtained by the frame complement algorithm are combined into a binocular image one by one. The detection algorithm is used to detect the two-dimensional coordinates of the same point in the left and right images. The left image is P L (X L ,Y L ), the right image is P R1 (X R1 ,Y R1 ),P R2 (X R2 ,Y R2 )…P Rn (X Rn ,Y Rn ), which is the same as the left image point P L Y coordinate value L The closest frame is the best frame that matches the left image the most.

[0053] S5. Using a binocular vision method to measure the three-dimensional coordinates of the target point on the obtained synchronous left and right images.

[0054] like Figure 5 As shown, the left and right camera internal parameters and rotation and translation matrices obtained in step S1 and the matching left target point position coordinates P obtained in step S4 are used. L (X L ,Y L ) and the right camera target point position coordinates P R (X R ,Y R ), B is the baseline length of the binocular camera (the distance between the optical centers of the left and right cameras), f is the focal length of the left and right cameras (the focal lengths of the left and right cameras are equal), and L is the width of the image captured by the left and right cameras. The parallax of point P in the left and right cameras is defined as:

[0055] d=|X L -X R |

[0056] Two imaging points P L and P R The distance between them is:

[0057]

[0058] According to the principle of similar triangles, we can conclude that:

[0059]

[0060] Then the distance Z from point P to the projection center plane can be obtained as:

[0061]

[0062] Taking the optical center of the left camera as the coordinate origin, according to the principle of similar triangles, we can get:

[0063]

[0064] From this, the three-dimensional coordinates of any point P in the field of view of the binocular camera can be calculated:

[0065]

[0066] Specifically, in step S4, the process of the best frame selection method is as follows: first, the reference left image and the right image generated by the frame complementation algorithm are combined into a binocular image one by one; the composed binocular image is subjected to epipolar correction using the internal parameters of the binocular camera; the target points in the left and right images are matched; the difference between the horizontal pixel coordinates (pixel coordinate Y value) of the target point in the right image and the horizontal pixel coordinates (pixel coordinate Y value) of the target point in the left image is calculated; the right image with the smallest difference is the one that matches the left image most closely.

[0067] The experimental platform built during the actual measurement is as follows Figure 2 As shown in Figure 1, the standard binocular camera image is used as the verification object, the left view is used as the reference, and the center position of a standard white sphere in the binocular camera field of view is used as the target point. The two frames of the right view before and after the left view are manually selected for interpolation. The two-dimensional coordinates and three-dimensional coordinate table of the target points in the series of right images generated by the left view and the interpolation are detected (compared with the standard synchronous binocular camera), as shown in Table 1.

[0068] Table 1 Schematic diagram of experimental data (mm)

[0069] Image number 2D target point coordinates Calculated 3D coordinates left1 645.5,296.5 right1 527.5,296.5 -13.0313,102.1224,-1306.0697 Fill frame 1 511.5,330.5 -11.4754,-89.9287,1150.1211 Fill Frame 2 508.5,320.5 -11.2241,-87.9595,1124.9360 Fill frame 3 514.5,324.5 -11.7382,-91.9881,1176.4597 Frame 4 514.5,324.5 -12.1079,-94.8854,1213.5136 Fill frame 5 518.5,311.5 -12.1079,-94.8854,1213.5136 Supplement frame 6 517.5,318.5 -12.0133,-94.1441,1204.0330 Fill frame 7 521.5,308.5 -12.4008,-97.1810,1242.8728 Fill frame 8 523.5,306.5 -12.6041,-98.7741,1263.2478 Frame 9 522.5,311.5 -12.5016,-97.9711,1252.9775 Fill frame 10 523.5,309.5 -12.7083,-99.5905,1273.6878 Fill frame 11 524.5,299.5 -12.7083,-99.5905,1273.6878 Fill frame 12 527.5,297.5 -13.0313,-102.1224,1306.0697 Fill frame 13 528.5,294.5 -13.1427,-102.9953,1317.2327 Fill frame 14 535.5,290.5 -13.9791,-109.5495,1401.0566 Fill frame 15 537.5,288.5 -14.2380,-111.5782,1427.0021 Fill frame 16 539.5,284.5 -14.5066,-113.6835,1453.9267

[0070] Among them, left1 and right1 are strictly matched binocular images, and interpolated frames 1-16 are a series of images generated by the interpolated frame algorithm (among which interpolated frames 1 and 16 are two frames of images input to the RIFE algorithm). It can be seen that in the series of right images generated by the interpolated frame algorithm, the pixel coordinate values ​​of the target points are in a changing state, and there is a right view whose Y coordinate value is the same as the reference left view, and the calculated three-dimensional coordinates are also the same as the three-dimensional coordinates calculated by the synchronized left and right images. Therefore, it is believed that the method proposed in this paper can quickly and accurately obtain the synchronized image of the asynchronous binocular camera and calculate its three-dimensional coordinates.

[0071] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion, characterized in that: The following steps are involved: S1. Calibrate the binocular camera and obtain the rough delay time of the binocular camera through testing; S2, collecting asynchronous left and right camera images under the same clock, taking the left camera image as the reference view, and selecting two consecutive frames of right camera images after a rough delay time; S3, interpolating the two right camera images using an interpolation algorithm to obtain a series of right camera images; S4, obtaining the image with the highest degree of synchronization with the left camera image in the series of images according to the best frame selection method; S5. Using a binocular vision method to measure the three-dimensional coordinates of the target point on the obtained synchronous left and right images.

2. According to claim 1, a method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion, characterized in that: In the step S1, the internal and external parameters of the binocular camera are first obtained by using Zhang Zhengyou's calibration method; and through testing, the target object information is captured at the same global time. There is a problem of asynchronous images between the left and right cameras. The rough delay time of the left and right cameras is obtained by artificial methods based on the timestamp of each frame.

3. According to claim 1, a method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion, characterized in that: In the step S3, after obtaining the rough delay time of the left and right cameras, the binocular image is divided into left and right images, the left image is used as a reference view, and two consecutive right image frames after the rough delay time are used as input, and the RIFE video interpolation algorithm is used to interpolate the two input right image frames.

4. According to claim 3, a method for acquiring and positioning synchronous frames of an asynchronous binocular camera based on binocular vision and video frame insertion, characterized in that: The RIFE video interpolation algorithm used is mainly composed of IFNet and FusionNet. Given two input frames I0 and I1, the purpose of the video interpolation algorithm is to generate an intermediate frame t→(0,1) The RIFE video interpolation algorithm directly estimates the intermediate flow F by inputting the input frame into IFNet. t→0 IFNet uses a "coarse-to-fine" approach to gradually improve the resolution. It uses multiple IFBlocks, together with upsampling and downsampling modules, to first calculate the coarse optical flow at low resolution, then calculate the fine optical flow at high resolution and use the linear motion assumption to approximate F t→1 , Then, by backwardwarping the input frame, we get two rough results: Then use FusionNet to backwardwarp the input frame to get two rough results The adjustment and fusion methods are as follows: Where M is the soft fusion map used to fuse two warp frames, and D is the reconstruction residual term used to refine image details.

5. According to claim 2, a method for acquiring and positioning synchronous frames of an asynchronous binocular camera based on binocular vision and video frame insertion, characterized in that: In step S4, to obtain the depth information of the synchronized binocular image, it is necessary to perform epipolar correction on the binocular image according to the binocular camera internal and external parameters obtained in step S1, so that any point P(X,Y) in the camera field of view is on the same horizontal line in the positions P1(X1,Y1) and P2(X2,Y2) in the left and right camera images. The left camera view used as a reference and the series of right images obtained by the frame complement algorithm are combined into a binocular image one by one, and the detection algorithm is used to detect the two-dimensional coordinates of the same point in the left and right images. The left image is P L (X L ,Y L ), the right image is P R1 (X R1 ,Y R1 ),P R2 (X R2 ,Y R2 )…P Rn (X Rn ,Y Rn ), which is the same as the left image point P L Y coordinate value L The closest frame is the best frame that matches the left image the most.

6. The method for synchronous frame acquisition and positioning of an asynchronous binocular camera based on binocular vision and video frame insertion according to claim 5, characterized in that: In step S5, the left and right camera internal parameters and rotation and translation matrices obtained in step S1 and the matching left target point position coordinates P obtained in step S4 are used. L (X L ,Y L ) and the right camera target point position coordinates P R (X R ,Y R ), B is the baseline length of the binocular camera, f is the focal length of the left and right cameras, and L is the width of the image captured by the left and right cameras. The parallax of point P in the left and right cameras is defined as: d=|X L -X R | Two imaging points P L and P R The distance between them is: According to the principle of similar triangles, we can conclude that: Then the distance Z from point P to the projection center plane can be obtained as: Taking the optical center of the left camera as the coordinate origin, according to the principle of similar triangles, we can get: From this, the three-dimensional coordinates of any point P in the field of view of the binocular camera can be calculated:

Images