Binocular camera splicing method based on movable splicing seams
Through motion detection and nonlinear mapping technology, the binocular camera splicing seam is dynamically adjusted, combined with multi-band hybrid fusion, the problem of transition traces of moving objects in binocular camera splicing is solved, achieving a more natural and uniform splicing effect.
Patent Information
- Application Number
- CN202310635254.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, binocular camera image stitching will show obvious transition marks when moving objects pass through the left and right image stitching seams, resulting in poor stitching effect.
Detect the position of moving objects through movement detection, and dynamically adjust the splicing seam positions of left and right images according to their position. Nonlinear mapping and multi-band mixing technology are used to fusion images to ensure the natural transition of moving objects during splicing.
It effectively eliminates the transition traces at the splicing seam, making the transition between the left and right cameras more natural and uniform, and improving the splicing effect.
Smart Images

Figure CN119090712A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to a binocular camera stitching method based on a movable stitching seam. Background Art
[0002] In the prior art, binocular camera image stitching is to stitch images taken by two cameras at different positions at the same time. Images with partial scene overlap are stitched into an image with a wide viewing angle. Usually, image stitching generally includes three parts: warp (mapping), compensation (light compensation) and blend (fusion). The warp part mainly depends on the accuracy of camera parameter estimation, the light compensation is mainly used to solve the illumination difference of different parts of the output image caused by different exposure of different images, and the blend is used to fuse the overlapping parts between different images, and generally a linear weighted method is used to obtain the final output image. The basic idea of multi-band blending is that the image can be decomposed into a superposition of images of different frequencies (similar to Fourier transform). At different frequencies, different weights should be used for fusion. In the low-frequency part, a weighted signal with a wider wavelength should be used (for example, the sigma in the Gaussian kernel function is relatively large), and in the high-frequency part, a narrower weighted signal should be used (for example, the sigma of the Gaussian kernel function is relatively small). The algorithm is as follows:
[0003] 1. Calculate the Gaussian pyramid of the input image. If the input image is A, B, then calculate G A0 ,G A1 ,G A2 ,G A3 ,… and G B0 ,G B1 ,G B2 ,G B3 ,……
[0004] The basic operations of the image pyramid are:
[0005] An image is continuously blurred and then downsampled to obtain images of different resolutions. At the same time, the width and height of each new image is 1 / 2 of the original image. The most common method is to sample after Gaussian blurring. The resulting series of images is called a Gaussian pyramid.
[0006] 2. Calculate the Laplacian pyramid of the input image. Note it as L A0 ,L A1 ,L A2 ,L A3 ,… and L B0 ,L B1 ,L B2 ,L B3 ,……
[0007] The Laplace pyramid is calculated as follows:
[0008] L(i)=G(i)–expand(G(i+1))
[0009] The i-th layer of the Laplacian pyramid is obtained by subtracting the i+1-th layer of the Gaussian pyramid expand from the i-th layer of the Gaussian pyramid.
[0010] 3. Fuse the Laplacian pyramids at the same level. For example, use simple linear fusion on both sides of the seam. Let the output image be C, then we get L C0 ,L C1 ,……
[0011] 4. Expand the high-level Laplacian pyramid in sequence until it has the same resolution as LC0. Recorded as L C00 ,L C11 ,L C22 …
[0012] 5. Superimpose the images obtained in 4 in sequence to obtain the final output image.
[0013] However, the defects in the prior art are that the existing image stitching methods will have obvious transition marks when a moving object passes through the stitching seam of the left and right images, and cannot transition naturally, resulting in poor stitching effect at the stitching seam.
[0014] In addition, commonly used technical terms in the prior art include:
[0015] 1. Binocular camera stitching: stitch and merge the photos taken by the left and right cameras placed at a certain angle into one photo.
[0016] 2. Camera calibration: In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations. The process of solving the parameters is called camera calibration (or camera calibration).
[0017] 3. memory copy: memory copy.
[0018] 4. Multi blend: The idea of multi-band fusion is to construct Laplacian pyramids for the fused images (Laplacian operator can extract high-frequency information of images. In the Laplacian pyramid, the higher the image layer, the higher the frequency). Then, the images in the same layer are fused according to certain rules, usually Alpha blending. Different rules are used to fuse images of different layers (images of different frequency bands), and the fused image pyramid is reconstructed to obtain the final result image.
[0019] 5. Motion detection: also generally called motion detection, which is to detect moving objects in the video and give specific coordinates. Summary of the invention
[0020] In order to solve the above problems, the purpose of this application is: This method uses motion detection to detect the position of the moving object, and dynamically adjusts the position of the stitching seam of the left and right images according to the position coordinates. This method solves the transition traces that appear when the moving object passes through the stitching seam of the left and right cameras, so that it transitions evenly and naturally from the left camera to the right camera or from the right camera to the left camera.
[0021] Specifically, the present invention provides a binocular camera stitching method based on a movable stitching seam, the method comprising the following steps:
[0022] S1, obtain real-time frames: left camera real-time frame, right camera real-time frame;
[0023] S2, after obtaining the result of step S1, read the config.json file that stores the coordinates of the four vertices of the chessboard;
[0024] S3, determine whether the vertex coordinates are obtained? If not, proceed to step S4; if yes, proceed to step S5;
[0025] S4, camera calibration, obtaining camera internal parameters and distortion parameters; chessboard detection, inferring the coordinates of the four vertices of the chessboard; returning to step S2;
[0026] S5, perspective transformation:
[0027] The coordinates of the four vertices of the chessboard detected in the left and right camera images are weighted averaged and scaled to generate new coordinates of the four vertices of the chessboard. The calculation formula is as follows:
[0028] new_chessboard_x=(left_x+right_x+img_width) / 2
[0029] new_chessboard_y=(left_y+right_y) / 2
[0030] Among them, new_chessboard_x and new_chessboard_y are the x and y coordinates of the vertices of the new chessboard, left_x and left_y are the x and y coordinates of the vertices of the chessboard captured by the left camera, right_x and right_y are the x and y coordinates of the vertices of the chessboard captured by the right camera, and img_width is the width of the pictures taken by the left and right cameras. A total of 3 perspective transformations are performed during the stitching process;
[0031] S6, nonlinear mapping, and continue with S7 and S8 respectively;
[0032] Since the image after the third perspective is not in proportion to the memory copy images on the left and right ends, and there are missing pixels on the upper and lower edges of the perspective transformation part, in order to solve this problem, nonlinear mapping is used to compress the image left and right and stretch the image up and down; the calculation formula is as follows:
[0033] x i ′=img_width+a1*tan(a2*(x i -width)*π)
[0034]
[0035] Among them, img_width and img_height are the width and height of the left and right camera images, x i 、x i ′ are the X-axis coordinates of the pixels before and after mapping, and y i ,y i ′ is the Y-axis coordinate of the pixel before and after mapping; width is the width of the image after mapping; a i is the corresponding nonlinear mapping coefficient;
[0036] S7, memory copy; proceed to step S10;
[0037] The memory copy step is to copy the left half of the left camera image and the right half of the right camera image to the left and right ends of the mosaic image respectively. This process avoids the use of perspective transformation;
[0038] S7.1, in the memory storing the left image, sequentially copy the memory data of the left half row by row, and then sequentially paste the copied data by row from the beginning of the corresponding row of the spliced image until all the copied data are pasted into the spliced image;
[0039] S7.2, in the memory storing the right image, copy the memory data of the right half row row by row and arrange the data in reverse order, and then paste the copied data row by row starting from the end of the corresponding row of the spliced image in reverse order until all the copied data are pasted into the spliced image;
[0040] S8, left camera nonlinear mapping scaling
[0041] After the nonlinear mapping of the left camera image, copy the Y data in its YUV image and scale its length and width to 1 / 2 of the original. This is to reduce the size of the motion detection input image and improve efficiency. The pixel value formula before and after scaling is as follows:
[0042] P′(x i ,y i )=P(2*x i ,2*y i )
[0043] Among them, P′ and P are the pixel values before and after scaling, respectively, and x i ,y i They are the X and Y axis coordinate values of the pixel after scaling respectively;
[0044] S9, Motion Detection
[0045] S9.1, input the scaled nonlinear mapping image of the left camera, detect the moving object therein, return the detection result, the detection result is a list, which stores the coordinates and width of the moving object, and execute step S9.2;
[0046] S9.2, judging whether there is a moving object according to the detection result returned by S9.1. If the list is empty, there is no moving object, and the motion detection process ends, and returns to the default stitching position (1 / 2 of the stitching image width); if the list is not empty, there is a moving object, and executes step S9.3;
[0047] S9.3, judging whether the moving object is within the movable stitching range according to the returned x-coordinate of the moving object position and the object width, the movable stitching range is the default stitching position ±a, where a is an integer and is adjustable. If it is outside the movable stitching range, the motion detection process is terminated and the default stitching position is returned; otherwise, executing step S9.4;
[0048] S9.4, taking the intersection of the area occupied by the moving object on the X axis and the movable stitching range, and then removing the intersection in the movable stitching range; taking the processed movable stitching range as the stitching position candidate area; to ensure the stitching effect, taking the median of the stitching position candidate area as the stitching position, and returning;
[0049] S10, multi-band blending, the specific process is as follows:
[0050] S10.1, along the height direction, take 24 pixels on the left and right sides of the stitching position returned by step S9.4 in the left and right mapping images respectively as the left and right images to be fused in multi blend;
[0051] S10.2, constructing a Laplacian pyramid of the left and right images to be fused, and then constructing a masked Gaussian pyramid as a weight coefficient for weighted fusion;
[0052] S10.3, multiply the Laplacian pyramids of the left and right images by the weight coefficients respectively, and sum them up to obtain the final fused image pyramid. The calculation formula is as follows:
[0053] p i,j,k =pl i,j,k *w i,j,k +pr i,j,k *(1-w i,j,k )
[0054] where p i,j,k ,pl i,j,k ,pr i,j,k are the pixel values of the fused image pyramid, the Laplacian pyramid on the left, and the Laplacian pyramid on the right, i.e., the jth row and the kth column. i,j,k is the weight value of the i-th layer, j-th row, and k-th column of the masked Gaussian pyramid;
[0055] S10.4, using the fused image pyramid, reconstructing the final fused image; and placing the fused image back to the splicing position in the splicing image;
[0056] S10.5, final output effect.
[0057] In step S4, the camera is calibrated:
[0058] The binocular cameras are calibrated separately to obtain the camera lens distortion parameters, that is, the coefficients a and b in the following formula are obtained;
[0059] y=a*arctan(b*x)
[0060] Where y is the distance from a certain pixel in the correction image to the center of the correction image, and x is the distance from the corresponding pixel in the distortion image to the center of the distortion image. For the same model of lens, it only needs to be calibrated once, which is the data preparation before image stitching;
[0061] The chessboard detection:
[0062] When the stereo camera is initialized, it is necessary to detect the coordinates of the corner points of the chessboard in the stitching image and save them in the configuration file. When the stereo camera is restarted later, it does not need to detect the chessboard again, but reads it directly from the configuration file.
[0063] The step S5 further comprises:
[0064] S5.1, the first and second times are to calculate the perspective transformation matrix from the checkerboard to the new checkerboard in the left and right camera images respectively, so that the right half of the left camera image and the left half of the right camera image are mapped to a temporary image. The specific perspective transformation formula is:
[0065]
[0066]
[0067] Among them, X i , Y i is the coordinate of the i-th vertex among the four vertices of the new chessboard; a ij are the jth element value of the ith row in the perspective transformation matrix of size 3×3, where a33 is a constant value 1 and the rest are values to be evaluated; x i ,y i is the coordinate of the i-th vertex among the 4 vertices of the left / right chessboard. Solve a based on the coordinates of the 4 vertices of the left / right chessboard and the 4 vertices of the new chessboard. ij Value, get the perspective transformation matrix of the left / right image mapped to the spliced image;
[0068] S5.2, the third perspective transformation is to transform the temporary puzzle in the previous step S5.1 to the mosaic image. This step is to ensure that the transition between the memory copy part image and the perspective transformed image is more natural. The specific calculation process is the same as the previous step S5.1, using the perspective transformation matrix.
[0069] Therefore, the advantages of the present application are: the method is simple and can well solve the problems in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0071] Figure 1 is a schematic diagram of the process of this method.
[0072] Figure 2 It is a schematic diagram of image capture by the left camera in this method.
[0073] Figure 3 It is a schematic diagram of image capture by the right camera in this method.
[0074] Figure 4 It is a schematic diagram of the projection transformation principle in this method.
[0075] FIG. 5( a ) is one of schematic diagrams of the movable seam effect of the present method.
[0076] FIG. 5( b ) is a second schematic diagram of the movable seam effect of the present method.
[0077] FIG. 5( c ) is a third schematic diagram of the movable seam effect of the present method. DETAILED DESCRIPTION
[0078] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0079] like Figure 1 As shown, the present application provides a binocular camera stitching method based on a movable stitching seam, comprising the following steps:
[0080] S1, obtain real-time frames: left camera real-time frame, right camera real-time frame;
[0081] S2, after obtaining the result of step S1, read the config.json file that stores the coordinates of the four vertices of the chessboard;
[0082] S3, determine whether the vertex coordinates are obtained? If not, proceed to step S4; if yes, proceed to step S5;
[0083] S4, camera calibration, obtaining camera internal parameters and distortion parameters; chessboard detection, inferring the coordinates of the four vertices of the chessboard; returning to step S2;
[0084] The camera calibration:
[0085] The binocular cameras are calibrated separately to obtain the camera lens distortion parameters, that is, the coefficients a and b in the following formula are obtained;
[0086] y=a*arctan(b*x)
[0087] Where y is the distance from a certain pixel in the correction image to the center of the correction image, and x is the distance from the corresponding pixel in the distortion image to the center of the distortion image. For the same model of lens, it only needs to be calibrated once, which is the data preparation before image stitching;
[0088] The chessboard detection:
[0089] When the stereo camera is initialized, it is necessary to detect the coordinates of the corner points of the chessboard in the stitching image and save them in the configuration file; when the stereo camera is restarted later, it does not need to detect the chessboard again, but reads it directly from the configuration file. Figure 2 and Figure 3 shown.
[0090] S5, perspective transformation:
[0091] like Figure 4 As shown, the coordinates of the four vertices of the chessboard in the detected left and right camera images are weighted averaged and scaled to generate new coordinates of the four vertices of the chessboard. The calculation formula is as follows:
[0092] new_chessboard_x=(left_x+right_x+img_width) / 2
[0093] new_chessboard_y=(left_y+right_y) / 2
[0094] Among them, new_chessboard_x and new_chessboard_y are the x and y coordinates of the vertices of the new chessboard, left_x and left_y are the x and y coordinates of the vertices of the chessboard captured by the left camera, right_x and right_y are the x and y coordinates of the vertices of the chessboard captured by the right camera, and img_width is the width of the pictures taken by the left and right cameras. A total of 3 perspective transformations are performed during the stitching process.
[0095] S5.1, the first and second times are to calculate the perspective transformation matrix from the checkerboard to the new checkerboard in the left and right camera images respectively, so that the right half of the left camera image and the left half of the right camera image are mapped to a temporary image. The specific perspective transformation formula is:
[0096]
[0097]
[0098] Among them, X i , Y i is the coordinate of the i-th vertex among the four vertices of the new chessboard; a ij are the jth element value of the ith row in the perspective transformation matrix of size 3×3, where a33 is a constant value 1 and the rest are values to be evaluated; x i ,y i is the coordinate of the i-th vertex among the 4 vertices of the left / right chessboard. Solve a based on the coordinates of the 4 vertices of the left / right chessboard and the 4 vertices of the new chessboard. ij Value, get the perspective transformation matrix of the left / right image mapped to the spliced image;
[0099] S5.2, the third perspective transformation is to transform the temporary puzzle in the previous step S5.1 to the mosaic image. This step is to ensure that the transition between the memory copy part image and the perspective transformed image is more natural. The specific calculation process is the same as the previous step S5.1, using the perspective transformation matrix.
[0100] S6, nonlinear mapping, and continue with S7 and S8 respectively;
[0101] Since the image after the third perspective is not in proportion to the memory copy images on the left and right ends, and there are missing pixels on the upper and lower edges of the perspective transformation part, in order to solve this problem, nonlinear mapping is used to compress the image left and right and stretch the image up and down; the calculation formula is as follows:
[0102] x i ′=img_width+a1*tan(a2*(x i -width)*π)
[0103]
[0104] Among them, img_width and img_height are the width and height of the left and right camera images, x i 、x i ′ are the X-axis coordinates of the pixels before and after mapping, and y i ,y i ′ is the Y-axis coordinate of the pixel before and after mapping; width is the width of the image after mapping; a i is the corresponding nonlinear mapping coefficient;
[0105] S7, memory copy; proceed to step S10;
[0106] This step is to copy the left half of the left camera image and the right half of the right camera image to the left and right ends of the mosaic image respectively. This process avoids the use of perspective transformation, greatly reduces the amount of calculation, and improves the efficiency by about 45%.
[0107] S7.1, in the memory storing the left image, sequentially copy the memory data of the left half row by row, and then sequentially paste the copied data by row from the beginning of the corresponding row of the spliced image until all the copied data are pasted into the spliced image;
[0108] S7.2, in the memory storing the right image, copy the memory data of the right half row row by row and arrange the data in reverse order, and then paste the copied data row by row starting from the end of the corresponding row of the spliced image in reverse order until all the copied data are pasted into the spliced image;
[0109] S8, left camera nonlinear mapping scaling
[0110] After the nonlinear mapping of the left camera image, copy the Y data in its YUV image and scale its length and width to 1 / 2 of the original. This is to reduce the size of the motion detection input image and improve efficiency. The pixel value formula before and after scaling is as follows:
[0111] P′(x i ,y i )=P(2*x i,2*y i )
[0112] Among them, P′ and P are the pixel values before and after scaling, respectively, and x i ,y i They are the X and Y axis coordinate values of the pixel after scaling respectively;
[0113] S9, Motion Detection
[0114] The purpose of motion detection is to detect the coordinates of moving objects in the image to avoid the stitching position being on the moving object. This solves the problem of unnatural stitching transition. The specific processing steps are as follows:
[0115] S9.1, input the scaled nonlinear mapping image of the left camera, detect the moving object therein, return the detection result, the detection result is a list, which stores the coordinates and width of the moving object, and execute step S9.2;
[0116] S9.2, judging whether there is a moving object according to the detection result returned by S9.1. If the list is empty, there is no moving object, and the motion detection process ends, and returns to the default stitching position (1 / 2 of the stitching image width); if the list is not empty, there is a moving object, and executes step S9.3;
[0117] S9.3, judging whether the moving object is within the movable stitching range according to the returned x-coordinate of the moving object position and the object width, the movable stitching range is the default stitching position ±a, where a is an integer and is adjustable. If it is outside the movable stitching range, the motion detection process is terminated and the default stitching position is returned; otherwise, executing step S9.4;
[0118] S9.4, taking the intersection of the area occupied by the moving object on the X axis and the movable stitching range, and then removing the intersection in the movable stitching range; taking the processed movable stitching range as the stitching position candidate area; to ensure the stitching effect, taking the median of the stitching position candidate area as the stitching position, and returning;
[0119] S10, multi-band blending, the specific process is as follows:
[0120] S10.1, along the height direction, take 24 pixels on the left and right sides of the stitching position returned by step S9.4 in the left and right mapping images respectively as the left and right images to be fused in multi blend;
[0121] S10.2, constructing a Laplacian pyramid of the left and right images to be fused, and then constructing a masked Gaussian pyramid as a weight coefficient for weighted fusion;
[0122] S10.3, multiply the Laplacian pyramids of the left and right images by the weight coefficients respectively, and sum them up to obtain the final fused image pyramid. The calculation formula is as follows:
[0123] p i,j,k =pl i,j,k *w i,j,k +pr i,j,k *(1-w i,j,k )
[0124] where p i,j,k ,pl i,j,k ,pr i,j,k are the pixel values of the fused image pyramid, the Laplacian pyramid on the left, and the Laplacian pyramid on the right, i.e., the jth row and the kth column. i,j,k is the weight value of the i-th layer, j-th row, and k-th column of the masked Gaussian pyramid;
[0125] S10.4, using the fused image pyramid, reconstructing the final fused image; and placing the fused image back to the splicing position in the splicing image;
[0126] S10.5, the final output effect is as follows Figure 5(a)-5(c) shown.
[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A binocular camera stitching method based on a movable stitching seam, characterized in that: The method comprises the following steps: S1, obtain real-time frames: left camera real-time frame, right camera real-time frame; S2, after obtaining the result of step S1, read the config.json file that stores the coordinates of the four vertices of the chessboard; S3, determine whether the vertex coordinates are obtained? If not, proceed to step S4; if yes, proceed to step S5; S4, camera calibration, obtaining camera internal parameters and distortion parameters; chessboard detection, inferring the coordinates of the four vertices of the chessboard; returning to step S2; S5, perspective transformation: The four vertices of the chessboard detected in the left and right camera images are weighted averaged and scaled to generate new coordinates of the four vertices of the chessboard. The formula is as follows: new_chessboard_x=(left_x+right_x+img_width) / 2 new_chessboard_y=(left_y+right_y) / 2 Among them, new_chessboard_x and new_chessboard_y are the x and y coordinates of the vertices of the new chessboard, left_x and left_y are the x and y coordinates of the vertices of the chessboard captured by the left camera, right_x and right_y are the x and y coordinates of the vertices of the chessboard captured by the right camera, and img_width is the width of the pictures taken by the left and right cameras. A total of 3 perspective transformations are performed during the stitching process; S6, nonlinear mapping, and continue with S7 and S8 respectively; Since the image after the third perspective is not in proportion to the memory copy images on the left and right ends, and there are missing pixels on the upper and lower edges of the perspective transformation part, in order to solve this problem, nonlinear mapping is used to compress the image left and right and stretch the image up and down; the formula is as follows: x i ′=img_width+a1*tan(a2*(x i -width)*π) Among them, img_width and img_height are the width and height of the left and right camera images, x i 、x i ′ are the X-axis coordinates of the pixels before and after mapping, and y i ,y i ′ is the Y-axis coordinate of the pixel before and after mapping; width is the width of the image after mapping; a i is the corresponding nonlinear mapping coefficient; S7, memory copy; proceed to step S10; The memory copy step is to copy the left half of the left camera image and the right half of the right camera image to the left and right ends of the mosaic image respectively. This process avoids the use of perspective transformation; S7.1, in the memory storing the left image, sequentially copy the memory data of the left half row by row, and then sequentially paste the copied data by row from the beginning of the corresponding row of the spliced image until all the copied data are pasted into the spliced image; S7.2, in the memory storing the right image, copy the memory data of the right half row row by row and arrange the data in reverse order, and then paste the copied data row by row starting from the end of the corresponding row of the spliced image in reverse order until all the copied data are pasted into the spliced image; S8, left camera nonlinear mapping scaling After the nonlinear mapping of the left camera image, copy the Y data in its YUV image and scale its length and width to 1 / 2 of the original. This is to reduce the size of the motion detection input image and improve efficiency. The pixel value formula before and after scaling is as follows: P′(x i ,y i )=P(2*x i ,2*y i ) Among them, P′ and P are the pixel values before and after scaling, respectively, and x i ,y i They are the X and Y axis coordinate values of the pixel after scaling respectively; S9, Motion Detection S9.1, input the scaled nonlinear mapping image of the left camera, detect the moving object therein, return the detection result, the detection result is a list, which stores the coordinates and width of the moving object, and execute step S9.2; S9.2, judging whether there is a moving object according to the detection result returned by S9.
1. If the list is empty, there is no moving object, and the motion detection process ends, and returns to the default stitching position, which is 1 / 2 of the stitching image width; if the list is not empty, there is a moving object, and executes step S9.3; S9.3, judging whether the moving object is within the movable stitching range according to the returned x-coordinate of the moving object position and the object width, the movable stitching range is the default stitching position ±a, where a is an adjustable integer. If it is outside the movable stitching range, the motion detection process is terminated and the default stitching position is returned; otherwise, executing step S9.4; S9.4, taking the intersection of the area occupied by the moving object on the X axis and the movable stitching range, and then removing the intersection in the movable stitching range; taking the processed movable stitching range as the stitching position candidate area; to ensure the stitching effect, taking the median of the stitching position candidate area as the stitching position, and returning; S10, multi-band blending, the specific process is as follows: S10.1, along the height direction, take 24 pixels on the left and right sides of the stitching position returned by step S9.4 in the left and right mapping images respectively as the left and right images to be fused in multi blend; S10.2, constructing a Laplacian pyramid of the left and right images to be fused, and then constructing a masked Gaussian pyramid as a weight coefficient for weighted fusion; S10.3, multiply the Laplacian pyramids of the left and right images by the weight coefficients respectively, and sum them up to obtain the final fused image pyramid. The calculation formula is as follows: p i,j,k =pl i,j,k *w i,j,k +pr i,j,k *(1-w i,j,k ) where p i,j,k ,pl i,j,k ,pr i,j,k are the pixel values of the fused image pyramid, the Laplacian pyramid on the left, and the Laplacian pyramid on the right, i.e., the jth row and the kth column. i,j,k is the weight value of the i-th layer, j-th row, and k-th column of the masked Gaussian pyramid; S10.4, using the fused image pyramid, reconstructing the final fused image; and placing the fused image back to the splicing position in the splicing image; S10.5, final output effect.
2. The binocular camera stitching method based on a movable stitching seam according to claim 1, characterized in that: In the step S4, The camera calibration: The binocular cameras are calibrated separately to obtain the camera lens distortion parameters, that is, the coefficients a and b in the following formula are obtained; y=a*arctan(b*x) Where y is the distance from a certain pixel in the correction image to the center of the correction image, and x is the distance from the corresponding pixel in the distortion image to the center of the distortion image. For the same model of lens, it only needs to be calibrated once, which is the data preparation before image stitching; The chessboard detection: When the stereo camera is initialized, it is necessary to detect the coordinates of the corner points of the chessboard in the stitching image and save them in the configuration file. When the stereo camera is restarted later, it does not need to detect the chessboard again, but reads it directly from the configuration file.
3. The binocular camera stitching method based on a movable stitching seam according to claim 1, characterized in that: The step S5 further comprises: S5.1, the first and second times are to calculate the perspective transformation matrix from the checkerboard to the new checkerboard in the left and right camera images respectively, so that the right half of the left camera image and the left half of the right camera image are mapped to a temporary image. The specific perspective transformation formula is: Among them, X i , Y i is the coordinate of the i-th vertex among the four vertices of the new chessboard; a ij are the jth element value of the ith row in the perspective transformation matrix of size 3×3, where a33 is a constant value 1 and the rest are values to be evaluated; x i ,y i is the coordinate of the i-th vertex among the 4 vertices of the left / right chessboard. Solve a based on the coordinates of the 4 vertices of the left / right chessboard and the 4 vertices of the new chessboard. ij Value, get the perspective transformation matrix of the left / right image mapped to the spliced image; S5.2, the third perspective transformation is to transform the temporary puzzle in the previous step S5.1 to the mosaic image. This step is to ensure that the transition between the memory copy part image and the perspective transformed image is more natural. The specific calculation process is the same as the previous step S5.1, using the perspective transformation matrix.
