A method for stitching images of a parallel fisheye camera based on moving least squares transformation

Abstract

The present invention proposes a method for stitching large-disparity parallel fisheye camera images based on moving least squares transformation. This method uses multiple fisheye cameras arranged in parallel at a relatively large distance along a straight line to form a parallel fisheye stitching system, collects multiple fisheye images, and can sense a large range of environmental information, providing a large-range continuous wide-angle field of view for smart cities, intelligent security, and target detection and tracking systems. This method restores the geometric shape of the overlapping area through inverse latitude-longitude mapping, interactive image segmentation, and moving least squares transformation, realizing the restoration and construction of the overlapping area under the conditions of wide spacing and few common views. Through the grid deformation optimization method, an energy function is designed by combining line features and grid shape preservation terms to achieve fine alignment of the overlapping area, improving the accuracy and precision of image stitching. It is extended from double-camera double-image stitching to multi-camera joint processing and stitching, effectively improving the large-range sensing ability of the system.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to large-parallax parallel fisheye image stitching based on moving least squares transformation and fisheye image unfolding. Background Art

[0002] With the continuous development of technology and economy and the improvement of people's living standards, artificial intelligence technology has gradually been integrated into people's daily lives and the construction of modern cities. Complete large-scale intelligent systems such as smart cities involve multiple complex tasks such as perception, control, and planning. Among them, the perception link, as the main way for the intelligent system to obtain external information, is the basis for the subsequent modules such as recognition, planning, decision-making, and control to operate well. Visual perception is one of the most common perception modules in the smart city system. Traditional visual perception usually mainly relies on a large number of surveillance cameras, combined with a small number of wide-angle lenses, to separately collect and display image data. Nowadays, it is difficult to meet people's growing demand for enhanced perception capabilities. Especially in the fields of intelligent security, target detection, and tracking, the fisheye camera image stitching technology that provides a continuous large-range wide-angle field of view and enhances visual perception capabilities has gradually highlighted the urgent need for research.

[0003] Image stitching technology is a very classic and complete system in the field of computer vision. This technology stitches multiple overlapping images of the same scene into a larger image, which includes important algorithms such as feature point detection, mapping estimation, matching alignment, projective transformation, and image fusion, and can expand the field of view and enhance visual perception capabilities from a software level. However, the existing image stitching methods are still in the stage of traditional optical theory and perform poorly under complex conditions such as wide baseline, large parallax, and dynamic scenes.

[0004] Fisheye cameras are a common means to expand the field of view and enhance visual perception capabilities from a hardware level. Such cameras are cameras with extremely wide-angle lenses, with extremely short focal lengths and viewing angles close to or even exceeding 180°, having a large viewing angle range and a long depth of field. However, in order to project as large a scene as possible onto a limited image plane, the reasonable existence of barrel distortion has to be allowed. Therefore, there is a large difference between the fisheye lens imaging and the real-world scene in people's eyes. Except for the scenery in the center of the picture remaining basically unchanged, other scenery that should be horizontal or vertical has changed accordingly.

[0005] The large-parallax parallel fisheye camera image stitching technology arranges multiple fisheye cameras at a relatively long distance in the same direction, making the optical axes of each camera parallel to each other. It performs stitching processing on the multi-channel image data collected at the same moment to form a long-distance wide-angle view, thereby obtaining a wider field of view and having an intuitive visual effect. The large viewing angle and wide field of view characteristics of the fisheye camera enable the cameras to be spaced at a relatively long distance, thus covering as large an area as possible while reducing the number of cameras and lowering costs. The current fisheye image stitching technology mainly stitches the fisheye cameras arranged in a panoramic view to obtain a 360° panoramic view, which is difficult to apply to the scenario of wide-spacing parallel arrangement; the mainstream image stitching method requires feature extraction and matching of the overlapping area, while the overlapping area of the fisheye camera images arranged in a wide baseline and large parallax parallel is small, and the overlapping area is located at the edge of the image, resulting in a large distortion. How to restore the image structure of the overlapping area and achieve precise alignment is the difficulty of this stitching problem. Summary of the Invention

[0006] In view of this, the present invention provides a large-parallax parallel fisheye camera image stitching method based on moving least squares transformation. The images collected by multiple fisheye cameras arranged at a relatively long distance in the same direction are unfolded according to the longitude and latitude transformation method, and the moving least squares method (MLS) is used to deform the images to obtain relatively normal images after restoring the geometric shape. After predicting the overlapping area, extracting and matching features, and finely aligning the images by grid deformation, the optimal stitching seam is found and image fusion is performed to complete the stitching of multi-fisheye camera images.

[0007] A parallel fisheye camera image stitching method based on moving least squares transformation includes:

[0008] Step S1: Arrange N fisheye cameras C i in a straight line, with the optical axes of the cameras parallel to each other. Define the plane formed by the optical axes of adjacent cameras C i and C i+1 and the connection line between the cameras as plane S i ; The image of the normal field of view collected by each fisheye camera C i is denoted as the normal image I Ni , i = 1...N;

[0009] Step S2: Place two calibration marker boards between adjacent two fisheye cameras, with their planes parallel to plane S i and placed on both sides of plane S i respectively; The image containing the calibration marker board collected by each fisheye camera C i is denoted as the calibration image I Ci ; Multiple parallel dividing lines are set in the calibration marker board, and the dividing lines are placed vertically when setting the calibration marker board;

[0010] Step S3: Preprocess the corrected image I Ci and the normal image I Ni as follows:

[0011] Step S4: Remap the corrected image I Ci and the normal image I Ni to unfold the fisheye image. The unfolded corrected image and normal image are denoted as I′ Ci and I′ Ni respectively;

[0012] Step S5: Segment the unfolded corrected image I′ Ci to extract the regional image between two adjacent dividing lines in the corrected marker board image. In the unfolded corrected image I′ Ci , set the pixel values of the areas outside each regional image to zero to obtain the image I′ Cij ;

[0013] Step S6: Traverse each row of each image I′ Cij to find the first point with a non - zero pixel value from left to right, and save the coordinates of these points into the matrix L ij ; Calculate the slopes k ij between each point in the matrix L up and its two adjacent points before and after, and k down . If the absolute value of k up or k down is greater than the threshold k threshold , then this point can be considered as a vertically arranged point and be retained, otherwise it is excluded; For the filtered matrix L ij , the point closest to the horizontal mid - line of the image, denote its abscissa as x0; Uniformly select points from L ij and append them to the matrix p i to provide the control point coordinates before deformation for the moving least - squares transformation; Then set the abscissas of these points to x0 and keep the ordinates unchanged, and append them to the matrix q i to provide the control point coordinates after deformation for the moving least - squares transformation;

[0014] Traverse each row of each image I′ Cij to find the first point with a non - zero pixel value from right to left, and save the coordinates of these points into the matrix R ij ; Calculate the slopes k ij between each point in the matrix R up and its two adjacent points before and after, and k down . If the absolute value of k up or k down is greater than the threshold k threshold, then it can be considered that this point is a vertically arranged point and it is retained; otherwise, it is removed; for the filtered matrix R ij , the point closest to the horizontal center line of the image, record its abscissa as x0; from R ij , points are evenly selected and appended to the matrix p i . Then, set the abscissas of these points to x0 and keep the ordinates unchanged, and append them to the matrix q i ;

[0015] Traverse each segmented image, and append the coordinates of the control points before deformation provided by the moving least squares transformation and the coordinates of the control points after deformation provided by the moving least squares transformation to the matrix p i and the matrix q i respectively;

[0016] Step S7: Calculate the moving least squares transformation matrix A i corresponding to each camera C i according to the matrix p i and the matrix q i . According to the calculated A i , perform a moving least squares transformation on the unfolded normal image I′ Ni to restore its geometric shape in the overlapping area, and the deformed image is denoted as I″ Ni ;

[0017] Step S8: For two adjacent deformed images obtained in Step S7, first perform binarization processing on each of them. Divide each binarized image into multiple groups by columns, set the number of groups to n groups, count the number of black pixels in each group of each image, and save them to the arrays B i and B i+1 respectively; Place a sliding window with a width of m on B i and B i+1 , where m < n; Calculate the variance of the number of black pixels in the two sliding windows. When the variance is the smallest, the intervals where the two windows are located are the overlapping areas of the two images, and determine the position and width of the overlapping area as the preliminary alignment of the two deformed images;

[0018] Step S9: For two adjacent deformed images preliminarily aligned in Step S8, perform feature matching within the determined overlapping area to achieve precise alignment of the overlapping area;

[0019] Step S10: For all pairwise adjacent images processed in Step S9, determine the optimal stitching seam, select the corresponding image parts on both sides of the optimal stitching seam respectively, and use a multi-band fusion algorithm on both sides of the seam line to obtain the final stitched and fused image of all N cameras.

[0020] Preferably, in the said Step S2, the calibration marker plate plane and the plane S iThe distance between them is approximately half of the camera interval.

[0021] Preferably, in the step S2, several rectangular figures with strong color contrast are set in the marking board, and the area between two figures is the dividing line.

[0022] Preferably, the preprocessing method in the step S3 includes: using the line scanning method to extract the circular effective imaging area in the image, rotating it so that the scene located on the connection line of two adjacent cameras is on the horizontal center line of the image, and finally obtaining a preprocessed fisheye image with equal length and width after exposure compensation.

[0023] Preferably, in the step S4, the longitude and latitude transformation method using inverse interpolation is used to calculate the image remapping matrix, and the image is remapped to unfold the fisheye image.

[0024] Preferably, in the step S5, the GrabCut algorithm is used to perform interactive image segmentation on the corrected image expansion diagram I′ Ci for interactive image segmentation.

[0025] Preferably, in the step S6, the threshold k threshold takes the value of 1.

[0026] Preferably, the specific method of the step S9 is: using the SIFT algorithm to extract features in the determined overlapping area, using the Euclidean distance for matching, and then using the RANSAC algorithm to screen the matching points to obtain better matching; using the LSD algorithm to detect line features and perform matching; dividing the image into grids, combining the feature matching reprojection error, line feature matching alignment and grid structure preservation to design an energy function, and minimizing this energy function to perform grid deformation to achieve precise alignment of the overlapping area.

[0027] Preferably, the energy function of the grid deformation in the step S9 is:

[0028] E = λ lp E lp + λ pa E pa + λ la E la + λ g E g

[0029] Wherein, E lp represents the line retention term; E pa and E la are respectively the point alignment term and the line alignment term; E g represents the grid shape retention term, and λ lp 、λ pa 、λ la and λ g are respectively the weights of the corresponding terms.

[0030] Preferably, in step S10, for all pairwise adjacent camera images, the graph cut method is used to optimize the global energy function on the Markov random field to obtain a stitching seam that minimizes the loss under the given loss definition. Image parts on the corresponding sides are selected on both sides of the optimal stitching seam, and a multi-band fusion algorithm is adopted on both sides of the seam line to obtain the final stitched and fused image of all N cameras. The present invention has the following beneficial effects:

[0031] The present invention proposes a large-parallax parallel fisheye camera image stitching method based on moving least squares transformation. This method uses multiple fisheye cameras arranged in parallel at a relatively large distance along a straight line to form a parallel fisheye stitching system, collects multiple paths of fisheye images, can perceive a large range of environmental information, and can provide a large-range continuous wide-angle field of view for smart cities, intelligent security, and target detection and tracking systems. By reverse longitude and latitude mapping, interactive image segmentation, and moving least squares transformation, the geometric shape of the overlapping area is restored, and the restoration and construction of the overlapping area in the case of wide spacing and few common views are realized; through the grid deformation optimization method, an energy function is designed by combining line features and grid shape preservation terms to achieve fine alignment of the overlapping area, improving the accuracy and precision of image stitching; it is extended from double-camera double-image stitching to multi-camera joint processing and stitching, effectively improving the large-range perception ability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the hardware structure of the parallel fisheye stitching system in the stitching method of the present invention;

[0033] Figure 2 It is a schematic diagram of the placement layout of the calibration marker board in the stitching method of the present invention;

[0034] Figure 3 It is a flowchart of the large-parallax parallel fisheye camera image stitching algorithm of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] The following combines the drawings and gives examples to describe the present invention in detail. Those skilled in the art can understand the advantages and functions of the present invention according to the content described in this specification. The present invention can also be implemented and applied in other different ways.

[0036] The present invention relates to a large-parallax parallel fisheye camera image stitching method based on moving least squares transformation, including the hardware structure and software algorithm of the parallel fisheye stitching system. The hardware of the parallel fisheye stitching system consists of multiple fisheye cameras arranged in the same direction at a relatively large distance. The algorithm flow of the large-parallax parallel fisheye camera image stitching method is as follows:

[0037] Step S1, install and fix the parallel fisheye stitching system, and install N fisheye cameras C i, (i=1…N) are arranged in a straight line, the optical axes of the cameras are parallel to each other, and the spacing is wide, but it should be ensured that there is an overlapping area in the field of view of adjacent cameras. Define adjacent cameras C i and C i+1 The plane formed by the line between the optical axis and the camera is plane S i . Start the fisheye camera to collect image data. Each fisheye camera C i The collected image of normal field of view is recorded as normal image I Ni ,(i=1…N);

[0038] The hardware structure of the parallel fisheye stitching system is composed of a number of identical fisheye cameras, which have the same image acquisition performance. The overall perception range can cover the surrounding area of ​​the system carrier in the direction of the camera connection, realizing environmental information perception without blind spots in the direction of the camera connection. The field of view of the fisheye camera used in the parallel fisheye stitching system is close to or even exceeds 180°, and the cameras are arranged at a long distance to ensure a certain overlapping area, which reduces the number of cameras installed and has a larger perception range, thereby reducing system costs.

[0039] Step S2: Place two calibration marking plates between two adjacent fisheye cameras, with their planes aligned with plane S i Parallel and placed on plane S i On both sides, calibrate the marking plate plane and plane S i The distance between the two adjacent fisheye cameras is about half of the camera spacing. A calibration mark plate is placed between each two adjacent fisheye cameras according to this layout; the fisheye camera is started again to collect image data. Each fisheye camera C i The collected image containing the calibration mark plate is recorded as the calibration image I Ci ,(i=1…N);

[0040] Among them, several obvious parallel dividing lines are set in the correction mark plate, and the dividing lines are placed vertically when setting the correction mark plate; in order to better identify the dividing lines, several rectangular figures with strong color contrast are set in the mark plate, and the area between the two figures is the dividing line.

[0041] The calibration mark plate used in the parallel fisheye camera image stitching method is used for auxiliary calibration. It only needs to be of appropriate size, bright color, and can be detected and identified. Compared with commonly used calibration plates such as checkerboard calibration plates, it does not need to ensure specific size and accuracy, is easy to make, and can also be replaced by environmental elements with multiple vertical lines such as door frames.

[0042] Step S3: Corrected image I collected by the parallel fisheye stitching system Ci and normal image I NiPreprocessing: Use the line scanning method to extract the circular effective imaging area in the image, rotate it appropriately so that the scene located on the line connecting two adjacent cameras is on the horizontal center line of the image, and finally obtain the preprocessed fisheye image with equal length and width after exposure compensation;

[0043] Among them, the parallel fisheye camera image stitching method uses the line scanning method and the exposure compensation algorithm to preprocess the collected fisheye images. By using two pairs of horizontal and vertical straight lines on the grayscale image, squeezing from four directions of up, down, left, and right respectively, the circular effective area is extracted, thereby automatically cropping the possible black edges in the original image; then the exposure compensation algorithm is used to make each image have the same exposure degree, making the stitching effect of the overlapping area more natural.

[0044] Step S4: According to the size of the preprocessed fisheye image, calculate the image remapping matrix using the longitude and latitude transformation method of inverse interpolation and save it. Remap the corrected image I Ci and the normal image I Ni to expand the fisheye image. The expanded corrected image and normal image are respectively denoted as I' Ci and I' Ni ;

[0045] Among them, the parallel fisheye camera image stitching method uses the longitude and latitude transformation method to expand the fisheye image. Its forward mapping rule is: project the fisheye image onto the surface of the unit sphere, and then project it onto the circumscribed cylinder of the unit sphere. Unroll the cylinder to obtain the target image. This method only needs to obtain the size of the fisheye image to expand the fisheye image according to the geometric relationship, without pixel loss and having a natural visual effect in the expansion direction. It can retain the circumferential area of the fisheye image and is suitable for expanding the fisheye image when the camera interval is far and the overlapping area of the field of view is small. The longitude and latitude transformation method of inverse interpolation is a method that starts from the target image, for each coordinate point on the target image, calculates the coordinates of its corresponding point on the original fisheye image in reverse, and then calculates the pixel value of the target point using the bilinear interpolation method. Since the points on the corrected cylindrical image and the points on the original fisheye image are not a surjective relationship, using inverse mapping correction can fill the image gaps caused by forward mapping correction.

[0046] Step S5: Use the GrabCut algorithm to perform interactive image segmentation on the expanded image I' Ci of the corrected image to extract the regional images between two adjacent dividing lines in the corrected marker board image. The user manually frames each area of the corrected marker board image for preliminary graph cutting, and performs a small amount of point-line annotation on the preliminary graph cutting result for iterative segmentation. Suppose each corrected image can extract M regional images, denoted as I' Cij , (i = 1…N, j = 1…M); Among them, I' CijIn the corrected image expansion diagram I′ Ci an image obtained by setting the pixel values of the areas outside each regional image to zero;

[0047] Among them, the parallel fisheye camera image stitching method uses the GrabCut algorithm for interactive image segmentation. This method is improved based on graph-cut and uses the Gaussian mixture model (GMM) as statistical prior information. By inputting a bounding box by the user as the segmentation target position, the segmentation of the target and the background is achieved. The GarbCut algorithm has a fast segmentation speed, supports iterative repair and simple and fast user interaction, and can conveniently segment the calibration marker board image from a complex background.

[0048] Step S6: Traverse each row of each image I′ Cij at a certain interval step size, find the first point with a non-zero pixel value from left to right, and save the coordinates of these points into the matrix L ij Calculate the slopes k ij between each point in the matrix L up and its two adjacent points before and after, if the absolute value of k down or k up is greater than the threshold k down (with a value of 1), then this point can be considered a vertically arranged point and is retained, otherwise it is excluded. For the filtered matrix L threshold , the point closest to the horizontal midline of the image, record its abscissa as x0; uniformly take points from L ij at an interval of dy, and append them to the matrix p ij to provide the control point coordinates before deformation for the moving least squares transformation; then set the abscissas of these points to x0 and keep the ordinates unchanged, and append them to the matrix q i to provide the control point coordinates after deformation for the moving least squares transformation. i Similarly, traverse each row of each image I′

[0049] at a certain interval step size, find the first point with a non-zero pixel value from right to left in the same way, and save the coordinates of these points into the matrix R Cij Calculate the slopes k ij between each point in the matrix R ij and its two adjacent points before and after, if the absolute value of k up or k down is greater than the threshold k up or k down is greater than the threshold k threshold (with a value of 1), then this point can be considered a vertically arranged point and is retained, otherwise it is excluded. For the filtered matrix R ij , the point closest to the horizontal midline of the image, record its abscissa as x0; uniformly take points from R ijRandomly sample points from it and append them to matrix p i Then, set the abscissa of these points as x0 and keep the ordinate unchanged, and append them to matrix q i in it.

[0050] Traverse each segmented image, and append the coordinates of the control points before deformation provided by the moving least squares transformation and the coordinates of the control points after deformation provided by the moving least squares transformation to matrix p i and matrix q i in it.

[0051] The above parallel fisheye camera image stitching method automatically obtains the coordinates p of the control points before the moving least squares transformation by detecting the coordinates of non-zero pixel points, calculating the slopes of adjacent non-zero pixel points, and extracting the coordinates of the left and right edge points of the calibration marker board image at a given interval, and automatically calculates the coordinates q of the control points after deformation according to the given rules, avoiding a large amount of tedious work of manually reading and filling in the coordinates of the control points before and after deformation.

[0052] Step S7: Calculate the moving least squares transformation matrix A i corresponding to each camera C i and matrix q i and save it. According to the calculated A i perform a moving least squares transformation on the normal image expansion I′ i to restore its geometric shape in the overlapping area, and the deformed image is denoted as I″ Ni ; Ni ;

[0053] Step S8: For two adjacent images I″ Ni and I″ N(i+1) : Perform binarization processing through the adaptive threshold and median filtering methods respectively. Divide each binarized image into n groups by columns, count the number of black pixels in each group of each image, and save them to arrays B i and B i+1 in it. Adopt a sliding window search strategy. Place a sliding window with a width of m (m < n) on B i and B i+1 respectively, calculate the variance of the number of black pixels in the two sliding windows. When the variance is the smallest, the intervals where the two windows are located are the overlapping areas of the two images, and determine the position and width of the overlapping area as the preliminary alignment of the two images;

[0054] The overlapping area prediction algorithm is an algorithm that uses a binary image to count the number of black pixels column by column and calculates the variance to achieve preliminary alignment. Since the camera spacing is relatively far and the overlapping area is small, directly performing feature extraction and matching on two complete images has a large computational amount and low efficiency, and is prone to incorrect matching in the case of low texture or repetitive texture. Using the overlapping area prediction algorithm, feature extraction and matching are only performed in the overlapping area, which can effectively reduce the computational amount and the probability of incorrect matching, improve the computational efficiency, and determine the width and the upper left coordinate of the overlapping area to achieve the preliminary alignment of the two images.

[0055] Step S9: For two adjacent images I″ Ni and I″ N(i+1) , use the SIFT algorithm to perform feature extraction within the determined overlapping area, use the Euclidean distance for matching, and then use the RANSAC algorithm to screen the matching points to obtain better matching. Use the LSD algorithm to detect line features and perform matching. Divide the image into an M*N grid, and combine the reprojection error of feature matching, line feature matching alignment, and grid structure preservation to design an energy function. Minimize this energy function to perform grid deformation to achieve precise alignment of the overlapping area;

[0056] Among them, the parallel fisheye camera image stitching method uses grid deformation optimization to achieve precise alignment. Based on feature point extraction and matching, line features and grid structure preservation terms are introduced to construct an energy function. Line features can effectively improve the matching and stitching effects in low-texture scenarios, and the grid structure preservation term can make the transition of the non-overlapping area of the original image smooth and natural. It not only improves the fine alignment effect of the overlapping area but also makes the visual effect of the complete stitched image natural. The grid deformation energy function is:

[0057] E = λ lp E lp + λ pa E pa + λ la E la + λ g E g

[0058] Among them, E lp represents the line preservation term, which is used to maintain the linear structure of the line features; E pa and E la are the point alignment term and the line alignment term respectively, which are used to improve the correspondence between the matching points and the lines; E g represents the grid shape preservation term, which reduces the distortion by maintaining the grid structure. λ lp , λ pa , λ la and λ g are the weights of the corresponding terms respectively.

[0059] Step S10: For all pairs of adjacent camera images, use the graph cut method to optimize the global energy function on the Markov random field, obtain the stitching seam that achieves the minimum loss under the given loss definition, select the corresponding image parts on both sides of the optimal stitching seam respectively, and adopt a multi-band fusion algorithm on both sides of the seam line to obtain the final stitched and fused image of all N cameras.

[0060] Among them, the parallel fisheye camera image stitching method uses the graph cut method to find the optimal stitching seam, and obtains the optimal stitching seam by minimizing the global energy function based on the Graph-cut method on the Markov random field (MRF). Select the corresponding image parts on both sides of the obtained stitching seam respectively, instead of simply fusing the two overlapping images, which can be used to hide the image blur and artifacts caused by incomplete alignment, and make the final image have a better visual effect. The global energy function is defined as:

[0061]

[0062] Among them, E d is the image gradient term, representing the data loss energy reflecting pixel saliency. E s is the smoothness loss term, measuring the smoothness energy of discontinuity between adjacent pixels. Adjacent pixels are defined according to the 4-neighborhood.

[0063] The parallel fisheye camera image stitching method uses a multi-band fusion algorithm to fuse different images on both sides of the seam line. By respectively establishing the Laplacian pyramids of each image, the parts in the areas on both sides of the seam line are weighted and fused according to the same layer of the pyramid, and finally the inverse Laplacian transform is performed on the merged pyramid to obtain the final fused image, which can make the transition on both sides of the seam line coherent and have a better visual effect.

[0064] The parallel fisheye camera image stitching method can perform joint processing and stitching on the images collected by multiple cameras, not limited to the traditional dual-camera dual-image stitching. In the case of cameras arranged at a relatively long distance, it can effectively improve the perception ability of the system and provide a large-range continuous wide-angle field of view.

[0065] Embodiment:

[0066] This embodiment provides a large-disparity parallel fisheye camera image stitching method based on moving least squares transformation. The schematic diagram of the hardware structure used in this method is as Figure 1 shown. This figure shows a parallel fisheye stitching system composed of 3 fisheye cameras. The cameras are installed on the ceiling of the inner corridor of the building looking down, arranged in a straight line along the extension direction of the corridor, and the field of view angle of each fisheye camera is 180°; Figure 2It is a schematic layout diagram of the calibration marker board placement in the splicing method. According to the requirements for the field of view range of the parallel fisheye splicing system in the present invention, its field of view range in the direction perpendicular to the corridor extension covers the corridor walls, and its field of view range in the direction along the corridor extension covers the placed calibration marker board. There is a certain overlapping area between adjacent cameras, and only 3 cameras are used to cover a large area of the corridor.

[0067] As Figure 3 shown, the large parallax parallel fisheye camera image splicing method includes the following steps:

[0068] Step S1, Camera installation and normal image acquisition. According to the Figure 1 shown hardware structure diagram and the design method, 3 fisheye cameras are installed looking down on the corridor ceiling in sequence, numbered C i , (i = 1...3), define the plane formed by the optical axis of adjacent cameras C i and C i+1 and the connection line between the cameras as plane S i , and ensure that the internal parameters of the fisheye cameras and the relative position relationship between each camera no longer change. Start the 3 fisheye cameras to collect normal images I Ni , (i = 1...3). In this implementation case, the distance between adjacent cameras is 5.5 meters, the cameras are 3 meters from the ground, and the obtained image resolution is 1344*1400.

[0069] Step S2, Calibration marker board arrangement and calibration image acquisition. In this implementation case, 4 four-color calibration marker boards with a length of 3 meters and a height of 2.4 meters are used as auxiliary calibration objects, and are placed on both sides of plane S Figure 2 respectively according to the i shown calibration marker board placement layout schematic diagram. The distance from each calibration marker board to plane S i is 1.4 meters, so that all the calibration marker boards are within the field of view range of adjacent two fisheye cameras. Start the 3 fisheye cameras to collect calibration images I Ci , (i = 1...3).

[0070] Step S3, Image preprocessing. In this implementation case, select an image with bright edges from the collected images, convert it to a grayscale image and save it as a copy. Use a pair of horizontal and vertical straight lines on the grayscale image to squeeze from the four directions of up, down, left and right respectively to obtain the coordinates of 4 tangent points, so as to calculate the center coordinates and radius, extract the circular effective area and crop all images according to its size and coordinates. The resolution of the cropped image is 1340*1340. Rotate the corrected image I Ci appropriately so that the scene located on the camera connection line is on the horizontal center line of the image, and record the rotation angle θ i corresponding to each camera C i, for the normal image I captured by the camera Ni Rotate automatically according to θ i Perform exposure compensation on all images to balance the exposure level.

[0071] Step S4, Fisheye image unfolding. In this embodiment, the radius of the preprocessed fisheye image is 670 pixels, the maximum incident angle of light is 90°, the fisheye camera uses an equidistant projection model r = fθ, calculates the image remapping matrices mapx and mapy using the longitude and latitude transformation method with reverse interpolation and saves them. For the corrected image I Ci and the normal image I Ni Perform remapping to unfold the fisheye image. The unfolded corrected image and normal image are denoted as I' Ci and I' Ni .

[0072] Step S5, GrabCut image segmentation. In this embodiment, each calibration marker board consists of 4 color blocks. For each unfolded calibration image I' Ci , use the GrabCut algorithm to segment the 1st and 3rd color block images near the center of the image on each calibration marker board. After the user frames the area where the target color block is located, execute the GrabCut algorithm for image segmentation, perform simple annotation on the graph cut result and iterate for optimization until the target color block image is completely extracted. 2 segmentation images can be extracted from each calibration marker board in each calibration image. According to the number of calibration marker boards within the camera's field of view, 4 / 8 segmentation images can be extracted from each calibration image, denoted as I' Cij .

[0073] Step S6, Calculate the control point coordinates before and after MLS deformation. In this embodiment, for each segmentation image I' Cij , traverse one row of image pixels at intervals of one row, search for the coordinates of the first non-zero pixel point from left to right, calculate the slope of the line connecting two adjacent non-zero pixel points above and below, select the points with the absolute value of the slope greater than 1 as the left contour points, select the point closest to the horizontal midline of the image from the left contour points, and record its abscissa as x0. Select one point every 4 points from the left contour points of this image, add its coordinates to the control point coordinates p i before deformation, then set its abscissa value to x0 and keep the ordinate unchanged, and add it to the control point coordinates q i after deformation. Search for the coordinates of the first non-zero pixel point from right to left in the same way, extract the right contour points and add the control point coordinates before and after deformation to p i and q i .

[0074] Step S7, Perform moving least squares transformation. According to each camera C i corresponding segmentation image I'Cij and the control point coordinates p before and after deformation i and q i , calculate the moving least squares transformation matrix A i and save it. According to the calculated A i perform a moving least squares transformation on the unfolded normal image I′ Ni to restore its geometric shape in the overlapping area. The deformed image is denoted as I″ Ni .

[0075] Step S8, overlapping area prediction. In this embodiment, for two adjacent images I″ Ni and I″ N(i+1) , perform binarization through median filtering and adaptive thresholding, divide the grayscale image into 200 columns, count the number of black pixels in each column, traverse each column of the two images, calculate the variance of the number of black pixels in the subsequent 40 columns, and select the part with the smallest variance as the overlapping area to determine the position and width of the overlapping area of two adjacent images.

[0076] Step S9, grid deformation and fine alignment. For two adjacent images I″ Ni and I″ N(i+1) , use the SIFT algorithm to extract features within the determined overlapping area, use the Euclidean distance for matching, and then use the RANSAC algorithm to screen the matching points to obtain better matches. Use the LSD algorithm to detect line features and perform matching. Divide the image into grids, combine the reprojection error of feature matching, line feature matching alignment, and grid structure preservation to construct an energy function, and minimize this energy function for grid deformation to achieve precise alignment of the overlapping areas of adjacent images.

[0077] Step S10, stitching and fusion of stitched images. In this embodiment, for two sets of finely aligned images, use the graph cut method to find the optimal stitching seam within the overlapping area, and adopt a multi-frequency fusion algorithm within a 30-pixel width range on both sides of the stitching seam to obtain the final stitched and fused image of all 3 cameras.

[0078] In summary, the above are only the preferred embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for stitching images of a parallel fisheye camera based on moving least squares transformation, characterized in that, Including: Step S1: Arrange N fisheye cameras C i in a straight line for installation, with the optical axes of the cameras parallel to each other. Define the plane formed by the optical axis of adjacent cameras C i and C i+1 and the connection line between the cameras as plane S i ; The image of the normal field of view captured by each fisheye camera C i is denoted as normal image I Ni , where i = 1...N; Step S2: Place two calibration marker plates between two adjacent fish-eye cameras. The plane of the calibration marker plates is parallel to plane S i and place them on both sides of plane S i respectively; The image containing the calibration marker plate captured by each fish-eye camera C i is denoted as calibration image I Ci ; Multiple parallel dividing lines are set in the calibration marker plate, and the dividing lines are placed vertically when setting the calibration marker plate; Step S3: Preprocess the corrected image I Ci and the normal image I Ni collected by the parallel fisheye stitching system: Step S4. Remap the corrected image I Ci collected in step S3 and the normal image I Ni to unwrap the fisheye image. The unwrapped corrected image and normal image are denoted as I′ Ci and I′ Ni respectively; Step S5: Unfold the corrected image I' Ci Perform image segmentation, extract the regional images between two adjacent dividing lines in the corrected marker plate image, and in the unfolded corrected image I' Ci set the pixel values of the areas outside each regional image to zero to obtain the image I' Cij ; Step S6: Traverse each image I′ Cij row by row, find the point with the first non - zero pixel value from left to right, and save the coordinates of these points into matrix L ij ; Calculate matrix L ij For each point in up calculate the slopes k down and k up with its two adjacent points before and after. If the absolute value of k down or k threshold is greater than the threshold k ij , then this point can be considered as a vertically arranged point and retained, otherwise it is removed; for the filtered matrix L ij , find the point closest to the horizontal midline of the image, and record its abscissa as x0; uniformly select points from L i and append them to matrix p i to provide the control point coordinates before deformation for the moving least squares transformation; then set the abscissas of these points to x0 and keep the ordinates unchanged, and append them to matrix q to provide the control point coordinates after deformation for the moving least squares transformation; Traverse each image I′ Cij For each row of , find the first point with a non-zero pixel value from right to left, and save the coordinates of these points to the matrix R ij Calculate the matrix R ij The slope k of each point in the up and k down , if k up or k down The absolute value of threshold , then the point can be considered as a vertical arrangement point and retained, otherwise it will be removed; for the screened matrix R ij , the point closest to the horizontal midline of the image, record its horizontal coordinate as x0; from R ij Uniformly select points from the matrix and append them to the matrix p i Then set the horizontal coordinates of these points to x0, keep the vertical coordinates unchanged, and append them to the matrix q i middle; Traverse each segmented image, and append the control point coordinates before deformation provided by its moving least squares transformation and the control point coordinates after deformation provided by the moving least squares transformation to matrix p i and matrix q i respectively; Step S7. According to matrix p i and matrix q i calculate the moving least squares transformation matrix A i corresponding to each camera C i . According to the calculated A i perform a moving least squares transformation on the unfolded normal image I′ Ni to restore its geometric shape in the overlapping area. The deformed image is denoted as I′ N ′ i ; Step S8: For two adjacent deformed images obtained in step S7, first perform binarization processing on each of them. Divide each binarized image into multiple groups by columns. Assume the number of groups is n groups. Count the number of black pixels in each group of each image and save them into arrays B i and B i+1 respectively; Place a sliding window with a width of m on both B i and B i+1 , where m < n; Calculate the variance of the number of black pixels in the two sliding windows. The interval where the two windows are located when the variance is the smallest is the overlapping area of the two images. Determine the position and width of the overlapping area as the preliminary alignment of the two deformed images. Step S9: For two adjacent deformed images that are preliminarily aligned in step S8, perform feature matching within the determined overlapping area to achieve precise alignment of the overlapping area. Step S10: For all pairwise adjacent images processed in step S9, determine the optimal stitching seam, select the corresponding image parts on both sides of the optimal stitching seam respectively, and use a multi-band fusion algorithm on both sides of the seam line to obtain the final stitched and fused image of all N cameras.

2. The method for stitching images of a parallel fisheye camera based on moving least squares transformation according to claim 1, characterized in that, In the step S2, the distance between the plane of the calibration marker plate and the plane S i is approximately half of the camera interval.

3. A method for stitching images of a parallel fisheye camera based on moving least squares transformation according to claim 1, characterized in that In step S2, several rectangular patterns with strong color contrast are set in the marker board, and the area between two patterns is the dividing line.

4. A method for stitching parallel fisheye camera images based on moving least squares transformation according to claim 1, characterized in that, The preprocessing method in step S3 includes: using the line scanning method to extract the circular effective imaging area in the image, rotating it so that the scene located on the connection line of two adjacent cameras is on the horizontal center line of the image, and finally obtaining the preprocessed fisheye image with equal length and width after exposure compensation.

5. A method for stitching images of a parallel fisheye camera based on moving least squares transformation according to claim 1, characterized in that, In step S4, the longitude and latitude transformation method with reverse interpolation is used to calculate the image remapping matrix, and the image is remapped to unfold the fisheye image.

6. A method for stitching parallel fisheye camera images based on moving least squares transformation according to claim 1, characterized in that, In the step S5, the GrabCut algorithm is used to perform interactive image segmentation on the corrected unfolded image I'. Ci ​ 7. A method for stitching parallel fisheye camera images based on moving least squares transformation according to claim 1, characterized in that In the step S6, the threshold k threshold takes the value of 1.

8. The method for stitching images of a parallel fisheye camera based on moving least squares transformation according to claim 1, characterized in that, The specific method of step S9 is: use the SIFT algorithm to extract features within the determined overlapping area, use the Euclidean distance for matching, and then use the RANSAC algorithm to screen the matching points to obtain better matches; use the LSD algorithm to detect line features and perform matching; divide the image into grids, combine the feature matching reprojection error, line feature matching alignment, and grid structure preservation to design an energy function, and minimize this energy function to perform grid deformation to achieve precise alignment of the overlapping area.

9. A method for stitching parallel fisheye camera images based on moving least squares transformation according to claim 8, characterized in that, The energy function of the grid deformation in step S9 is: E = λ lp E lp + λ pa E pa + λ la E la + λ g E g Among them, E lp represents a line retention term; E pa and E la are a point alignment term and a line alignment term respectively; E g represents a grid shape retention term, λ lp , λ pa , λ la and λ g are the weights of the corresponding terms respectively.

10. A method for stitching images of a parallel fisheye camera based on moving least squares transformation according to claim 8, characterized in that, In step S10, for all pairwise adjacent camera images, use the graph cut method to optimize the global energy function on the Markov random field to obtain the stitching seam that minimizes the loss under the given loss definition, select the corresponding image parts on both sides of the optimal stitching seam respectively, and use a multi-band fusion algorithm on both sides of the seam line to obtain the final stitched and fused image of all N cameras.

Images