A method for stitching images acquired by multiple cameras

Abstract

The present invention discloses a method for stitching images acquired by multiple cameras. The method includes: arranging cameras and determining the format information of a single camera; calibrating the cameras using a calibration board and calculating the camera parameters; calculating the physical coordinates of the processing alignment pattern corresponding to each camera according to a preset camera spacing; performing image processing on a machine processing plane based on the physical coordinates of the centers of the camera alignment patterns to determine the alignment patterns corresponding to each camera and calculating an alignment transformation matrix; performing a stitching transformation on the images acquired by the cameras to determine a stitched transformation image and establishing a set of stitched transformation images; performing image stitching on the stitched transformation images in the set of stitched transformation images in the row direction to obtain a row-direction stitched image; and performing image stitching on the row-direction stitched image in the column direction to determine a final stitched image. According to the processing field-of-view information of the processing format input by the user, the present invention effectively completes the stitching of the images acquired by multiple cameras and improves the accuracy of image stitching.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for stitching images acquired by multiple cameras. Background Art

[0002] In visual applications, for scenes with a large field of view, the field of view and resolution of a single camera are difficult to meet the requirements, and usually multiple cameras are required to cooperate to complete image acquisition and stitching. At present, multi-camera image stitching technology has been widely used in industrial processing, virtual reality, autonomous driving and other fields, and has broad application prospects. However, there are still some problems in the existing multi-camera image stitching methods, such as unsatisfactory stitching effect, slow processing speed, high requirements for image quality, etc.

[0003] Therefore, the existing technology has defects and needs to be improved urgently. Summary of the Invention

[0004] In view of the above problems, the purpose of the present invention is to provide a method for stitching images acquired by multiple cameras, which can effectively complete the stitching of images acquired by multiple cameras according to the processing format field of view information input by the user, improve the stitching effect of the images, and is applicable to various types of cameras.

[0005] The first aspect of the present invention provides a method for stitching images acquired by multiple cameras, including:

[0006] Arrange cameras and determine the format information of a single camera; the format information includes format width and format height;

[0007] Calibrate the cameras through a calibration board and calculate the camera parameters; the camera parameters include the internal parameters, external parameters and correction coefficients of the cameras;

[0008] Calculate the physical coordinates of the processing alignment pattern corresponding to each camera according to the preset camera spacing; the preset camera spacing includes the camera arrangement spacing in the x direction and the camera arrangement spacing in the y direction;

[0009] Perform image processing on the machine processing plane based on the physical coordinates of the centers of the camera alignment patterns, determine the alignment patterns corresponding to each camera, and calculate the alignment change matrix according to the alignment patterns;

[0010] Perform stitching transformation on the images acquired by the cameras according to the camera parameters and the alignment change matrix, determine the stitched transformation images, and establish a set of stitched transformation images;

[0011] Perform image stitching on the stitched transformation images in the set of stitched transformation images in the row direction to obtain the row-direction stitched image;

[0012] Perform image stitching on the row-direction stitched image in the column direction to determine the final stitched image.

[0013] In this solution, determining the frame information of a single camera includes:

[0014] Obtain the width W and height H of the machining frame input by the user;

[0015] Analyze based on the width W and height H of the machining frame, and combine with the number n of cameras in the x direction of the machining screen x and the number n of cameras in the y direction y , the arrangement pitch d of cameras in the x direction 0 and the arrangement pitch d of cameras in the y direction 1 for calculation to determine the width w and height h of the frame of a single camera; ; .

[0016] In this solution, calibrating the camera with a calibration plate and calculating the camera parameters include:

[0017] Place the calibration plate on the machining plane at the center of the camera's field of view, and construct a three-dimensional point set P of the world coordinates of the calibration plate corner points according to the physical size information of the calibration plate 0 ;

[0018] Obtain the calibration plate image;

[0019] Identify the calibration plate corner points of the calibration plate image and construct a two-dimensional pixel point set P of the image 1 ;

[0020] Analyze based on a preset camera model, the three-dimensional point set P of the world coordinates of the calibration plate corner points 0 and the two-dimensional pixel point set P of the image 1 to determine the internal parameters, external parameters, and correction coefficients of the camera.

[0021] In this solution, calculating the physical coordinates of the machining alignment pattern corresponding to each camera according to the preset camera pitch includes:

[0022] Determine the camera in the r i th row and c i th column as camera N rc ;

[0023] Calculate the physical coordinates (x rc , y rc , y rc ) of the center position of the machining alignment pattern corresponding to camera N in the machining frame;

[0024] ;

[0025] ;

[0026] Wherein, W is the width of the machine processing area, H is the height of the machine processing area, and n x is the number of cameras in the x - direction of the machine - processed screen, and n y is the number of cameras in the y - direction, d 0 is the arrangement pitch of cameras in the x - direction, and d 1 is the arrangement pitch of cameras in the y - direction, r i and c i are the number of rows and columns corresponding to camera N rc respectively.

[0027] In this solution, calculating the alignment transformation matrix according to the alignment pattern includes:

[0028] Obtaining the first distorted image I i containing the alignment pattern collected by camera N 0 ;

[0029] Performing distortion correction on the first distorted image I 0 according to the internal parameters, external parameters and correction coefficients of the camera to obtain the corrected image I 1 ;

[0030] Identifying the pixel coordinates of the center of the alignment pattern from the corrected image I 1 ;

[0031] Fitting the pixel coordinates of the center of the alignment pattern and the corresponding physical coordinates to obtain the alignment transformation matrix M from pixel coordinates to physical coordinates.

[0032] In this solution, performing splicing transformation on the images collected by the camera according to the camera parameters and the alignment transformation matrix, determining the spliced transformation image, and establishing the set of spliced transformation images includes:

[0033] Obtaining the second distorted image I i collected by camera N 2 ;

[0034] Performing distortion correction on the second distorted image I 2 according to the internal parameters, external parameters and correction coefficients of the camera to obtain the corrected image I 3 ;

[0035] Performing alignment transformation on the corrected image I 3 through the alignment transformation matrix M to obtain the alignment transformation image I 4 ;

[0036] Cropping the alignment - transformed image I 4 according to the area information of a single camera to obtain the spliced transformation image I 5 ;

[0037] The stitched transformed image I 5 Add to the stitching transformation image set O.

[0038] In this solution, the image stitching of the stitching transformation images in the stitching transformation image set based on the row direction to obtain the row-direction stitching image includes:

[0039] Obtain the rth i Row c i The stitched transformed image I corresponding to the camera column 5(ri-ci) ;

[0040] The first r i The row-wise stitched image R i Initialize to r i The concatenated transformed image I of row 0 5(ri-0) ;

[0041] Calculate the clipping boundary L r and the clipping boundary L 0 ;

[0042] ;

[0043] ;

[0044] Where w is the width of a single camera, d 0 For the x-direction camera layout spacing, L r is the row-wise stitching image R i The right clipping boundary, L 0 is the stitching transformation image I 5(ri-0) The left clipping boundary, w r is the row-wise stitching image R i The width of , e is the fusion constant;

[0045] By the clipping boundary L r Stitch the image R in the row direction i Crop and stitch the row-wise image R according to the cropping result. i Make updates;

[0046] By clipping the boundary L 0 For the r i Row c a The stitched transformed image I corresponding to the camera column 5(ri-ca) Perform cropping and determine the cropped image I 6(ri-ca) ; Among them, 0<c a ≤c i ; The spliced ​​transformed image I 5(ri-ca) is the row-wise stitching image R i Adjacent stitched images;

[0047] Stitch the cropped row-direction stitched image R i and the cropped image I 6(ri-ca) Perform image fusion, and update the cropped row-direction stitched image R according to the fused image i ;

[0048] Continue to perform image fusion on the updated row-direction stitched image R i with the next stitched transformation image.

[0049] In this solution, the cropped row-direction stitched image R r is cropped through the cropping boundary L i to obtain the cropped row-direction stitched image R, and the r 0 -th row and c i -th column camera-corresponding stitched transformation image I a is cropped through the cropping boundary L 5(ri-ca) to determine the cropped image I 6(ri-ca) , including:

[0050] Determine the first ROI box (0, 0, L r , h) according to the cropping boundary L r ;

[0051] Crop the row-direction stitched image Ri through the first ROI box (0, 0, L r , h);

[0052] Determine the second ROI box (L 0 , 0, w - L 0 , h) according to the cropping boundary L 0 ;

[0053] Crop the stitched transformation image I 0 through the second ROI box (L 0 , 0, w - L 5(ri-ca) .

[0054] In this solution, the step of performing image fusion on the cropped row-direction stitched image R i and the cropped image I 6(ri-ca) includes:

[0055] Analyze the cropped row-direction stitched image R i and the cropped image I 6(ri-ca) to determine the overlapping field of view area and the non-overlapping field of view area;

[0056] According to the pixel gray value v i of the row-direction stitched image R 0 and the pixel gray value v 6(ri-ca) of the cropped image I1 Calculate the pixel grayscale value v of the overlapping field of view area;

[0057] ;

[0058] where k is the pixel column where the overlapping area pixel is located, and w 3 is the width of the overlapping area of the row - direction stitched image R i and the cropped image I 6(ri-ca) ;

[0059] Update the image of the overlapping field of view area according to the pixel grayscale value v.

[0060] The present invention discloses a method for stitching images acquired by multiple cameras. The method includes: arranging cameras and determining the format information of a single camera; calibrating the cameras through a calibration board and calculating camera parameters; calculating the physical coordinates of the machining alignment pattern corresponding to each camera according to a preset camera spacing; performing image machining on the machine - processing plane based on the physical coordinates of the centers of the camera alignment patterns to determine the alignment patterns corresponding to each camera and calculating the alignment transformation matrix; performing stitching transformation on the images acquired by the cameras to determine the stitched transformation images and establishing a set of stitched transformation images; performing image stitching on the stitched transformation images in the set of stitched transformation images in the row direction to obtain the row - direction stitched image; and performing image stitching on the row - direction stitched image in the column direction to determine the final stitched image. The present invention effectively completes the stitching of the images acquired by multiple cameras according to the processing format field - of - view information input by the user, improving the accuracy of image stitching. Brief Description of the Drawings

[0061] Figure 1 Shows a flowchart of a method for stitching images acquired by multiple cameras provided by the present invention;

[0062] Figure 2 Shows a flowchart of a method for calculating camera parameters provided by the present invention;

[0063] Figure 3 Shows a flowchart of a method for calculating the alignment transformation matrix provided by the present invention. Detailed Embodiments

[0064] In order to more clearly understand the above - mentioned objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0065] In the following description, many specific details are set forth to facilitate a thorough understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0066] Figure 1 Shows a flowchart of a method for stitching images collected by multiple cameras provided by the present invention.

[0067] As Figure 1 shown, the present invention discloses a method for stitching images collected by multiple cameras, including:

[0068] S102, arranging cameras and determining the frame information of a single camera; the frame information includes the frame width and the frame height;

[0069] S104, calibrating the cameras through a calibration board and calculating the camera parameters; the camera parameters include the internal parameters, external parameters and correction coefficients of the cameras;

[0070] S106, calculating the physical coordinates of the processing alignment pattern corresponding to each camera according to the preset camera spacing; the preset camera spacing includes the camera arrangement spacing in the x direction and the camera arrangement spacing in the y direction;

[0071] S108, performing image processing on the machine processing plane based on the physical coordinates of the center of the camera alignment pattern, determining the alignment pattern corresponding to each camera, and calculating the alignment change matrix according to the alignment pattern;

[0072] S110, performing stitching transformation on the images collected by the cameras according to the camera parameters and the alignment change matrix, determining the stitched transformation images, and establishing a set of stitched transformation images;

[0073] S112, performing image stitching on the stitched transformation images in the set of stitched transformation images in the row direction to obtain the row-direction stitched image;

[0074] S114, performing image stitching on the row-direction stitched image in the column direction to determine the final stitched image.

[0075] According to the embodiments of the present invention, during the process of arranging cameras, a plurality of cameras are arranged in a row-column form, and the spacing between different cameras is made as approximately the same as possible, so that there is a certain field of view overlap between different adjacent cameras, and at the same time, the fields of view of all cameras include the machine processing field of view preset by the user after being superimposed. According to the machine processing frame width W and frame height H input by the user, combined with the number of cameras in the x direction and y direction of the machine processing plane and the camera arrangement spacing, the frame information (including the frame width and the frame height) of each single camera is determined. All cameras are calibrated using a calibration board, and a three-dimensional point set P of the world coordinates of the corner points of the calibration board is constructed according to the physical size information of the calibration board 0 , and an image two-dimensional pixel point P is constructed according to the identified corner points of the calibration board image 1 . Through P 0 and P 1Calculate the internal parameters, external parameters, and correction coefficients of the camera. Calculate the center position of the machining alignment pattern corresponding to each camera (i.e., the physical coordinates of the center of the camera alignment pattern). Place processing materials such as white paper or wooden boards on the machine machining plane, and machine the alignment pattern corresponding to each camera on the processing material according to the physical coordinates of the center of the camera alignment pattern. Identify the pixel coordinates of the alignment pattern and fit them with the corresponding physical coordinates to obtain the alignment transformation matrix M from pixel coordinates to physical coordinates.

[0076] When the camera captures an image, perform stitching transformation on the image captured by the camera according to the internal parameters, external parameters, correction coefficients, and alignment transformation matrix of the camera to establish a set O of stitched transformation images. First, perform image stitching in the row direction. For each row, determine the stitched transformation image of the 0th column as the initial image of the row-direction stitched image. Select the stitched transformation images of adjacent columns, perform boundary cropping on the two and then perform image fusion, and update the row-direction stitched image according to the fused image. Then continue to stitch and fuse the updated row-direction stitched image with the stitched transformation image of the next adjacent column until the final row-direction stitched image is determined and added to the set R of row-direction stitched images. Finally, take out all r row-direction stitched result images from the set R of row-direction stitched images and stitch them vertically into one image to obtain the final stitched result image.

[0077] According to an embodiment of the present invention, determining the frame information of a single camera includes:

[0078] Obtain the width W and height H of the machine machining frame input by the user;

[0079] Analyze based on the width W and height H of the machine machining frame, and combine the number n of cameras in the x direction of the machine machining screen x , the number n of cameras in the y direction y , the arrangement pitch d of cameras in the x direction 0 and the arrangement pitch d of cameras in the y direction 1 to calculate and determine the width w and height h of the frame of a single camera; ; .

[0080] It should be noted that the frame information of the camera includes the width w and height h of the camera frame. Input the width W and height H of the machine machining frame input by the user into the calculation formula of the camera frame information to determine the width w and height h of the frame of a single camera

[0081] Figure 2 Fig. shows the flowchart of the camera parameter calculation method provided by the present invention.

[0082] As Figure 2 shown, according to an embodiment of the present invention, calibrate the camera through a calibration board and calculate the camera parameters, including:

[0083] S202. Place the calibration board on the machining plane at the exact center of the camera's field of view, and construct a three-dimensional set P of world coordinates of the calibration board corner points based on the physical dimension information of the calibration board. 0 ;

[0084] S204. Obtain the calibration board image.

[0085] S206. Identify the calibration board corner points of the calibration board image, and construct a two-dimensional set P of image pixel points. 1 ;

[0086] S208. Analyze based on a preset camera model, the three-dimensional set P of world coordinates of the calibration board corner points 0 and the two-dimensional set P of image pixel points 1 to determine the internal parameters, external parameters, and calibration coefficients of the camera.

[0087] It should be noted that the calibration board can adopt a checkerboard structure. Define the world coordinate system at the first inner corner point in the upper left corner of the calibration board. The Z-axis is perpendicular to the calibration board plane. Determine the coordinates of each inner corner point of the checkerboard and determine the three-dimensional set P of world coordinates. 0 . Extract the two-dimensional pixel coordinates of the calibration board corner points through the image processing algorithm preset by the system, and the two-dimensional set P of image pixel points 1 .

[0088] The preset camera model is preset by the system and can be a pinhole imaging model, a mei model, etc. The internal parameters, external parameters, and calibration coefficients of the camera can be calculated using solution methods such as Zhang Zheng-you calibration and Tsai's two-step calibration.

[0089] According to the embodiments of the present invention, calculate the physical coordinates of the machining alignment pattern corresponding to each camera according to the preset camera spacing, including:

[0090] Determine the r i th row and c i th column corresponding camera as camera N rc ;

[0091] Calculate the physical coordinates (x rc , y rc , y rc ) of the center position of the machining alignment pattern corresponding to camera N in the machining format.

[0092] ;

[0093] ;

[0094] where W is the width of the machining format, H is the height of the machining format, n x is the number of cameras in the x direction of the machining screen, ny is the number of cameras in the y direction, d 0 is the camera arrangement pitch in the x direction, d 1 is the camera arrangement pitch in the y direction, r i and c i are the number of rows and columns corresponding to camera N rc respectively.

[0095] It should be noted that based on the total number of cameras n, the value ranges of r i and c i can be determined. r i = 1, 2... r; c i = 1, 2... c; r is the total number of rows of the camera arrangement, c is the total number of columns of the camera arrangement, and r × c = n.

[0096] Figure 3 shows the flowchart of the alignment change matrix calculation method provided by the present invention.

[0097] As Figure 3 shown, according to an embodiment of the present invention, calculating the alignment change matrix according to the alignment pattern includes:

[0098] S302, obtaining the first distorted image I i containing the alignment pattern collected by camera N 0 ;

[0099] S304, performing distortion correction on the first distorted image I 0 according to the internal parameters, external parameters and correction coefficients of the camera to obtain the corrected image I 1 ;

[0100] S306, identifying the pixel coordinates of the center of the alignment pattern from the corrected image I 1 ;

[0101] S308, fitting the pixel coordinates of the center of the alignment pattern and the corresponding physical coordinates to obtain the alignment transformation matrix M from pixel coordinates to physical coordinates.

[0102] It should be noted that for the first distorted image I i of the alignment pattern collected by camera N 0 (i = 0, 1, 2... n - 1; n is the number of cameras), preferably perform distortion correction according to the already determined internal parameters, external parameters and correction coefficients of the camera, and use fitting methods such as the least squares method to fit the pixel coordinates of the center of the alignment pattern in the corrected image I 1 with the corresponding physical coordinates to determine the alignment transformation matrix M of the pixel coordinates of the center of the alignment pattern identified in camera N i .

[0103] According to an embodiment of the present invention, the camera-acquired images are subjected to stitching transformation according to camera parameters and an alignment transformation matrix to determine stitched transformation images, and a set of stitched transformation images is established, including:

[0104] Obtain the Nth camera i The second distorted image I collected 2 ;

[0105] Perform distortion correction on the second distorted image I according to the internal parameters, external parameters and correction coefficients of the camera 2 to obtain the corrected image I 3 ;

[0106] Perform alignment transformation on the corrected image I through the alignment transformation matrix M 3 to obtain the alignment transformation image I 4 ;

[0107] Crop the alignment-transformed image I according to the frame information of a single camera 4 to obtain the stitched transformation image I 5 ;

[0108] Add the stitched transformation image I 5 to the set O of stitched transformation images.

[0109] It should be noted that first, the second distorted image I collected by the Nth camera i is subjected to stitching transformation. The stitching transformation includes internal distortion correction, alignment transformation, and cropping. The image is subjected to distortion correction through the internal parameters, external parameters and correction coefficients of the camera, the image is subjected to alignment transformation through the alignment transformation matrix M, and finally the image is cropped according to the frame information of a single camera to determine the stitched transformation image I 2 after the second distorted image I is subjected to stitching transformation 2 . According to the above method, the images collected by each camera are sequentially subjected to stitching transformation, and the obtained stitched transformation images are added to the set O of stitched transformation images. 5

[0110] According to an embodiment of the present invention, image stitching is performed on the stitched transformation images in the set of stitched transformation images in the row direction to obtain a row-direction stitched image, including:

[0111] Obtain the stitched transformation image I corresponding to the camera in the rth row and cth column from the set O of stitched transformation images i row i column 5(ri-ci) ;

[0112] Initialize the row-direction stitched image R i in the rth row direction i as the stitched transformation image I in the rth row and 0th column i ​5(ri-0) ;

[0113] Calculate the cropping boundary L r and the cropping boundary L 0 ;

[0114] ;

[0115] ;

[0116] where w is the width of the field of view of a single camera, d 0 is the camera arrangement pitch in the x direction, L r is the cropping boundary on the right side of the row-direction stitched image R i ; L 0 is the cropping boundary on the left side of the stitched transformation image I 5(ri-0) ; w r is the width of the row-direction stitched image R i ; e is the fusion constant;

[0117] Crop the row-direction stitched image R r using the cropping boundary L i , and update the row-direction stitched image R i according to the cropping result;

[0118] Crop the stitched transformation image I 0 corresponding to the camera in the r i -th row and c a -th column using the cropping boundary L 5(ri-ca) to determine the cropped image I 6(ri-ca) ; where 0 < c a ≤ c i ; the stitched transformation image I 5(ri-ca) is the adjacent stitched image of the row-direction stitched image R i ;

[0119] Fuse the cropped row-direction stitched image R i and the cropped image I 6(ri-ca) , and update the cropped row-direction stitched image R i according to the fused image;

[0120] Continue to fuse the updated row-direction stitched image R i with the next stitched transformation image.

[0121] It should be noted that the stitched transformation images corresponding to each row of cameras are stitched left and right in the row direction (i.e., stitching an entire row of images into a single row-direction image) to obtain r sets of row-direction stitched images. Before stitching starts, the stitched transformation image in the r i -th row and 0-th column is determined as the ri The initial value of the row-direction stitched image in the row direction. First, it is combined with the stitched transformation image of the first column in the r-th i row for boundary stitching. Then, the cropped image is subjected to image fusion, and the resulting fused image is used as the new row-direction stitched image. Next, continue to traverse the stitched transformation image of the next column in the r-th i row for image fusion until all columns in the r-th i row are traversed, obtaining the final row-direction stitched image for the r-th i row.

[0122] According to an embodiment of the present invention, the row-direction stitched image R is cropped by the cropping boundary L r and the stitched transformation image I corresponding to the camera in the c-th column of the r-th i row is cropped by the cropping boundary L 0 to determine the cropped image I i a 5(ri-ca) 6(ri-ca) including:

[0123] Determine the first ROI box (0, 0, L r , h) according to the cropping boundary L; r

[0124] Crop the row-direction stitched image Ri with the first ROI box (0, 0, L r , h);

[0125] Determine the second ROI box (L 0 , 0, w - L 0 , h) according to the cropping boundary L; 0

[0126] Crop the stitched transformation image I with the second ROI box (L 0 , 0, w - L 0 , h). 5(ri-ca)

[0127] It should be noted that both the first ROI box and the second ROI box are rectangular regions, and the ROI box parameters are, from left to right, the abscissa, ordinate, width, and height of the upper left corner of the ROI box. For example, the first ROI box (0, 0, L r , h) is a rectangular region with the upper left corner coordinates of the image being (0, 0), the width being L r , and the height being h.

[0128] According to an embodiment of the present invention, the cropped row-direction stitched image R i and the cropped image I 6(ri-ca) are subjected to image fusion, including:

[0129] ​​​​​​The stitched image R in the row direction after cropping i and the cropped image I 6(ri-ca) are analyzed to determine the overlapping field of view area and the non - overlapping field of view area;

[0130] Based on the pixel gray value v of the stitched image R in the row direction i and the pixel gray value v of the cropped image I 0 and the pixel gray value v of the cropped image I 6(ri-ca) calculate the pixel gray value v of the overlapping field of view area; 1 ;

[0131] ;

[0132] where k is the pixel column where the overlapping area pixel is located, and w 3 is the width of the overlapping area of the stitched image R in the row direction i and the cropped image I 6(ri-ca) ;

[0133] Update the image of the overlapping field of view area according to the pixel gray value v.

[0134] It should be noted that during the image fusion process, first, the overlapping field of view area where the stitched image in the row direction and the cropped image overlap is identified. The pixel gray value of the stitched image in the row direction and the pixel gray value of the cropped image in the overlapping field of view area are input into the pixel gray value calculation formula preset by the system to calculate the pixel gray value of the overlapping field of view area, and the image of the overlapping field of view area is adjusted. The non - overlapping areas retain the pixel gray values of their respective images, thus completing the image fusion.

[0135] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between the user terminal and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions. For example, "obtaining the width W and height H of the machine - processed format input by the user" and "obtaining the calibration plate image" involved in this disclosure are all obtained under full authorization.

[0136] The present invention discloses a method for stitching images acquired by multiple cameras. The method includes: arranging cameras and determining the format information of a single camera; calibrating the cameras using a calibration board and calculating camera parameters; calculating the physical coordinates of the machining alignment patterns corresponding to each camera according to a preset camera spacing; performing image machining on a machine machining plane based on the physical coordinates of the centers of the camera alignment patterns to determine the alignment patterns corresponding to each camera and calculating an alignment transformation matrix; performing a stitching transformation on the images acquired by the cameras to determine a stitched transformation image and establishing a set of stitched transformation images; performing image stitching on the stitched transformation images in the set of stitched transformation images in the row direction to obtain a row-direction stitched image; and performing image stitching on the row-direction stitched image in the column direction to determine a final stitched image. The present invention effectively completes the stitching of the images acquired by multiple cameras according to the processed field-of-view information of the input processing format by the user, and improves the accuracy of image stitching. In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the couplings, direct couplings, or communication connections between the various components shown or discussed may be through some interfaces. The indirect couplings or communication connections of the devices or units may be electrical, mechanical, or other forms.

[0137] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they may be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, in each embodiment of the present invention, the various functional units may all be integrated in a processing unit, or each unit may be separately used as a unit, or two or more units may be integrated in one unit; the above-mentioned integrated units may be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0139] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including those of the above method embodiments. The aforementioned storage medium includes various media that can store program codes, such as removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0140] Alternatively, if the above integrated units of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program codes, such as removable storage devices, ROM, RAM, magnetic disks, or optical discs.

Claims

1. A method for stitching images collected by multiple cameras, characterized in that: include: Arrange cameras and determine format information of a single camera; the format information includes format width and format height; Calibrate the camera using a calibration plate to calculate camera parameters; the camera parameters include the camera's intrinsic parameters, extrinsic parameters, and correction coefficients; Calculate the physical coordinates of the processing alignment pattern corresponding to each camera according to a preset camera spacing; the preset camera spacing includes the camera arrangement spacing in the x direction and the camera arrangement spacing in the y direction; Performing image processing on a machine processing plane based on the physical coordinates of the center of the camera alignment pattern, determining the alignment pattern corresponding to each camera, and calculating an alignment change matrix based on the alignment pattern; Performing splicing transformation on the camera captured images according to the camera parameters and the alignment change matrix, determining a splicing transformation image, and establishing a splicing transformation image set; Performing image stitching on the stitching transformation images in the stitching transformation image set based on the row direction to obtain a row-direction stitching image; Performing image stitching on the row-direction stitched images based on the column direction to determine a final stitched image; The determining of the format information of a single camera includes: Obtain the machine processing width W and height H input by the user; According to the machine processing width W and height H, the number of cameras n in the x direction of the machine processing screen is analyzed. x , the number of cameras in the y direction n y , calculate the camera arrangement spacing d0 in the x direction and the camera arrangement spacing d1 in the y direction to determine the format width w and format height h of a single camera; w=W-(n x -1)d0; h=H-(n y -1)d1; The step of performing image stitching on the stitching transformation images in the stitching transformation image set based on the row direction to obtain a row-direction stitching image includes: Obtain the rth i Row c i The stitched transformed image I corresponding to the camera column 5(ri-ci) ; The first r i The row-wise stitched image R i Initialized to r i The concatenated transformed image I of row 0 5(ri-0) ; Calculate the clipping boundary L r and clipping boundary L0; Where w is the width of a single camera, d0 is the distance between cameras in the x direction, and L r is the row-wise stitching image R i The cropping boundary on the right, L0 is the stitching transformation image I 5(ri-0) The left clipping boundary, w r is the row-wise stitching image R i The width of , e is the fusion constant; By the clipping boundary L r Stitch the image R in the row direction i Crop and stitch the row-wise image R according to the cropping result. i Make updates; By clipping the boundary L0 to the rth i Row c a The stitched transformed image I corresponding to the camera column 5(ri-ca) Perform cropping and determine the cropped image I 6(ri-ca) ; Among them, 0<c a ≤c i ; The spliced ​​transformed image I 5(ri-ca) is the row-wise stitching image R i Adjacent stitched images; The cropped row-wise stitched image R i and cropped image I 6(ri-ca) Perform image fusion and stitch the cropped row-wise image R according to the fused image i Make updates; The updated row-wise stitched image R i Continue image fusion with the next stitching transformation image.

2. The multi-camera image stitching method according to claim 1, characterized in that: The step of calibrating the camera by using the calibration board and calculating the camera parameters includes: Place the calibration plate on the machined plane at the exact center of the camera field of view, and construct a world coordinate three-dimensional point set P0 of the corner points of the calibration plate according to the physical size information of the calibration plate; Get the calibration plate image; Identify the calibration plate corner points of the calibration plate image and construct a two-dimensional pixel point set P1 of the image; Based on the preset camera model, the world coordinate three-dimensional point set P0 of the corner points of the calibration plate and the image two-dimensional pixel point set P1, the camera's intrinsic parameters, extrinsic parameters and correction coefficients are determined.

3. The multi-camera image stitching method according to claim 1, characterized in that: The step of calculating the physical coordinates of the processing alignment pattern corresponding to each camera according to the preset camera spacing includes: The first r i Row, c i The camera corresponding to the column is determined as camera N rc ; Calculate the camera N rc The physical coordinates (x rc ,y rc ); Among them, W is the width of the machine processing area, H is the height of the machine processing area, and n x is the number of cameras in the x direction of the machine processing screen, n y is the number of cameras in the y direction, d0 is the spacing between cameras in the x direction, d1 is the spacing between cameras in the y direction, r i and c i They are camera N rc The corresponding number of rows and columns.

4. The method for stitching images collected by multiple cameras according to claim 1, characterized in that: The calculating the alignment change matrix according to the alignment pattern comprises: Get Camera N i A first distorted image I0 containing an alignment pattern is acquired; Performing distortion correction on the first distorted image I0 according to the intrinsic parameters, extrinsic parameters and correction coefficients of the camera to obtain a corrected image I1; Identifying pixel coordinates of a center of an alignment pattern from the rectified image I1; The pixel coordinates of the center of the alignment pattern and the corresponding physical coordinates are fitted to obtain an alignment transformation matrix M from pixel coordinates to physical coordinates.

5. The multi-camera image stitching method according to claim 1, characterized in that: The step of performing splicing transformation on the camera captured images according to the camera parameters and the alignment change matrix, determining a splicing transformation image, and establishing a splicing transformation image set includes: Get Camera N i A second distorted image I2 is acquired; Performing distortion correction on the second distorted image I2 according to the intrinsic parameters, extrinsic parameters and correction coefficients of the camera to obtain a corrected image I3; Performing an alignment transformation on the corrected image I3 by using an alignment transformation matrix M to obtain an alignment transformation image I4; The aligned transformed image I4 is cropped according to the format information of a single camera to obtain a spliced ​​transformed image I5; The stitched transformed image I5 is added to the stitched transformed image set O.

6. The multi-camera image stitching method according to claim 1, characterized in that: The clipping boundary L r Stitch the image R in the row direction i Cutting is performed by cutting the boundary L0 for the rth i Row c a The stitched transformed image I corresponding to the camera column 5(ri-ca) Perform cropping and determine the cropped image I 6(ri-ca) ,include: According to the clipping boundary L r Determine the first ROI box (0, 0, L r , h); Through the first ROI box (0, 0, L r , h) cropping the row-wise stitched image Ri; Determine the second ROI frame (L0, 0, w-L0, h) according to the cropping boundary L0; The stitching transformation image I is processed by the second ROI frame (L0, 0, w-L0, h) 5(ri-ca) to crop.

7. The multi-camera image stitching method according to claim 1, characterized in that: The cropped row-wise stitched image R i and cropped image I 6(ri-ca) Perform image fusion, including: The cropped row-wise stitched image R i and cropped image I 6(ri-ca) Perform analysis to determine overlapping visual field areas and non-overlapping visual field areas; Concatenate images R in row direction i The pixel gray value v0 and the cropped image I 6(ri-ca) The pixel gray value v1 calculates the pixel gray value v in the overlapping field of view; Among them, k is the pixel column where the overlapping area pixels are located, and w3 is the row-wise stitched image R i and cropped image I 6(ri-ca) The width of the overlapping area; The image of the overlapping field of view is updated according to the pixel gray value v.

Images