Compound eye camera image fusion method

Abstract

The application discloses a kind of compound eye camera image fusion methods, comprising: the global camera image is interpolated and enlarged, and the global camera image after amplification is obtained;The global camera image after amplification is divided into regions, obtain multiple local areas, and the coordinate range of each local area is obtained;Multiple sub-camera images and corresponding local area are extracted and matched with image features, and the final matching point pair is obtained;Registration is carried out based on the homography matrix formed by the final matching point pair each sub-camera image and corresponding local area, and the restoration image composed of all registered sub-camera images is obtained;The restoration image is color corrected based on the fusion algorithm of gradient fitting, to fuse the restoration image into the global camera image after amplification, and the final fusion image is obtained.The application solves the problems of traditional multi-image splicing algorithm splicing process cumbersome, image picture stretch distortion, photo brightness and chroma inconsistency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image fusion, and particularly relates to a compound eye camera image fusion method. BACKGROUND

[0002] Image stitching technology refers to stitching two or more images with overlapping regions to form a high-resolution image with super-wide field of view. With the continuous development and progress of society and technology, people's demand for large field of view and high resolution images is increasing, and compound eye cameras are increasingly applied in various fields, such as traffic monitoring, medical imaging, military field and virtual reality field. The mainstream image stitching method can be roughly divided into three steps: feature extraction, image registration and image fusion. According to different processing ideas in different steps, image stitching can be divided into three directions: global alignment method, space variation drawing method and seam line leading method.

[0003] Global alignment method: align images with a homography matrix. This method requires no depth change in the overlapping region of the scene, and is suitable for scenes with no or very small parallax. Space variation drawing method: divide the image into a dense grid, and align each grid with a homography matrix. Grid optimization is used to solve the image stitching problem, which is equivalent to optimizing each local homography matrix independently. It has more degrees of freedom and is suitable for scenes with small parallax. Seam line leading method: take the seam line as the leading, and obtain the best seam line in the overlapping region of the images to be stitched according to the dynamic programming idea. This method does not need to strictly align the entire overlapping region, but only aligns the region near the seam line. This method can stitch large parallax scenes.

[0004] At the same time, since the sub-images used for stitching are obtained by multiple image sensors, and the positions and angles of the sub-cameras are different, the brightness and color of the obtained sub-images are different, and the influence of the brightness and color difference on the fusion effect of multiple sub-images needs to be considered. The existing common color difference and brightness correction mainly includes global correction and local correction. Global correction is to set a global correction coefficient for the whole image, and adjust the color and brightness of each part of the image according to the coefficient to obtain an image that is close in the whole. However, this method adjusts the color of all sub-images to the same degree, and the correction granularity is low. A global parameter cannot accurately describe the uneven color difference between sub-images, and is only suitable for cases where the color difference of sub-images is not obvious. When the color difference is large, color distortion is easily caused. Local correction is to divide the stitched image into different sub-blocks, and set a correction coefficient for each sub-block for local adjustment. This method can overcome the shortcomings of the global method, has high granularity, and can achieve good fusion effect in the overlapping region. However, it does not correct the non-overlapping region, resulting in unnatural transition between the overlapping region and the non-overlapping region. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention provides a compound eye camera image fusion method.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] A compound eye camera image fusion method, the compound eye camera image fusion method comprising:

[0008] Acquire a global camera image and multiple sub-camera images, wherein the corresponding regions of the sub-camera images and the global camera images have the same resolution.

[0009] The global camera image is interpolated and enlarged to obtain an enlarged global camera image;

[0010] The magnified global camera image is divided into regions to obtain multiple local regions corresponding to the sub-camera images, and the coordinate range of each local region is obtained.

[0011] Image features are extracted and matched between the local regions corresponding to the multiple sub-camera images and the magnified global camera image to obtain the final matching point pairs;

[0012] Based on the homography matrix formed by the final matching point pairs, each of the sub-camera images and the corresponding local regions are registered to obtain the registered sub-camera images. Then, the coordinates of the registered sub-camera images are restored to the coordinate system of the magnified global camera image to obtain the restored image composed of all the registered sub-camera images.

[0013] A gradient-fitting-based fusion algorithm is used to perform color correction on the restored image, so as to fuse the restored image into the magnified global camera image to obtain the final fused image.

[0014] Optionally, the global camera image is interpolated and enlarged to obtain an enlarged global camera image, including:

[0015] Based on the global camera image magnification factor k, the global camera image is interpolated and magnified using bilinear interpolation or bicubic interpolation to obtain the magnified global camera image, wherein the global camera image magnification factor k is... s g s is the resolution size of the global camera image. l The resolution of the sub-camera image in the corresponding region of the global camera image.

[0016] Optionally, the magnified global camera image is divided into regions to obtain multiple regions, and the coordinate range of each region is obtained, including:

[0017] Calculate the ratio R of the length and width of the sub-camera image to the length of the entire magnified global camera image. h The ratio of width to height R w ;

[0018] Based on the length ratio R h The width ratio R w The overlap rate between two adjacent sub-camera images is used to obtain the width-direction overlap region and the length-direction overlap region of the two adjacent sub-camera images;

[0019] According to the length ratio R h The width ratio R w The magnified global camera image is divided into multiple local regions corresponding to the sub-camera images by considering the overlapping regions in the width and length directions of two adjacent sub-camera images, and the coordinate range of each local region is obtained.

[0020] Optionally, image feature extraction and matching are performed on the local regions corresponding to the multiple sub-camera images and the magnified global camera image to obtain the final matching point pairs, including:

[0021] Image feature points of multiple local regions corresponding to each sub-camera image and the magnified global camera image are extracted using the Scale Invariant Feature Transform (SIFT) algorithm.

[0022] The K-nearest neighbor matching method is used to perform preliminary feature point matching between the image feature points corresponding to the extracted sub-camera image and the image feature points of the local region to obtain preliminary matching point pairs;

[0023] The preliminary matching point pairs are filtered using a grid motion statistics algorithm to retain correct matching points and remove incorrect matching points, so as to obtain the final matching point pairs.

[0024] Optionally, based on the homography matrix formed by the final matching point pairs, each of the sub-camera images and the corresponding local regions are registered to obtain registered sub-camera images. Then, the coordinates of the registered sub-camera images are restored to the coordinate system of the magnified global camera image to obtain a restored image composed of all the registered sub-camera images, including:

[0025] Based on the random sampling consistency algorithm, the homography matrix between the sub-camera image and the corresponding local region is calculated according to the coordinate correspondence of the final matching point pairs.

[0026] The sub-camera image is subjected to homography transformation according to the homography matrix to obtain the transformed sub-camera image, wherein the transformed sub-camera image and the plane containing the corresponding local region are in the same coordinate system;

[0027] Interpolation techniques are used to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image;

[0028] All the registered sub-camera images are reconstructed in coordinates, and the registered sub-camera images are mapped onto a canvas of the same size as the magnified global camera image, so that the coordinates of the registered sub-camera images are restored to the coordinate system of the magnified global camera image, so as to obtain a reconstructed image composed of all the registered sub-camera images.

[0029] Optionally, interpolation techniques are used to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image, including:

[0030] Back-mapping is used to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image.

[0031] Optionally, a gradient-fit-based fusion algorithm is used to perform color correction on the restored image to fuse the restored image into the magnified global camera image, resulting in a final fused image, including:

[0032] Based on the pixels at the four corners of the preprocessed region in the magnified global camera image, the pixels at the four corners of the preprocessed region in the restored image, and the pixels at the corresponding positions of the pixels to be fused in the restored image, the pixels to be fused in the first row, last row, first column, and last column of the preprocessed region in the magnified global camera image, excluding those located at the four corners, are fused to obtain the first fused pixel. In each preprocessed region, the width between the pixels at the two corners in the row and column directions is n, and the interval is n-2. Each preprocessed region is obtained by moving it in the magnified global camera image first from left to right and then from top to bottom, moving n-1 pixels each time.

[0033] The internal pixels of the preprocessed region in the magnified global camera image are fused based on the fused first pixel and the pixel in the restored image corresponding to the position of the fused first pixel to obtain the fused second pixel, thereby obtaining the final fused image.

[0034] Optionally, the formula for calculating the fused first pixel is:

[0035]

[0036]

[0037]

[0038]

[0039] Among them, G 1m G is the first pixel after fusion in the first row and m column of the preprocessed region in the magnified global camera image. m1 G is the first pixel after fusion in the m-th row and 1-th column of the preprocessed region in the magnified global camera image. nm G is the first pixel in the nth row and mth column of the preprocessed region in the magnified global camera image after fusion. mn S is the first pixel after fusion in the m-th row and n-th column of the preprocessed region in the magnified global camera image. 1m S is the pixel in the first row and m column of the preprocessed region in the restored image. m1 S is the pixel in the m-th row and 1-th column of the restored image corresponding to the preprocessed region. nm S is the pixel in the nth row and mth column of the preprocessed region in the restored image. mn G is the pixel in the m-th row and n-th column of the preprocessed region in the restored image. a G represents the original pixel in the first row and first column of the preprocessed region in the magnified global camera image. b G represents the original pixel in the first row and nth column of the preprocessed region in the magnified global camera image. c G represents the original pixel in the nth row and 1st column of the preprocessed region in the magnified global camera image. d S is the original pixel in the nth row and nth column of the preprocessed region in the magnified global camera image. a S is the pixel in the first row and first column of the restored image corresponding to the preprocessed region. b S is the pixel in the first row and nth column of the restored image corresponding to the preprocessed region. c S is the pixel in the nth row and 1st column of the restored image corresponding to the preprocessed region. d The pixel in the nth row and nth column of the restored image corresponding to the preprocessed region;

[0040] in, Here, t corresponds to S. 1m S m1 S nm or Smn A corresponds to G a G b G c or G d B corresponds to S a S b S c or S d .

[0041] Optionally, the internal pixels of the preprocessed region in the magnified global camera image are fused based on the fused first pixel and the pixel in the restored image corresponding to the position of the fused first pixel, to obtain the fused second pixel, including:

[0042] Based on a row-first, column-first, or a weighted combination of row-first and column-first methods, the internal pixels of the preprocessed region in the magnified global camera image are fused according to the fused first pixel and the pixel in the restored image corresponding to the position of the fused first pixel, so as to obtain the fused second pixel.

[0043] Optionally, for the row-first approach, the formula for calculating the fused second pixel is:

[0044]

[0045] For the column-first approach, the formula for calculating the merged second pixel is:

[0046]

[0047] Among them, G mx G is the second pixel in the m-th row and x-th column of the preprocessed region in the magnified global camera image after fusion. ym G is the fused second pixel in the y-th row and m-th column of the preprocessed region in the magnified global camera image. m1 G is the first pixel after fusion in the m-th row and 1-th column of the preprocessed region in the magnified global camera image. mn G is the first pixel in the m-th row and n-th column of the preprocessed region in the magnified global camera image after fusion. 1m G is the first pixel after fusion in the first row and m column of the preprocessed region in the magnified global camera image. nm S is the first pixel after fusion in the nth row and mth column of the preprocessed region in the magnified global camera image. mx S is the pixel in the m-th row and x-th column of the preprocessed region in the restored image. m1S is the pixel in the m-th row and 1-th column of the restored image corresponding to the preprocessed region. ym S is the pixel in the y-th row and m-th column of the preprocessed region in the restored image. mn S is the pixel in the m-th row and n-th column of the restored image corresponding to the preprocessed region. 1m S is the pixel in the first row and m column of the preprocessed region in the restored image. ym S is the pixel in the y-th row and m-th column of the preprocessed region in the restored image. nm Let x = 2, ..., n-1, y = 2, ..., n-1 in the restored image corresponding to the preprocessed region;

[0048] in, Here, t corresponds to S. mx or S ym A corresponds to G m1 G mn G 1m or G nm B corresponds to S m1 S mn S 1m or S nm .

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention first interpolates and enlarges the global camera image. The sub-camera image and the enlarged global camera image are then registered in local regions. After calculating the homography matrix, the pixels of the registered sub-camera image and the global camera image at the same coordinate point are matched one-to-one. Then, a gradient fitting fusion algorithm is used to calculate the pixel coordinate pixel value of the integer point in the global camera image and the corresponding pixel value in the sub-camera image, regenerating the pixel fusion value of that point and replacing the pixel value obtained by the interpolation algorithm in the corresponding global camera image. This achieves the adjustment of the brightness and color of the sub-camera image and its fusion into the global camera image, resulting in a high-resolution, consistent, and detail-rich wide-field panoramic image.

[0051] This invention uses images from a compound eye camera with a global camera and sub-camera architecture as source material. It employs image registration and pixel fusion to stitch images together. Sub-camera images with small overlapping areas and inconsistent shooting parameters are stitched together to create images with good consistency, high resolution, and rich detail. This solves the problems of cumbersome stitching process, image stretching and distortion, and inconsistent brightness and color of traditional multi-image stitching algorithms.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating an image fusion method using a compound eye camera provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the correspondence between a global camera image and a sub-camera image provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of a global camera image region division provided by an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of copying a registered sub-image into a panoramic canvas according to an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of an image that needs to be fused, provided by an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the corresponding pixels of a global camera image and a sub-camera image provided in an embodiment of the present invention;

[0059] Figure 7 This is a magnified schematic diagram of corresponding pixels in a global camera image and a sub-camera image provided in an embodiment of the present invention. Detailed Implementation

[0060] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0061] Example 1

[0062] To address the problems of cumbersome image stitching processes, difficult registration, and inconsistent brightness and color among sub-images in compound-eye cameras with a global camera and several local sub-cameras, this invention proposes an image fusion method for compound-eye cameras. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating an image fusion method for a compound eye camera provided in an embodiment of the present invention. The image fusion method for a compound eye camera includes:

[0063] Step 1: Acquire the global camera image and multiple sub-camera images. The corresponding areas of the sub-camera images and the global camera images have the same resolution.

[0064] Here, the global camera image is the global image obtained by the global camera, and the sub-camera image is the local image obtained by the sub-camera corresponding to each region in the global image.

[0065] Step 2: Interpolate and enlarge the global camera image to obtain the enlarged global camera image.

[0066] Specifically, based on the global camera image magnification factor k, the global camera image is interpolated and magnified using bilinear interpolation or bicubic interpolation to obtain the magnified global camera image. The global camera image magnification factor k is... s g s represents the resolution size of the global camera image. l This represents the resolution of the sub-camera image within the corresponding region of the global camera image.

[0067] In other words, firstly, based on the field of view contained in the sub-camera image, the location region of that field of view in the global camera image is found, such as... Figure 2 As shown, based on the resolution size s of the entire global camera image g Area resolution size s corresponding to the sub-camera image l ratio This yields the size ratio between the sub-camera image and the global camera image. Using k as the global camera image magnification factor, the height and width of the global camera image are then interpolated and magnified using bilinear interpolation or bicubic interpolation. Calculate the interpolated pixels between neighboring pixels in the panoramic image to guide the positioning of the sub-camera images.

[0068] Here, the principle of bilinear interpolation is as follows: using the four pixels around the point P to be interpolated, perform two linear interpolations in one direction, and then perform one linear interpolation in the other direction. The value obtained is then used as the gray value of point P.

[0069] Bicubic interpolation principle: As a further improvement of bilinear interpolation, this algorithm uses the gray values ​​of 16 points around the sampling point for cubic interpolation. It not only takes into account the influence of the gray values ​​of the four directly neighboring points, but also the influence of the gray value change rate between each neighboring point.

[0070] Step 3: Divide the magnified global camera image into regions, obtain multiple local regions corresponding to the sub-camera images, and obtain the coordinate range of each local region.

[0071] Since the positions of the sub-cameras and the global camera in the compound eye camera are fixed, the corresponding regions of the global camera and several sub-cameras contained within its field of view can be obtained in advance. Furthermore, the overlap ratio of two sub-camera images can be calculated based on the images of adjacent sub-cameras, which is approximately 10%-15%. The magnified global image can be divided into regions, and the divided local regions are not smaller than the size of the sub-camera images.

[0072] In one specific embodiment, step 3 may include:

[0073] Step 3.1: Calculate the ratio R of the length and width of the sub-camera image to the length of the entire magnified global camera image. h Ratio of width to height w .

[0074] Step 3.2, based on the length ratio R h Width ratio R w The overlap rate between two adjacent sub-camera images yields the width and length overlap regions of the two adjacent sub-camera images.

[0075] Here, the overlapping region W in the width direction overlap For W overlap =W*R w *q, overlapping region H in the length direction overlap For H overlap =H*R h *q, where W is the width of the sub-camera image, H is the length of the sub-camera image, and q is the overlap rate.

[0076] Step 3.3: Based on the length ratio R h Width ratio R w The magnified global camera image is divided into multiple local regions corresponding to the sub-camera images by identifying the overlapping regions in the width and length directions of the images and obtaining the coordinate range of each local region.

[0077] For example, see Figure 3 ,son Figure 1 The local region in the global camera image is (0, W*R). w ) and (0, H*R h ),son Figure 2 The local region in the global camera image is (W*R) h -W overlap ,2*W*R w -W overlap ) and (0, H*R h ),son Figure 3 The local region in the global camera image is (0, W*R). w ) and (H*R h -H overlap 2H*R h -H overlapThis process is repeated to divide the image into local regions of the global camera image and the coordinate range of each local region within the global image. Then, using several local regions of the global camera image as reference images and corresponding sub-camera images as input images, feature points are extracted and matched. This allows feature point extraction from the sub-camera images to be performed within a small local area of ​​the global camera image, significantly reducing the time required for feature point extraction and improving processing speed.

[0078] Step 4: Extract and match the image features of the local regions corresponding to the multiple sub-camera images and the magnified global camera image to obtain the final matching point pairs.

[0079] In one specific embodiment, step 4 may include:

[0080] Step 4.1: Extract image feature points of multiple local regions corresponding to each sub-camera image and the magnified global camera image using the Scale-invariant feature transform (SIFT) algorithm.

[0081] Step 4.2: Use the K-nearest neighbor matching method to perform preliminary feature point matching on the image feature points corresponding to the extracted sub-camera images and the image feature points of the local regions to obtain preliminary matching point pairs.

[0082] Here, feature point matching uses K-nearest neighbor matching. Specifically, a feature point is first selected from the sub-camera image, and the K (K=2) nearest feature points to this feature point are found in the corresponding local region of the global camera image. The ratio of the closest distance to the second closest distance between these two feature points is calculated based on the Euclidean distance. It is then determined whether this ratio is less than 0.5. If it is, the selected feature point from the sub-camera image is matched with the nearest feature point between the two feature points in the corresponding local region of the global camera image, and the matched feature point pair is added to the feature point matching pair set; otherwise, the feature point of the sub-image is discarded, and a new feature point is selected.

[0083] Step 4.3: Use grid-based motion statistics (GMS) to filter the initial matching point pairs, retaining correct matching points and filtering out incorrect matching points to obtain the final matching point pairs.

[0084] Here, the GMS algorithm is a fast and ultra-robust feature matching algorithm based on the statistical properties of grid motion. The selection of inliers is mainly based on the assumption that each pixel in the matching image has motion smoothness, and there will be several correct matching points next to the correct matching point as support; while there are almost no matching points with the same error around an incorrect matching point (assuming that the incorrect matching occurs randomly).

[0085] Step 5: Based on the homography matrix formed by the final matching point pairs, register each sub-camera image and its corresponding local region to obtain the registered sub-camera image. Then, restore the coordinates of the registered sub-camera image to the coordinate system of the magnified global camera image to obtain the restored image composed of all the registered sub-camera images.

[0086] In one specific embodiment, step 5 may include:

[0087] Step 5.1: Based on the Random Sample Consensus Algorithm (RANSAC), calculate the homography matrix between the sub-camera image and the corresponding local region according to the coordinate correspondence of the final matching point pairs.

[0088] Specifically, by using the coordinate correspondence of the matching points, the coordinate equations are solved to calculate the image transformation model, i.e., the homography matrix H, between the sub-image and the corresponding local region of the global image.

[0089] Homography (projective transformation) is a spatial mapping relationship between pixels in one image and pixels in another image. It can be achieved by solving the homography matrix to find matching points (x, y) in the sub-image. i y i ) mapped to the corresponding point (x) in a local region of the global camera image i ',y i The position of '). The homography matrix H is represented as:

[0090]

[0091] Among them, A 2×2 T represents the affine transformation parameters. 2×1 V represents the translation transformation parameter. T =[v1,v2] represents the intersection relationship of edge points after transformation, which is generally normalized to make s=1. The correspondence between matching points between images is as follows:

[0092]

[0093] The coordinate system can be converted to homogeneous coordinates, and the above formula can be expressed as:

[0094]

[0095]

[0096] Further transformation:

[0097] (h 31 x i +h 32 y i +h 33 )·x i ′=h 11 x i +h 12 y i +h 13

[0098] (h 31 x i +h 32 y i +h 33 )·y i ′=h 21 x i +h 22 y i +h 23

[0099] Convert to the form Ax = 0:

[0100]

[0101] In the specific calculation of the RANSAC algorithm, the minimum number of randomly selected target points are input into the algorithm each time to construct the model as the point set. For the homography matrix transformation model H, calculating 8 parameters requires 4 pairs of matching points. The specific steps are as follows:

[0102] i. First, randomly select 4 non-collinear matching point pairs from the matching point pairs obtained in step 4 as the assumed subset of interior points, and solve the preliminary transformation model H using the system of equations based on the transformation relationship;

[0103] ii. Calculate the projection error from all matching points to model H, set a projection error threshold E, and consider points with errors within the threshold range as "interior points" suitable for this model, denoted as the interior point set I. Count the number of feature pairs for the current I; the projection error is:

[0104]

[0105] iii. When the number of points in I is greater than the current optimal set I best When counting the points in the data, update I. best = I, update iteration count k = k + 1;

[0106] iv. If the number of iterations k equals the preset maximum number of iterations K, denote the final set of interior points as I. bestThe transformation model H is recalculated using this set of points, resulting in the final mathematical model H. The maximum number of iterations K is preset to be:

[0107]

[0108] Where p is the confidence level (usually taken as 0.995), w is the proportion of inliers, and m is the minimum number of samples required for calculation, which is 4 here.

[0109] Step 5.2: Perform homography transformation on the sub-camera image according to the homography matrix to obtain the transformed sub-camera image, wherein the transformed sub-camera image and the plane containing the corresponding local region are in the same coordinate system.

[0110] Step 5.3: Use interpolation techniques to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image.

[0111] Here, image registration refers to mapping the pixel coordinates of the sub-camera image to the pixel coordinates of the corresponding local region of the global camera image. Using the parameters of the homography matrix H obtained in the above steps, the camera image is homography-transformed so that the transformed sub-camera image and the corresponding local region are in the same coordinate system. Since the sub-camera image undergoes homography transformation, the pixel coordinates (x′, y′) of the output image after transformation at point (x, y) in the sub-image may not be integers. Therefore, interpolation techniques are needed to obtain the transformed pixel values. This is usually divided into forward mapping and backward mapping. Because forward mapping can easily lead to some areas in the output image being uncovered or covered multiple times, resulting in discontinuities, backward mapping is used for image transformation here. Specifically:

[0112] Back-mapping is used to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image.

[0113] Here, backward mapping is also called pixel fill mapping. Given the position (x,y) of an integer point (x′,y′) in the output image and its position (x,y) in the input image (usually a non-integer point), the pixel value of that point is calculated by interpolating the pixel values ​​of the input image at the surrounding integer point positions.

[0114] Step 5.4: Restore the coordinates of all registered sub-camera images. Map the registered sub-camera images onto a canvas of the same size as the magnified global camera image, so that the coordinates of the registered sub-camera images are restored to the coordinate system of the magnified global camera image, thus obtaining the restored image composed of all registered sub-camera images.

[0115] Specifically, in step 3, to improve the speed of feature point extraction and matching between the sub-camera image and the magnified global camera image, the global camera image is divided into regions. At this time, the sub-camera image is matched with a local region of the global camera image. After registration, the coordinate system is also the coordinate system of the local region image, not the coordinate system of the final panoramic image canvas. Therefore, it is necessary to restore the coordinates of the registered sub-camera image.

[0116] Please see Figure 4 For each registered sub-camera image, a panoramic canvas with the same resolution and size as the magnified global camera image is first created, with pixel values ​​filled to 0. Then, using the coordinate range of the local region obtained in step 3, the image pixel information of the registered sub-camera image is copied back to the pixels within that coordinate range in the panoramic canvas. The result is the restored image composed of all the registered sub-camera images.

[0117] Step 6: Perform color correction on the restored image using a gradient fitting-based fusion algorithm, so as to fuse the restored image into the magnified global camera image to obtain the final fused image.

[0118] Here, when multiple cameras capture images simultaneously, due to factors such as the hardware of the individual sub-cameras and the shooting environment (e.g., lighting, shooting angle), images from different sub-cameras within the same area will differ in brightness, color, and other aspects. If directly fused onto the global camera image, obvious seams will appear between the overlapping and non-overlapping areas of the panoramic camera image, and the brightness and color differences between the corresponding sub-camera images and the global camera image will worsen the fusion effect. Therefore, it is necessary to address the problem of uneven brightness and color in the multi-camera sub-camera images.

[0119] Based on the brightness and chromaticity of the global camera image, the RGB values ​​of the fusion points of the sub-camera images are adjusted to achieve consistent chromaticity and brightness. Figure 5 In the diagram, the red dots represent the original pixels of the global image, while the black dots represent the sub-pixel points after linear interpolation, which are also the points where the sub-camera images will be fused. For color images, fusion processing is performed separately in the R, G, and B domains. For black and white images, processing can be done in the grayscale domain.

[0120] The core idea of ​​fusion is to compare the pixels of the registered sub-camera image with the global camera image, and adjust the pixel value of the sub-camera image to replace the pixel value of the global camera image obtained by interpolation at that point based on the relationship between corresponding points of the sub-camera image and the global camera image.

[0121] Step 5 yields all sub-camera images that have been registered with the global camera image, thus realizing pixel mapping of the sub-camera images. The image content in the registered sub-camera images corresponds to the corresponding position in the global camera image, that is, the pixels at the same coordinate point correspond one-to-one. Subsequently, the pixel values ​​of the sub-images can be adjusted to replace the corresponding pixels in the global image through a gradient fitting fusion algorithm.

[0122] like Figure 6 As shown, G represents the magnified global camera image, S represents the reconstructed image composed of the registered sub-camera images, and the four points G... x (x = a, b, c, d) represents the original pixels of the preprocessed region in the magnified global camera image, S x (x = a, b, c, d) represents four pixels in the preprocessed region of the reconstructed image. For example... Figure 7 As shown, gradient calculations are performed using the pixel values ​​in two four-point pixel matrices corresponding to the magnified global camera image and the restored image to adjust the pixel values ​​of the sub-image. The preprocessed region is a pixel matrix, where... Figure 7 G in a G b G c G d The region consists of a pixel matrix. Figure 7 S in a S b S c S d The resulting region is also a pixel matrix. The width of any two adjacent vertex pixels in the pixel matrix is ​​n, and the interval is n-2. The number of intervals can be determined by the magnification factor of the global image. Calculated, i.e. After calculating a pixel matrix, the calculation is performed by moving the pixels from left to right, then from top to bottom, moving n-1 pixels each time. For example... Figure 7 As shown, G ij (i,j=1,..,n) represent pixels generated by interpolation. Point G in the magnified global camera image. x (x=a,b,c,d) and S in the restored image x (x = a, b, c, d) and each pixel G in between. ij (i,j=1,..,n) and S ij The pixels (i, j = 1, ..., n) are in a one-to-one correspondence and all lie on the same straight line. The number of pixels that need to be regenerated in the magnified global camera image can be calculated using the following formula.

[0123] In one specific embodiment, step 6 may include:

[0124] Step 6.1: Based on the pixels at the four corners of the preprocessed region in the magnified global camera image, the pixels at the four corners of the corresponding preprocessed region in the restored image, and the pixels at the corresponding positions of the pixels to be fused in the restored image, fuse the pixels to be fused in the first row, last row, first column, and last column of the preprocessed region in the magnified global camera image, excluding those located at the four corners, to obtain the first fused pixel. The width between the two corner pixels in each preprocessed region in the row and column directions is n, and the interval is n-2. Each preprocessed region is obtained by moving it from left to right and then from top to bottom in the magnified global camera image, moving n-1 pixels each time.

[0125] Here, the pixels at the four corners of the preprocessed region in the magnified global camera image are G. x (x = a, b, c, d), the pixels at the four corners of the preprocessed region in the restored image are S. x (x = a, b, c, d).

[0126] In this embodiment, there are four Gs in the global camera image G. x The rows and columns are merged using the following formula:

[0127]

[0128]

[0129]

[0130]

[0131] Among them, G 1m G represents the first pixel in the first row and m column of the preprocessed region in the magnified global camera image after fusion. m1 G represents the first pixel in the fused matrix of the m-th row and 1-th column of the preprocessed region in the magnified global camera image. nm G represents the first pixel in the nth row and mth column of the preprocessed region in the magnified global camera image after fusion. mn S is the first pixel in the m-th row and n-th column of the preprocessed region in the magnified global camera image after fusion. 1m To restore the pixel in the 1st row and mth column corresponding to the preprocessed region in the image, S m1 To restore the pixel in the m-th row and 1-th column corresponding to the preprocessed region in the image, S nm To restore the pixel in the nth row and mth column of the preprocessed region in the image, S mn To restore the pixel in the m-th row and n-th column corresponding to the preprocessed region in the image, G aG represents the original pixels in the first row and first column of the preprocessed region in the magnified global camera image. b G represents the original pixel in the 1st row and nth column of the preprocessed region in the magnified global camera image. c G represents the original pixel in the nth row and 1st column of the preprocessed region in the magnified global camera image. d S represents the original pixel in the nth row and nth column of the preprocessed region in the magnified global camera image. a To restore the pixels in the first row and first column of the preprocessed region in the image, S b To restore the pixel in the 1st row and nth column corresponding to the preprocessed region in the image, S c To restore the pixel in the nth row and 1st column corresponding to the preprocessed region in the image, S d To restore the pixel in the nth row and nth column of the preprocessed region in the image;

[0132] in, Here, t corresponds to S. 1m S m1 S nm or S mn A corresponds to G a G b G c or G d B corresponds to S a S b S c or S d .

[0133] Step 6.2: Based on the first pixel after fusion and the pixel at the corresponding position of the first pixel after fusion in the restored image, fuse the internal pixels of the preprocessed area in the magnified global camera image to obtain the second pixel after fusion. After obtaining all the fused pixels, the final fused image is obtained.

[0134] Specifically, based on row-first, column-first, or a weighted combination of row-first and column-first methods, the internal pixels of the preprocessed area in the magnified global camera image are fused according to the first pixel after fusion and the corresponding pixels in the restored image to obtain the second pixel after fusion.

[0135] For the row-first approach, the formula for calculating the second pixel after fusion is:

[0136]

[0137] For the column-first approach, the formula for calculating the second pixel after fusion is:

[0138]

[0139] Among them, G mx For the second pixel in the m-th row and x-th column of the preprocessed region in the magnified global camera image, G is the fused second pixel. ym For the fused second pixel in the y-th row and m-th column of the preprocessed region in the magnified global camera image, G m1 G represents the first pixel in the fused matrix of the m-th row and 1-th column of the preprocessed region in the magnified global camera image. mn G represents the first pixel in the m-th row and n-th column of the preprocessed region in the magnified global camera image after fusion. 1m G represents the first pixel in the first row and m column of the preprocessed region in the magnified global camera image after fusion. nm S represents the first pixel in the nth row and mth column of the preprocessed region in the magnified global camera image after fusion. mx To restore the pixel in the m-th row and x-th column corresponding to the preprocessed region in the image, S m1 To restore the pixel in the m-th row and 1-th column corresponding to the preprocessed region in the image, S ym To restore the pixel in the y-th row and m-th column of the preprocessed region in the image, S mn To restore the pixel in the m-th row and n-th column corresponding to the preprocessed region in the image, S 1m To restore the pixel in the 1st row and mth column corresponding to the preprocessed region in the image, S ym To restore the pixel in the y-th row and m-th column of the preprocessed region in the image, S nm To restore the pixel in the nth row and mth column of the preprocessed region in the image, x = 2, ..., n-1, y = 2, ..., n-1;

[0140] in, Here, t corresponds to S. mx or S ym A corresponds to G m1 G mn G 1m or G nm B corresponds to S m1 S mn S 1m or S nm .

[0141] In addition, the weighting method for row priority and column priority can be, for example, by adding the result obtained by row priority and the result obtained by column priority and dividing by 2. Furthermore, those skilled in the art can also set other weighting methods according to actual needs, without making specific limitations.

[0142] (1) The problem of small overlap of sub-camera fields of view and uneven distribution of feature points.

[0143] Because the overlap between sub-camera images is small, the extracted feature points are unevenly distributed, and a single image transformation model has poor transition between overlapping and non-overlapping areas, leading to errors in the subsequent registration process and poor stitching results. However, in the fusion method provided by this invention, the overlap between each sub-camera image and the global camera image is greatly increased, solving the problem of the overlap ratio affecting image registration. This results in a larger number of extracted feature points with a more uniform distribution, enabling more effective and accurate registration.

[0144] (2) For the geometric splicing problem.

[0145] The system uses compound-eye cameras to meet the requirements of high resolution and large field of view. Due to the non-overlapping optical paths of compound-eye cameras, if the sub-camera images are stitched together sequentially, the accumulation of stitching errors will inevitably occur, causing the stitched sub-camera images to be overly stretched and the geometric shape of the images to be distorted. This results in obvious distortion and stretching of the stitched panoramic image. The method provided by this invention completes the stitching by registering and fusing the sub-camera images into the global camera image, so that the mapping plane of all sub-camera images is on the plane of the global camera image, which effectively solves the problem of geometric distortion caused by the accumulation of stitching errors.

[0146] (3) Regarding the issue of brightness and color consistency in sub-camera images.

[0147] Images captured by each sub-camera may exhibit variations in chroma and brightness due to differences in imaging angle and environment, leading to inconsistencies in color and brightness in the stitched image. Correcting solely using sub-camera images is insufficient to achieve consistent chroma and brightness correction across the images. Since the overall image from the global camera maintains color uniformity, this invention utilizes a matching region for each sub-camera as a standard for corresponding color and brightness correction, achieving consistency between the stitched images.

[0148] This invention differs from traditional direct registration and stitching of individual sub-camera images when processing multi-image stitching. Instead, it utilizes a compound-eye camera structure consisting of a global camera and sub-cameras. Leveraging the global camera image's overall field of view, registration and fusion are performed between local regions of each sub-camera image and the global camera image. This avoids the problem of small overlap areas and difficulty in registration when stitching sub-camera images together, and ensures the stitching result has no significant stretching distortion. While maintaining good stitching quality, it reduces the cumbersome process of stitching multiple sub-camera images and lowers the algorithm implementation difficulty. Furthermore, when dealing with color and brightness correction of sub-camera images, it directly uses the color and brightness of the global camera image as the standard, designing a gradient-fitting-based fusion algorithm to adjust the pixel values ​​of the sub-images, achieving consistent color and avoiding the problem of inconsistent standard image quality during color correction of multiple sub-images.

[0149] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0150] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0151] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0152] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for image fusion using a compound eye camera, characterized in that, The compound eye camera image fusion method includes: Acquire a global camera image and multiple sub-camera images, wherein the corresponding regions of the sub-camera images and the global camera images have the same resolution. The global camera image is interpolated and enlarged to obtain an enlarged global camera image; The magnified global camera image is divided into regions to obtain multiple local regions corresponding to the sub-camera images, and the coordinate range of each local region is obtained. Image features are extracted and matched between the local regions corresponding to the multiple sub-camera images and the magnified global camera image to obtain the final matching point pairs; Based on the homography matrix formed by the final matching point pairs, each of the sub-camera images and the corresponding local regions are registered to obtain the registered sub-camera images. Then, the coordinates of the registered sub-camera images are restored to the coordinate system of the magnified global camera image to obtain the restored image composed of all the registered sub-camera images. This includes: based on the random sampling consensus algorithm, calculating the homography matrix between the sub-camera images and the corresponding local regions according to the coordinate correspondence of the final matching point pairs; The sub-camera image is subjected to homography transformation according to the homography matrix to obtain the transformed sub-camera image, wherein the transformed sub-camera image and the plane containing the corresponding local region are in the same coordinate system; Interpolation techniques are used to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image; All the registered sub-camera images are reconstructed in coordinates, and the registered sub-camera images are mapped onto a canvas of the same size as the magnified global camera image, so that the coordinates of the registered sub-camera images are restored to the coordinate system of the magnified global camera image, so as to obtain a reconstructed image composed of all the registered sub-camera images. A gradient-fitting-based fusion algorithm is used to perform color correction on the restored image, so as to fuse the restored image into the magnified global camera image to obtain the final fused image.

2. The compound eye camera image fusion method according to claim 1, characterized in that, The global camera image is interpolated and enlarged to obtain an enlarged global camera image, including: Based on the global camera image magnification factor k, the global camera image is interpolated and magnified using bilinear interpolation or bicubic interpolation to obtain the magnified global camera image, wherein the global camera image magnification factor k is... s g s is the resolution size of the global camera image. l The resolution of the sub-camera image in the corresponding region of the global camera image.

3. The compound eye camera image fusion method according to claim 1, characterized in that, The magnified global camera image is divided into regions to obtain multiple regions, and the coordinate range of each region is obtained, including: Calculate the ratio R of the length and width of the sub-camera image to the length of the entire magnified global camera image. h Ratio of width to height w ; Based on the length ratio R h The width ratio R w The overlap rate between two adjacent sub-camera images is used to obtain the width-direction overlap region and the length-direction overlap region of the two adjacent sub-camera images; According to the length ratio R h The width ratio R w The magnified global camera image is divided into multiple local regions corresponding to the sub-camera images by considering the overlapping regions in the width and length directions of two adjacent sub-camera images, and the coordinate range of each local region is obtained.

4. The compound eye camera image fusion method according to claim 1, characterized in that, Image feature extraction and matching are performed on the local regions corresponding to the multiple sub-camera images and the magnified global camera image to obtain the final matching point pairs, including: Image feature points of multiple local regions corresponding to each sub-camera image and the magnified global camera image are extracted using the Scale Invariant Feature Transform (SIFT) algorithm. The K-nearest neighbor matching method is used to perform preliminary feature point matching between the image feature points corresponding to the extracted sub-camera image and the image feature points of the local region to obtain preliminary matching point pairs; The preliminary matching point pairs are filtered using a grid motion statistics algorithm to retain correct matching points and remove incorrect matching points, so as to obtain the final matching point pairs.

5. The compound eye camera image fusion method according to claim 1, characterized in that, The non-integer coordinates in the transformed sub-camera image are transformed into integer coordinates using interpolation techniques to obtain the registered sub-camera image, including: Back-mapping is used to transform the non-integer coordinates in the transformed sub-camera image into integer coordinates to obtain the registered sub-camera image.

6. The compound eye camera image fusion method according to claim 1, characterized in that, A gradient-fit-based fusion algorithm performs color correction on the restored image to fuse it into the magnified global camera image, resulting in a final fused image, including: Based on the pixels at the four corners of the preprocessed region in the magnified global camera image, the pixels at the four corners of the preprocessed region in the restored image, and the pixels at the corresponding positions of the pixels to be fused in the restored image, the pixels to be fused in the first row, last row, first column, and last column of the preprocessed region in the magnified global camera image, excluding those located at the four corners, are fused to obtain the first fused pixel. In each preprocessed region, the width between the pixels at the two corners in the row and column directions is n, and the interval is n-2. Each preprocessed region is obtained by moving it in the magnified global camera image first from left to right and then from top to bottom, moving n-1 pixels each time. The internal pixels of the preprocessed region in the magnified global camera image are fused based on the fused first pixel and the pixel in the restored image corresponding to the position of the fused first pixel to obtain the fused second pixel, thereby obtaining the final fused image.

7. The compound eye camera image fusion method according to claim 6, characterized in that, The formula for calculating the first pixel after fusion is: Among them, G 1m G is the first pixel after fusion in the first row and m column of the preprocessed region in the magnified global camera image. m1 G is the first pixel after fusion in the m-th row and 1-th column of the preprocessed region in the magnified global camera image. nm G is the first pixel in the nth row and mth column of the preprocessed region in the magnified global camera image after fusion. mn S is the first pixel after fusion in the m-th row and n-th column of the preprocessed region in the magnified global camera image. 1m S is the pixel in the first row and m column of the preprocessed region in the restored image. m1 S is the pixel in the m-th row and 1-th column of the restored image corresponding to the preprocessed region. nm S is the pixel in the nth row and mth column of the preprocessed region in the restored image. mn G is the pixel in the m-th row and n-th column of the preprocessed region in the restored image. a G represents the original pixel in the first row and first column of the preprocessed region in the magnified global camera image. b G represents the original pixel in the first row and nth column of the preprocessed region in the magnified global camera image. c G represents the original pixel in the nth row and 1st column of the preprocessed region in the magnified global camera image. d S is the original pixel in the nth row and nth column of the preprocessed region in the magnified global camera image. a S is the pixel in the first row and first column of the restored image corresponding to the preprocessed region. b S is the pixel in the first row and nth column of the restored image corresponding to the preprocessed region. c S is the pixel in the nth row and 1st column of the restored image corresponding to the preprocessed region. d The pixel in the nth row and nth column of the restored image corresponding to the preprocessed region; in, Here, t corresponds to S. 1m S m1 S nm or S mn A corresponds to G a G b G c or G d B corresponds to S a S b S c or S d .

8. The compound eye camera image fusion method according to claim 6, characterized in that, Based on the fused first pixel and the pixel in the restored image corresponding to the position of the fused first pixel, the internal pixels of the preprocessed region in the magnified global camera image are fused to obtain the fused second pixel, including: Based on a row-first, column-first, or a weighted combination of row-first and column-first methods, the internal pixels of the preprocessed region in the magnified global camera image are fused according to the fused first pixel and the pixel in the restored image corresponding to the position of the fused first pixel, so as to obtain the fused second pixel.

9. The compound eye camera image fusion method according to claim 8, characterized in that, For the row-first method, the formula for calculating the merged second pixel is: For the column-first approach, the formula for calculating the merged second pixel is: Among them, G mx G is the second pixel in the m-th row and x-th column of the preprocessed region in the magnified global camera image after fusion. ym G is the fused second pixel in the y-th row and m-th column of the preprocessed region in the magnified global camera image. m1 G is the first pixel after fusion in the m-th row and 1-th column of the preprocessed region in the magnified global camera image. mn G is the first pixel in the m-th row and n-th column of the preprocessed region in the magnified global camera image after fusion. 1m G is the first pixel after fusion in the first row and m column of the preprocessed region in the magnified global camera image. nm S is the first pixel in the nth row and mth column of the preprocessed region in the magnified global camera image after fusion. mx S is the pixel in the m-th row and x-th column of the preprocessed region in the restored image. m1 S is the pixel in the m-th row and 1-th column of the restored image corresponding to the preprocessed region. ym S is the pixel in the y-th row and m-th column of the preprocessed region in the restored image. mn S is the pixel in the m-th row and n-th column of the restored image corresponding to the preprocessed region. 1m S is the pixel in the first row and m column of the preprocessed region in the restored image. ym S is the pixel in the y-th row and m-th column of the preprocessed region in the restored image. nm Let x = 2, ..., n-1, y = 2, ..., n-1 in the restored image corresponding to the preprocessed region; in, Here, t corresponds to S. mx or S ym A corresponds to G m1 G mn G 1m or G nm B corresponds to S m1 S mn S 1m or S nm .

Images