A method and system for binocular camera image fusion based on homography transformation matrix

Abstract

The application discloses a binocular camera image fusion method and system based on homography transformation matrix, and relates to the field of intelligent traffic management.The method comprises the following steps: acquiring a homography transformation matrix corresponding to a long-range camera and a short-range camera according to a homography transformation matrix corresponding to each grid in a checkerboard image in a long-range image and a short-range image; and fusing a long-range image collected by the long-range camera and a short-range image collected by the short-range camera according to the homography transformation matrix.The method realizes the calculation of a local homography transformation matrix, and fuses the long-range image collected by the long-range camera and the short-range image collected by the short-range camera according to the homography transformation matrix, so that the binocular image fusion is not required to be performed by the method of multiple iterations for an inlier point through a RANSAC algorithm, the calculation amount and complexity of the binocular fusion process are reduced, and the real-time performance is high.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic management, and in particular to a binocular camera image fusion method and system based on homography transformation matrix. Background Technology

[0002] With the increasing number of cars in cities, road conditions are becoming increasingly complex, especially at intersections where vehicles, non-motorized vehicles, and pedestrians converge. Therefore, a dual-camera setup is often used to track and detect vehicles at multiple intersections. However, using a single camera with a deep learning target detection algorithm that is only suitable for detecting large, near targets often results in missed detections and camera frame flickering at long distances. Therefore, a second camera is needed to magnify distant targets, making the deep learning detection model suitable for large, near targets more effective, thus improving the camera's target detection rate and stability. Therefore, how to fuse binocular cameras is a key issue.

[0003] Currently, when performing binocular camera fusion, the Harris algorithm is first used with a sliding window to detect pixels with large grayscale value changes and extract feature corner points. To determine the spatial transformation model of the two images, four feature corner points in the overlapping area of ​​the two images are selected. Finally, the RANSAC algorithm is used iteratively multiple times to find the optimal multiple matching points and calculate the optimal homography transformation matrix, thereby obtaining the matching relationship between the two images and completing the fusion. However, the Harris algorithm requires calculating the grayscale changes after the window moves in any direction, and the RANSAC algorithm requires multiple iterations to calculate the homography transformation matrix and repeatedly calculate the error of other feature points until a feature point set that meets the specified number of points is found, resulting in a large computational load. Furthermore, the Harris algorithm requires setting the sliding window length and corner point threshold, and the parameters need to be adjusted and determined based on the detection results; the RANSAC algorithm requires setting the error threshold and inlier number threshold, and the parameters need to be adjusted and determined based on the detection results, making the algorithm more complex. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a binocular camera image fusion method and system based on homography transformation matrix, which can solve the problems of large computational load and complex implementation process in existing binocular camera image fusion processes.

[0005] To achieve the above objectives, on the one hand, the present invention provides a binocular camera image fusion method based on homography transformation matrix, the method comprising:

[0006] Acquire a distant image captured by a distant camera containing a checkerboard pattern and a close-up image captured by a close-up camera, wherein the checkerboard pattern is arranged on the road surface captured by the distant camera and the close-up camera.

[0007] The corner points of each quadrilateral in the checkerboard image are extracted from the distant and close-up images, and the homography transformation matrix corresponding to each grid is obtained based on the corner points of each quadrilateral.

[0008] The homography transformation matrices corresponding to the distant camera and the close-up camera are obtained based on the homography transformation matrices corresponding to each grid.

[0009] The distant image captured by the distant camera and the close-up image captured by the close-up camera are fused according to the homography transformation matrix.

[0010] Further, the step of extracting the corner points of each quadrilateral in the checkerboard image from the distant and close-up images, and obtaining the homography transformation matrix corresponding to each grid based on the corner points of each quadrilateral, includes:

[0011] Based on the corner pixel coordinates of each quadrilateral grid in the distant and close-up images, the homography transformation matrix corresponding to each grid is obtained.

[0012] Further, the step of obtaining the homography transformation matrices corresponding to the distant camera and the close-up camera based on the homography transformation matrices corresponding to each grid includes:

[0013] The sum of the homography transformation matrices corresponding to each grid is used as the homography transformation matrix corresponding to the distant camera and the close-up camera.

[0014] Further, the step of fusing the distant image captured by the distant camera and the close-up image captured by the close-up camera according to the homography transformation matrix includes:

[0015] Based on the current pixel of the close-up image and the homography transformation matrix, the pixel coordinates of the close-up image pixel projected onto the distant plane are obtained.

[0016] If the projected pixel coordinates are within the size of the preset mapped image and are integers, then the projected pixel coordinates are assigned to the current pixel of the preset mapped image. The preset mapped image is an empty image and is on the same plane as the foreground image.

[0017] If the coordinates of the projected pixel are within the size of the preset mapped image and are not all integers, then the projected pixel value is obtained by interpolation with the values ​​of nearby pixels, and this corrected projected pixel value is used as the current pixel value of the preset mapped image.

[0018] Furthermore, the step of acquiring the distant image captured by the distant camera and the close-up image captured by the close-up camera, which contain the checkerboard pattern image, includes:

[0019] Median filtering was applied to the distant view image captured by the distant camera and the close-up image captured by the close-up camera to obtain the filtered distant view image, the distant view image filtering error, the filtered close-up image, and the close-up image filtering error.

[0020] Further, the step of obtaining the homography transformation matrices corresponding to the distant camera and the close-up camera based on the homography transformation matrices corresponding to each grid includes:

[0021] Based on the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error, the error value of each grid center pixel in the chessboard image is obtained;

[0022] Based on the homography transformation matrix corresponding to each grid and the error value of the center pixel of each grid, the homography transformation matrix corresponding to the distant camera and the close-up camera is obtained.

[0023] On the other hand, the present invention provides a binocular camera image fusion system based on homography transformation matrix. The system includes: an acquisition unit, used to acquire a distant image captured by a distant camera and a close-up image captured by a close-up camera, the checkerboard being arranged on the road surface captured by the distant camera and the close-up camera.

[0024] The acquisition unit is further configured to extract the corner points of each quadrilateral in the checkerboard image from the distant and close-up images, and to obtain the homography transformation matrix corresponding to each grid based on the corner points of each quadrilateral.

[0025] The acquisition unit is further configured to acquire the homography transformation matrix corresponding to the distant camera and the close-up camera based on the homography transformation matrix corresponding to each grid.

[0026] The fusion unit is used to fuse the distant image captured by the distant camera and the close-up image captured by the close-up camera according to the homography transformation matrix.

[0027] Furthermore, the acquisition unit is specifically used to acquire the homography transformation matrix corresponding to each grid according to the corner pixel coordinates of each grid quadrilateral in the distant and close-up images.

[0028] Furthermore, the acquisition unit is specifically used to take the sum of the homography transformation matrices corresponding to each grid as the homography transformation matrices corresponding to the distant camera and the close-up camera.

[0029] Further, the fusion unit is specifically used to obtain the pixel coordinates of the projected pixel of the near-field image onto the distant plane based on the current pixel of the near-field image and the homography transformation matrix; if the projected pixel coordinates are within the size of the preset mapped image and are integers, then the projected pixel coordinates are assigned to the current pixel of the preset mapped image, wherein the preset mapped image is an empty image and is on the same plane as the near-field image; if the projected pixel coordinates are within the size of the preset mapped image and are not all integers, then the corrected projected pixel value is obtained by interpolating the projected pixel value and the values ​​of nearby pixels, and is used as the current pixel value of the preset mapped image.

[0030] Furthermore, the acquisition unit is specifically used to perform median filtering on the distant image acquired by the distant camera and the close-up image acquired by the close-up camera, respectively, to obtain the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error.

[0031] Furthermore, the acquisition unit is specifically used to acquire the error value of each grid center pixel in the chessboard image based on the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error.

[0032] Based on the homography transformation matrix corresponding to each grid and the error value of the center pixel of each grid, the homography transformation matrix corresponding to the distant camera and the close-up camera is obtained.

[0033] This invention provides a binocular camera image fusion method and system based on homography transformation matrices. The method obtains the homography transformation matrices corresponding to the distant and near cameras based on the homography transformation matrices corresponding to each grid in the checkerboard image of the distant and near images. Then, it fuses the distant image acquired by the distant camera and the near image acquired by the near camera according to the homography transformation matrices. This method achieves the fusion of the distant and near images by calculating local homography transformation matrices and fusing them accordingly. It eliminates the need for multiple iterations of the RANSAC algorithm to find interior points during binocular image fusion, reducing the computational load and complexity of the binocular fusion process, and also offers strong real-time performance. Attached Figure Description

[0034] Figure 1 This is a flowchart of a binocular camera image fusion method based on homography transformation matrix provided by the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of a binocular camera image fusion system based on homography transformation matrix provided by the present invention. Detailed Implementation

[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0037] like Figure 1 As shown in the figure, an embodiment of the present invention provides a binocular camera image fusion method based on homography transformation matrix, which includes the following steps:

[0038] 101. Acquire distant images containing a checkerboard pattern from a distant camera and close-up images from a close-up camera.

[0039] The checkerboard pattern is configured on the road surface captured by the distant camera and the close-up camera. The number of horizontal grids of the checkerboard pattern is Nw, and the number of vertical grids is Nh. After the homography transformation matrix is ​​calculated, the checkerboard pattern can be removed. The matrix is ​​then used to perform dual-image fusion of the normal scene, ensuring that the checkerboard pattern is present in both the distant camera scene and the close-up camera scene.

[0040] In this embodiment of the invention, in order to further improve the accuracy of binocular camera image fusion, step 101 may specifically include: performing median filtering on the distant image acquired by the distant camera and the close-up image acquired by the close-up camera respectively, to obtain the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error.

[0041] For example, the images are first preprocessed using median filtering. Two images to be fused are obtained: a_far (far image) and a_near (near image). Both images were taken with the same camera and have the same dimensions: width * height = W * H, meaning the image is W pixels wide and H pixels high. Median filtering is then applied to each image separately, resulting in filtered images A_far and A_near, and filtering errors error_far and error_near. The processing steps for the distant and near images are the same. Given that the distant image a_far has W * H pixels, median filtering is first applied row-by-row to the distant image, using a 3x3 filtering window. Median filtering is performed on all three RGB dimensions, with the R dimension used as an example; the other dimensions are the same. The filtering pixel indexes are as follows: width from 2 to W-1; height from 2 to H-1. The values ​​of other pixels remain unchanged. The first filtering point has pixel coordinates (2, 2). Using this filtering point as the center point of the window, the R components of 9 pixels in the 3x3 filtering window are extracted and sorted. The median value of the sorted R components, i.e., the 5th R component, is selected as the R value of the center pixel of the filtering window. The R dimension error value of this pixel is the new R value minus the original R value. The same applies to the other G and B components. This yields the filtered values ​​and error values ​​for multiple dimensions of the center pixel of the filtering window. Next, the entire filtering window is moved one pixel to the right to obtain a new filtering window with its center pixel coordinates (2, 3). Subsequent processing is the same as above, continuing to move to the right until the center pixel coordinates are (2, W-1), at which point the filtering of that row is complete. Then, the pixel coordinates (3, W-1) of the next row are taken as the center point of the filtering window and processed. The filtering window is then continuously moved to the right for further processing. The median filtering of the entire image is completed when the rightmost pixel coordinates (H-1, W-1) of the last row are taken as the filtering center point and processed. This yields the filtered images A_far and A_near.

[0042] Furthermore, taking the method for obtaining `error_far` as an example, `error_near` is calculated similarly. `error_far` = [(2,2) pixel error, ..., (H-1, W-1) pixel error], taking a specific point as an example: the pixel error calculation for other points is similar. `(2,2) pixel error = (abs((2,2) point R-dimensional error) + abs((2,2) point G-dimensional error) + abs((2,2) point B-dimensional error)) / 3`, from which we can obtain the filtered errors `error_far` and `error_near`.

[0043] 102. Extract the corner points of each quadrilateral in the checkerboard image from the distant and close-up images, and obtain the homography transformation matrix corresponding to each grid based on the corner points of each quadrilateral.

[0044] In this embodiment of the invention, step 102 may specifically include: obtaining the homography transformation matrix corresponding to each grid according to the corner pixel coordinates of each grid quadrilateral in the distant and close-up images.

[0045] Specifically, for example, if each cell of the chessboard is treated as a quadrilateral region, and the four corner points of this quadrilateral region are selected as the four feature points for calculating the homography transformation matrix, then Nw*Nh matrices can be obtained. First, the four corner points of each cell in the chessboard are manually found. Here, to reduce manual costs, a 4*4 chessboard is selected, which has a total of 20 corner points. Taking the first grid cell as an example, in the distant image A_far, the four pixel coordinates of the grid cell corner points are obtained: Afar_leftup = [x1, y1], Afar_leftdown = [x2, y2], Afar_rightup = [x3, y3], and Afar_rightdown = [x4, y4]. In the close-up image A_near, the four pixel coordinates of the grid cell corner points are obtained: Anear_leftup = [X1, Y1], Anear_leftdown = [X2, Y2], Anear_rightup = [X3, Y3], and Anear_rightdown = [X4, Y4]. Then, the transformation model between the close-up and distant images is established by detecting the correspondence between feature points. Next, the homography transformation matrix h1 between the two images is calculated, and its definition is as follows: Next, calculate the homography transformation matrix h1 of the distant image plane projected onto the near image plane: Substituting the 4 pixel coordinates of the distant view and the 4 pixel coordinates of the foreground view into the equation, we get: Perform eigenvalue decomposition on the large matrix on the left, take the last column of the decomposed unitary matrix, rearrange it into a 3*3 matrix, which is the homography transformation matrix h1. Then you can get the set of local homography transformation matrices of 16 cells h_total=[h1,…,h16].

[0046] 103. Obtain the homography transformation matrices corresponding to the distant camera and the close-up camera based on the homography transformation matrices corresponding to each grid.

[0047] In this embodiment of the invention, step 103 may specifically include: using the sum of the homography transformation matrices corresponding to each grid as the homography transformation matrices corresponding to the distant camera and the close-up camera.

[0048] Based on the filtered image described above, the step of obtaining the homography transformation matrices corresponding to the distant camera and the near camera according to the homography transformation matrices corresponding to each grid includes the following steps: obtaining the error value of the center pixel of each grid in the checkerboard image based on the filtered distant image, the distant image filtering error, the filtered near image, and the near image filtering error; and obtaining the homography transformation matrices corresponding to the distant camera and the near camera according to the homography transformation matrices corresponding to each grid and the error values ​​of the center pixels of each grid.

[0049] Specifically, for example, firstly, the error values ​​of the 16 grid center pixels, error = [error1, ..., errori, ..., error16], are processed for monotonicity, because the larger the error, the smaller the component of the homography transformation matrix: error_temp = [In(1 / error1), ..., In(1 / error16)]. Then, normalization is performed, weight = error_temp / sum(error) = [weight1, ..., weighti, ..., weight16]. Finally, the optimal homography transformation matrix h = weight1*h1 + ... + weight16*h16 is calculated by weighting.

[0050] 104. Based on the homography transformation matrix, fuse the distant image captured by the distant camera and the close-up image captured by the close-up camera.

[0051] In this embodiment of the invention, step 104 may specifically include: obtaining the pixel coordinates of the projected pixel of the near-field image on the distant plane based on the current pixel of the near-field image and the homography transformation matrix; if the projected pixel coordinates are within the size of the preset mapping image and are integers, then the projected pixel coordinates are assigned to the current pixel of the preset mapping image, wherein the preset mapping image is an empty image and is on the same plane as the near-field image; if the projected pixel coordinates are within the size of the preset mapping image and are not all integers, then the corrected projected pixel value is obtained by interpolating the projected pixel value and the values ​​of nearby pixels, and is used as the current pixel value of the preset mapping image.

[0052] Specifically, for example, first create a new empty image, which is on the same plane as the foreground image, called the third image. Then, using the optimal homography transformation matrix h obtained from (4) for the current pixel point (i, j) of the foreground image, substitute it into the basic formula of the homography transformation matrix to determine the pixel coordinates on the plane of the distant image after projection. If the projected pixel coordinates are greater than the width and height of the image, then the current pixel point (i, j) of the third image is not assigned a value. If the projected pixel coordinates are within the width and height of the image, and if the projected pixel coordinates (i, j) are all integers, then the value of the pixel coordinates (i, j) of the distant image is directly assigned to the value of the current pixel point (i, j) of the third image. If the projected pixel coordinates (i, j) are not all integers, then the value of the projected pixel point (i, j) and the values ​​of nearby pixels are interpolated to obtain the accurate projected distant pixel value, as follows: Define the floor function as... Round up to Bilinear interpolation is used to interpolate the value of the projected pixel (I, J) and the values ​​of its four nearest corner points: the pixel coordinates of the top-left point are... The pixel coordinates of the top right point are The pixel coordinates of the bottom right point are The pixel coordinates of the bottom left point are First, perform single-linear interpolation on the image's X-axis at the top-left, top-right, bottom-left, and bottom-right points to obtain the pixel values ​​at the midpoints of the grid above and below: Pixel value at the midpoint of the grid above: Pixel value at the bottom center of the grid Then, a single linear interpolation is performed about the Y-axis of the image to obtain the precise value of the pixel (I,J) within the grid: Finally, the pixel coordinates (I,J) of the distant image are directly assigned to the value of the current pixel (i,j) of the third image.

[0053] This invention provides a binocular camera image fusion method based on homography transformation matrices. The method obtains the homography transformation matrices corresponding to the distant and near cameras based on the homography transformation matrices corresponding to each grid in the checkerboard image of the distant and near images. Then, it fuses the distant image acquired by the distant camera and the near image acquired by the near camera according to the homography transformation matrices. This method achieves fusion of the distant and near images by calculating local homography transformation matrices and fusing them accordingly. It eliminates the need for multiple iterations of the RANSAC algorithm to find interior points during binocular image fusion, reducing the computational load and complexity of the binocular fusion process, and also offers strong real-time performance.

[0054] To implement the method provided in the embodiments of the present invention, the embodiments of the present invention provide a binocular camera image fusion system based on homography transformation matrix, such as... Figure 2 As shown, the system includes: an acquisition unit 21 and a fusion unit 22.

[0055] The acquisition unit 21 is used to acquire a distant image captured by a distant camera and a close-up image captured by a close-up camera, wherein the checkerboard pattern is arranged on the road surface captured by the distant camera and the close-up camera.

[0056] The acquisition unit 21 is further configured to extract the corner points of each quadrilateral in the checkerboard image from the distant and close-up images, and obtain the homography transformation matrix corresponding to each grid according to the corner points of each quadrilateral.

[0057] The acquisition unit 21 is further configured to acquire the homography transformation matrix corresponding to the distant camera and the close-up camera based on the homography transformation matrix corresponding to each grid.

[0058] The fusion unit 22 is used to fuse the distant image captured by the distant camera and the close-up image captured by the close-up camera according to the homography transformation matrix.

[0059] Furthermore, the acquisition unit 21 is specifically used to acquire the homography transformation matrix corresponding to each grid according to the corner pixel coordinates of each grid quadrilateral in the distant and close-up images.

[0060] Furthermore, the acquisition unit 21 is specifically used to take the sum of the homography transformation matrices corresponding to each grid as the homography transformation matrices corresponding to the distant camera and the close-up camera.

[0061] Further, the fusion unit 22 is specifically used to obtain the pixel coordinates of the projected pixel of the near-field image onto the distant plane based on the current pixel of the near-field image and the homography transformation matrix; if the projected pixel coordinates are within the size of the preset mapping image and are integers, then the projected pixel coordinates are assigned to the current pixel of the preset mapping image, wherein the preset mapping image is an empty image and is on the same plane as the near-field image; if the projected pixel coordinates are within the size of the preset mapping image and are not all integers, then the corrected projected pixel value is obtained by interpolating the projected pixel value and the values ​​of nearby pixels, and is used as the current pixel value of the preset mapping image.

[0062] Furthermore, the acquisition unit 21 is specifically used to perform median filtering on the distant image acquired by the distant camera and the close-up image acquired by the close-up camera respectively, to obtain the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error.

[0063] Furthermore, the acquisition unit 21 is specifically used to acquire the error value of each grid center pixel in the checkerboard image based on the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error; and to acquire the homography transformation matrix corresponding to the distant camera and the close-up camera based on the homography transformation matrix corresponding to each grid and the error value of each grid center pixel.

[0064] This invention provides a binocular camera image fusion system based on homography transformation matrices. The system obtains the homography transformation matrices corresponding to the distant and near cameras based on the homography transformation matrices corresponding to each grid in the checkerboard image of the distant and near images. Then, it fuses the distant image acquired by the distant camera and the near image acquired by the near camera according to the homography transformation matrices. This system achieves fusion of the distant and near images by calculating local homography transformation matrices and fusing them accordingly. It eliminates the need for multiple iterations of the RANSAC algorithm to find interior points during binocular image fusion, reducing the computational load and complexity of the binocular fusion process, and also offers strong real-time performance.

[0065] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0066] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0067] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0068] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0069] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0070] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic system, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing systems, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0071] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0072] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage systems, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server, or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wirelessly, such as infrared, wireless, and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0073] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A binocular camera image fusion method based on homography transformation matrix, characterized in that, The method includes: Acquire a distant image captured by a distant camera containing a checkerboard pattern and a close-up image captured by a close-up camera, wherein the checkerboard pattern is arranged on the road surface captured by the distant camera and the close-up camera. The corner points of each quadrilateral in the checkerboard image are extracted from the distant and close-up images, and the homography transformation matrix corresponding to each grid is obtained based on the corner points of each quadrilateral. The homography transformation matrices corresponding to the distant camera and the close-up camera are obtained based on the homography transformation matrices corresponding to each grid. Based on the homography transformation matrix, the distant image captured by the distant camera and the close-up image captured by the close-up camera are fused. The steps of extracting the corner points of each quadrilateral in the checkerboard image from the distant and close-up images, and obtaining the homography transformation matrix corresponding to each grid based on the corner points of each quadrilateral, include: Based on the corner pixel coordinates of each quadrilateral grid in the distant and close-up images, obtain the homography transformation matrix corresponding to each grid. The step of fusing the distant image captured by the distant camera and the close-up image captured by the close-up camera according to the homography transformation matrix includes: Based on the current pixel of the close-up image and the homography transformation matrix, the pixel coordinates of the close-up image pixel projected onto the distant plane are obtained. If the projected pixel coordinates are within the size of the preset mapped image and are integers, then the projected pixel coordinates are assigned to the current pixel point of the preset mapped image. The preset mapped image is an empty image and is on the same plane as the foreground image. If the coordinates of the projected pixel are within the size of the preset mapped image and are not all integers, then the projected pixel value is obtained by interpolation with the values ​​of nearby pixels, and is used as the current pixel value of the preset mapped image. The steps of acquiring a distant image captured by a distant camera and a close-up image captured by a close-up camera, both containing a checkerboard pattern, include: Median filtering was applied to the distant view image captured by the distant view camera and the close-up image captured by the close-up camera to obtain the filtered distant view image, the distant view image filtering error, the filtered close-up image, and the close-up image filtering error. The step of obtaining the homography transformation matrices corresponding to the distant camera and the close-up camera based on the homography transformation matrices corresponding to each grid includes: Based on the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error, the error value of each grid center pixel in the chessboard image is obtained; Based on the error value of the center pixel of each grid, the weights corresponding to the homography transformation matrix of each grid are obtained. Based on the weights, the homography transformation matrices corresponding to each grid are weighted and summed to obtain the homography transformation matrices corresponding to the distant camera and the close-up camera.

2. A binocular camera image fusion system based on homography transformation matrix, characterized in that, The system includes: An acquisition unit is used to acquire a distant image captured by a distant camera and a close-up image captured by a close-up camera, the checkerboard pattern being arranged on the road surface captured by the distant camera and the close-up camera. The acquisition unit is further configured to extract the corner points of each quadrilateral in the checkerboard image from the distant and close-up images, and to obtain the homography transformation matrix corresponding to each grid based on the corner points of each quadrilateral. The acquisition unit is further configured to acquire the homography transformation matrix corresponding to the distant camera and the close-up camera based on the homography transformation matrix corresponding to each grid. The fusion unit is used to fuse the distant image captured by the distant camera and the close-up image captured by the close-up camera according to the homography transformation matrix. The acquisition unit is specifically used to acquire the homography transformation matrix corresponding to each grid according to the corner pixel coordinates of each grid quadrilateral in the distant and close-up images; The fusion unit is specifically used to obtain the pixel coordinates of the projected pixels of the near-field image onto the distant plane based on the current pixel of the near-field image and the homography transformation matrix; if the projected pixel coordinates are within the size of a preset mapping image and are integers, then the projected pixel coordinates are assigned to the current pixel of the preset mapping image, where the preset mapping image is an empty image and is on the same plane as the near-field image; if the projected pixel coordinates are within the size of the preset mapping image and are not all integers, then the projected pixel value is interpolated with the values ​​of nearby pixels to obtain the corrected projected pixel value, which is then used as the current pixel value of the preset mapping image. The acquisition unit is further configured to perform median filtering on the distant image acquired by the distant camera and the close-up image acquired by the close-up camera, respectively, to obtain the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error. The acquisition unit is further configured to acquire the error value of each grid center pixel in the chessboard image based on the filtered distant image, the distant image filtering error, the filtered close-up image, and the close-up image filtering error. Based on the error value of the center pixel of each grid, the weights corresponding to the homography transformation matrix of each grid are obtained. Based on the weights, the homography transformation matrices corresponding to each grid are weighted and summed to obtain the homography transformation matrices corresponding to the distant camera and the close-up camera.

Images