Binocular visible light image and infrared image fusion method, electronic equipment and medium

Abstract

The invention discloses a binocular visible light image and infrared image fusion method, electronic equipment and a medium. The method comprises the steps of obtaining a left eye image and a right eye image collected by a binocular camera and an infrared image collected by an infrared camera; based on a checkerboard corner detection algorithm, extracting a plurality of initial corner arrays from an overlapping region of a left eye image and a right eye image, and extracting a plurality of initial corner arrays from overlapping regions of the infrared image, the left eye image and the right eye image; the initial angular point array comprises a plurality of angular points with initial angular point coordinates; for any initial angular point array, sorting the angular points in the initial angular point array according to the checkerboard side length information and the initial angular point coordinates of the angular points in the initial angular point array to obtain an ordered angular point array; and determining a target fusion image according to the left eye image and each ordered corner array thereof, the right eye image and each ordered corner array thereof, and the infrared image and each ordered corner array thereof.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, electronic device and medium for fusing binocular visible light images and infrared images. Background Art

[0002] Fusion of visible light images with infrared images can improve the performance of target tracking. At present, the field has been able to achieve the fusion of single-purpose visible light images with infrared images and the splicing of binocular visible light images, but has not yet involved how to fuse binocular visible light images with infrared images, so the application scenarios of image processing are still limited. In addition, the current stitching algorithm usually first identifies the feature points of the actual scene data before stitching. This method has poor stitching effect when there are fewer feature points in the scene or the distribution is uneven. Although the method of using the checkerboard corner point recognition algorithm to identify feature points can improve the stitching effect, it has strict requirements on the placement and number of the checkerboards, so it is not easy to implement, and it takes a long time, so the stitching efficiency is low.

[0003] Therefore, there is an urgent need for a method that can fuse multi-purpose visible light images with infrared images and improve the efficiency of multi-purpose visible light image stitching. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art to a certain extent. To this end, the present application provides a method for fusing binocular visible light images with infrared images, an electronic device and a medium.

[0005] As a first aspect of the present application, a method for fusing a binocular visible light image and an infrared image is provided, wherein the method comprises:

[0006] Acquire a left-eye image and a right-eye image captured by a binocular camera, and acquire an infrared image captured by an infrared camera;

[0007] Based on a checkerboard corner point detection algorithm, a plurality of initial corner point arrays are extracted from the overlapping area of ​​the left-eye image and the right-eye image, and a plurality of initial corner point arrays are extracted from the overlapping area of ​​the infrared image and the left-eye image and the right-eye image; wherein the initial corner point array includes a plurality of corner points having initial corner point coordinates;

[0008] For any of the initial corner point arrays, sorting the corner points in the initial corner point array according to the chessboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array;

[0009] A target fused image is determined according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays.

[0010] Optionally, each corner point in the initial corner point array is arranged in multiple rows and columns, and for any of the initial corner point arrays, sorting each corner point in the initial corner point array according to the chessboard side length information and the initial corner point coordinates of each corner point in the initial corner point array to obtain an ordered corner point array includes:

[0011] For any of the initial corner point arrays, the corner point located in the second row and the second column is determined as the target left vertex, and the corner point located in the second row and the second to last column is determined as the target right vertex;

[0012] Determine the equation of a straight line between the left vertex of the target and the right vertex of the target according to the initial corner point coordinates of the left vertex of the target and the initial corner point coordinates of the right vertex of the target;

[0013] According to the initial corner point coordinates of other corner points in the initial corner point array, identifying the corner points located on the straight line identified by the straight line equation, and sorting the identified corner points;

[0014] According to the currently sorted row of corner points and the chessboard side length information, the next row of corner points is identified and the identified next row of corner points is sorted.

[0015] Optionally, determining a target fused image according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays includes:

[0016] According to the ordered corner point arrays of the left image and the ordered corner point arrays of the right image, the left image and the right image are registered to obtain visible road marking information;

[0017] According to the visible road marking information, coordinates of each pixel point in the left eye image and coordinates of each pixel point in the right eye image are respectively converted to obtain a converted left eye image and a converted right eye image;

[0018] According to the ordered corner point arrays of the converted left-eye image and the converted right-eye image and the infrared image, the infrared image is registered with the converted left-eye image and the converted right-eye image to obtain infrared road marking information;

[0019] According to the infrared road marking information, the coordinates of each pixel in the infrared image are converted to obtain a converted infrared image;

[0020] A target fused image is determined according to the converted left-eye image, the converted right-eye image and the converted infrared image.

[0021] Optionally, determining a target fused image according to the converted left-eye image, the converted right-eye image and the converted infrared image includes:

[0022] Matching the converted left-eye image, the converted right-eye image and the converted infrared image according to the coordinates of the pixels, identifying a first pixel matching group, a second pixel matching group and a third pixel matching group; wherein the first pixel matching group includes two matching pixels between the converted left-eye image and the converted infrared image, the second pixel matching group includes three matching pixels between the converted left-eye image, the converted right-eye image and the converted infrared image, and the third pixel matching group includes two matching pixels between the converted right-eye image and the converted infrared image;

[0023] For any of the pixel point matching groups, the color coding information of each matching pixel point therein is fused to obtain a fused pixel point;

[0024] A target fused image is determined according to each of the fused pixel points and their coordinates.

[0025] Optionally, the second pixel matching group includes left-eye matching pixels, right-eye matching pixels and infrared matching pixels, and in the step of fusing the color coding information of each matching pixel in any of the pixel matching groups to obtain a fused pixel, the step includes:

[0026] For any of the second pixel point matching groups, the color coding information of the left-eye matching pixel points therein is merged with the color coding information of the right-eye matching pixel points therein to obtain a spliced ​​pixel point;

[0027] The color coding information of the infrared matching pixel points in the second pixel matching group is fused with the color coding information of the splicing pixel points to obtain a fused pixel point.

[0028] Optionally, the color coding information of two matching pixels is fused using the following formula:

[0029] y after =xy before1 +(1-x)y before2 (1);

[0030] In formula (1), y after represents the color coding information obtained by fusing the color coding information of two matching pixels, ybefore1 Indicates the color coding information of the first matching pixel, y before2 represents the color coding information of the second matching pixel, x represents the weight coefficient and is determined according to the distance between the first matching pixel and the midline of the overlapping area of ​​the corresponding image.

[0031] Optionally, the dimensions of the color coding information include brightness values ​​and color component values.

[0032] Optionally, the calibration information is an affine transformation matrix.

[0033] As a second aspect of the present application, an electronic device is provided, wherein the electronic device includes:

[0034] one or more processors;

[0035] A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method for fusing binocular visible light images and infrared images as described in the first aspect of the present application.

[0036] As a third aspect of the present application, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the method for fusing binocular visible light images and infrared images described in the first aspect of the present application is implemented.

[0037] In the method for fusing binocular visible light images and infrared images provided in an embodiment of the present application, a left-eye image and a right-eye image captured by a binocular camera are obtained, and an infrared image captured by an infrared camera is obtained; based on a checkerboard corner point detection algorithm, a plurality of initial corner point arrays are extracted from the overlapping area of ​​the left-eye image and the right-eye image, and a plurality of initial corner point arrays are extracted from the overlapping area of ​​the infrared image and the left-eye image and the right-eye image; wherein the initial corner point array includes a plurality of corner points with initial corner point coordinates; for any of the initial corner point arrays, the corner points in the initial corner point array are sorted according to checkerboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array; a target fused image is determined according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays. Regardless of the placement and number of chessboard squares, the accuracy of feature point matching can be improved by improving the accuracy of corner point sorting, thereby improving the image stitching effect and efficiency, realizing the fusion of binocular visible light images and infrared images, and expanding the application scenarios of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present application is further described below in conjunction with the accompanying drawings:

[0039] Figure 1 It is a flow chart of an implementation method of a binocular visible light image and an infrared image fusion method provided in an embodiment of the present application;

[0040] Figure 2a is a schematic diagram of extracting multiple initial corner point arrays from overlapping areas provided by an embodiment of the present application;

[0041] Figure 2b is a schematic diagram of extracting multiple initial corner point arrays from overlapping areas provided by an embodiment of the present application;

[0042] Figure 2c is a schematic diagram of a chessboard provided in an embodiment of the present application;

[0043] Figure 2d It is a schematic diagram of the distance between a corner point and the line connecting two other corner points provided in an embodiment of the present application;

[0044] Figure 3 This is a flow chart of an implementation method of obtaining an ordered corner point array provided in an embodiment of the present application;

[0045] Figure 4 It is a flowchart of an implementation method of determining a target fused image according to a graphic and its ordered corner point arrays provided in an embodiment of the present application;

[0046] Figure 5 This is a flowchart of an implementation method of determining a target fused image based on a converted image provided in an embodiment of the present application;

[0047] Figure 6 This is a flowchart of an implementation method of obtaining a fused pixel point provided in an embodiment of the present application;

[0048] Figure 7 It is a module diagram of an implementation of an electronic device provided in an embodiment of the present application;

[0049] Figure 8 It is a schematic diagram of a computer-readable medium provided in an embodiment of the present application.

[0050] Description of Reference Numerals

[0051] 101: Processor 102: Memory

[0052] 103: I / O interface 104: bus DETAILED DESCRIPTION

[0053] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments based on the embodiments are intended to be used to explain the present application and cannot be understood as limiting the present application.

[0054] References to "one embodiment" or "an example" or "an example" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment itself may be included in at least one embodiment disclosed in the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0055] At present, how to fuse binocular visible light images with infrared images has not been discussed, and there are also many problems in splicing binocular visible light images. In view of this, the applicant of this application proposes to extract the checkerboard corner point array from the overlapping area of ​​the left-eye image, the right-eye image and the infrared image by using the checkerboard corner point detection algorithm, and sort the corner point array in combination with the checkerboard side length information and the corner point coordinates. The optimized sorting method can improve the accuracy of subsequent feature point matching, and the left-eye image, the right-eye image and the infrared image are subsequently processed such as registration and calibration based on the optimized sorted corner point array, which can not only realize the fusion of binocular visible light images with infrared images, but also effectively improve the splicing effect and there is no restriction on the placement and number of checkerboards.

[0056] As a first aspect of the embodiments of the present application, a method for fusing a binocular visible light image and an infrared image is provided, such as Figure 1 As shown, the method may include:

[0057] In step S110, a left-eye image and a right-eye image captured by a binocular camera are acquired, and an infrared image captured by an infrared camera is acquired;

[0058] In step S120, based on a checkerboard corner point detection algorithm, a plurality of initial corner point arrays are extracted from the overlapping area of ​​the left eye image and the right eye image, and a plurality of initial corner point arrays are extracted from the overlapping area of ​​the infrared image and the left eye image and the right eye image; wherein the initial corner point array includes a plurality of corner points having initial corner point coordinates;

[0059] In step S130, for any of the initial corner point arrays, the corner points in the initial corner point array are sorted according to the chessboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array;

[0060] In step S140, a target fused image is determined according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays.

[0061] There is usually an overlapping area between the left and right images captured by the binocular camera, and there may also be overlapping areas between the infrared image and the left and right images, such as Figure 2a As shown, it is a schematic diagram of an overlapping scene of the left-eye image, the right-eye image and the infrared image provided in an embodiment of the present application. The left-eye image includes area A and area B shown in the figure, the right-eye image includes area B and area C shown in the figure, and the infrared image includes area D, area E and area F shown in the figure. It can be seen that the overlapping area between the left-eye image and the right-eye image is area B, the overlapping area between the infrared image and the left-eye image is area D and area E, the overlapping area between the infrared image and the right-eye image is area E and area F, and the overlapping area among the left-eye image, the right-eye image and the infrared image is area E.

[0062] like Figure 2b As shown in FIG. 1 , it is a schematic diagram of extracting multiple initial corner point arrays from the overlapping area provided by the embodiment of the present application. Taking the overlapping area B as an example, each small rectangle therein represents a chessboard. Figure 2c As shown, it is a schematic diagram of a chessboard provided in an embodiment of the present application, wherein the four vertices of each black square or white square represent a corner point, that is, a pixel point or a feature point. These corner points are arranged as an initial corner point array, so a chessboard corresponds to an initial corner point array.

[0063] That is to say, to extract multiple initial corner point arrays from the overlapping area of ​​the left eye image and the right eye image, multiple initial corner point arrays need to be extracted from area B of the left eye image, and multiple initial corner point arrays need to be extracted from area B of the right eye image. To extract multiple initial corner point arrays from the overlapping area of ​​the infrared image and the left eye image and the right eye image, multiple initial corner point arrays need to be extracted from area D, area E, and area F of the infrared image, multiple initial corner point arrays need to be extracted from area D and area E of the left eye image, and multiple initial corner point arrays need to be extracted from area E and area F of the right eye image.

[0064] Since there are overlapping areas between images, based on the initial corner point array extracted from the overlapping area, the images can be aligned according to the overlapping area for splicing and fusion.

[0065] It should be noted that Figure 2a , Figure 2b , Figure 2cThis is just an example. In addition, there may be other situations such as the infrared image only overlaps with the left eye image, the infrared image only overlaps with the right eye image, the infrared image has areas that overlap with the left eye image and / or the right eye image, and areas that do not overlap with the left eye image and the right eye image, and the placement and number of the chessboard grids vary. However, no matter which of the above situations, the method for fusing binocular visible light images and infrared images provided in the embodiments of the present application is still feasible.

[0066] It can be understood that for the special case where the infrared image does not overlap with the left-eye image and the right-eye image, although it is impossible to perform fusion processing through the binocular visible light image and infrared image fusion method provided in the embodiment of the present application, based on the same principle, that is, only the left-eye image and the right-eye image are processed to obtain a stitched image, the image stitching effect can also be improved.

[0067] The applicant of the present application has found that the existing methods of using the checkerboard corner point recognition algorithm to identify feature points are all to directly sort the corner points according to their initial corner point coordinates. However, this method often results in data confusion (which may be caused by the fact that the corner point coordinates of different checkerboards are the same but the corresponding actual spatial positions are not the same), and the feature point matching results will also be greatly affected, so the corresponding splicing and fusion effects will also be greatly reduced. However, no matter what placement position or number of checkerboards are used, as long as the adjacent corner points of a corner point are strictly screened according to the distance between the corner points by combining the checkerboard side length information and the corner point coordinates, and the reordering of all corner points in an initial corner point array is completed, the sorting accuracy can be effectively improved, thereby improving the accuracy of the feature point matching results, and then improving the splicing and fusion effects of the image.

[0068] In the method for fusing binocular visible light images and infrared images provided in an embodiment of the present application, a left-eye image and a right-eye image captured by a binocular camera are obtained, and an infrared image captured by an infrared camera is obtained; based on a checkerboard corner point detection algorithm, a plurality of initial corner point arrays are extracted from the overlapping area of ​​the left-eye image and the right-eye image, and a plurality of initial corner point arrays are extracted from the overlapping area of ​​the infrared image and the left-eye image and the right-eye image; wherein the initial corner point array includes a plurality of corner points with initial corner point coordinates; for any of the initial corner point arrays, the corner points in the initial corner point array are sorted according to checkerboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array; a target fused image is determined according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays. Regardless of the placement and number of chessboard squares, the accuracy of feature point matching can be improved by improving the accuracy of corner point sorting, thereby improving the image stitching effect and efficiency, realizing the fusion of binocular visible light images and infrared images, and expanding the application scenarios of image processing.

[0069] As described above, by combining the chessboard side length information and the corner point coordinates, the adjacent corner points of a corner point are strictly screened according to the distance between the corner points, so as to complete the reordering of all corner points in an initial corner point array. Accordingly, in some embodiments, the corner points in the initial corner point array are arranged in multiple rows and columns. For any of the initial corner point arrays, the corner points in the initial corner point array are sorted according to the chessboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array (i.e., step S130), as shown in FIG. Figure 3 As shown, it may include:

[0070] In step S210, for any of the initial corner point arrays, the corner point located in the second row and the second column is determined as the target left vertex, and the corner point located in the second row and the second to last column is determined as the target right vertex;

[0071] In step S220, the equation of a straight line between the left vertex of the target and the right vertex of the target is determined according to the initial corner point coordinates of the left vertex of the target and the initial corner point coordinates of the right vertex of the target;

[0072] In step S230, according to the initial corner point coordinates of other corner points in the initial corner point array, the corner points located on the straight line identified by the straight line equation are identified, and the identified corner points are sorted;

[0073] In step S240, the next row of corner points is identified based on the currently sorted row of corner points and the chessboard side length information, and the identified next row of corner points is sorted.

[0074] Still like Figure 2c Taking the initial corner point array shown as an example, the corner point P located in the second row and second column is determined as the target left vertex, and the corner point Q located in the second row and second to last column is determined as the target right vertex. The corner point coordinates of the corner point P are expressed as (x1, y1), and the corner point coordinates of the corner point Q are expressed as (x2, y2). The equation of the straight line PQ can be calculated as y=kx+b. b=y1-kx1. Figure 2d As shown, it is a schematic diagram of the distance between a corner point and the line connecting two other corner points provided in an embodiment of the present application. Corner points in the same row should be located on the same straight line, so the distance from other corner points M (the coordinates of the corner points are expressed as (x0, y0)) to the straight line PQ is calculated: If d is 0, it means that the corner point M is on the line PQ. At this time, we can further calculate the distance between the corner point M and the corner points P and Q to determine the order of the corner points M, P, and Q. If the distance between the two corner points is the length of the chessboard side, it means that the two corner points are adjacent corner points. If d is not 0, it means that the corner point M is not on the line PQ, and we can turn to judge the next corner point. In this way, the corner points of the PQ row are identified and sorted. Then, with the chessboard side length as the step, we start to identify and sort the corner points in the next row on the line PQ.

[0075] It can be seen that this sorting method can very accurately sort the corner points in an initial corner point array by strictly comparing the distance between corner points with the side length of the chessboard, avoiding sorting confusion, and effectively improving the sorting accuracy, thereby improving the accuracy of feature point matching results, and further improving the image stitching and fusion effects.

[0076] After completing the reordering of the corner points in the initial corner point array, the image can be registered and calibrated. Accordingly, in some embodiments, the target fused image is determined based on the left eye image and its ordered corner point arrays, the right eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays (i.e., step S140), such as Figure 4 As shown, it may include:

[0077] In step S310, the left image and the right image are registered according to the ordered corner point arrays of the left image and the ordered corner point arrays of the right image to obtain visible road marking information;

[0078] In step S320, according to the visible road marking information, the coordinates of each pixel point in the left eye image and the coordinates of each pixel point in the right eye image are respectively converted to obtain a converted left eye image and a converted right eye image;

[0079] In step S330, the infrared image is registered with the converted left eye image and the converted right eye image according to the ordered corner point arrays of the converted left eye image and the converted right eye image, so as to obtain infrared road marking information;

[0080] In step S340, the coordinates of each pixel in the infrared image are converted according to the infrared road marking information to obtain a converted infrared image;

[0081] In step S350, a target fused image is determined according to the converted left-eye image, the converted right-eye image and the converted infrared image.

[0082] The visible road calibration information is used to characterize the conversion relationship between the corner point coordinates of the left eye image and the right eye image, and the infrared road calibration information is used to characterize the conversion relationship of the corner point coordinates of the infrared image to the left eye image and the right eye image. As for how to register the images according to the corner point coordinates to obtain the calibration information, the art already has an implementable method, and the embodiments of the present application do not specifically limit this.

[0083] The coordinates of each pixel point of the left eye image, the right eye image and the infrared image are converted respectively by using the visible road marking information and the infrared road marking information, so that the coordinates of each pixel point of the converted left eye image, the converted right eye image and the converted infrared image are located in the same coordinate system, which is convenient for splicing pixels with different coordinates and fusing pixels with the same coordinates.

[0084] Accordingly, in some embodiments, the target fused image is determined according to the converted left eye image, the converted right eye image and the converted infrared image (ie, step S350), such as Figure 5 As shown, it may include:

[0085] In step S410, the converted left-eye image, the converted right-eye image and the converted infrared image are matched according to the coordinates of the pixel points, and a first pixel point matching group, a second pixel point matching group and a third pixel point matching group are identified; wherein the first pixel point matching group includes two matching pixel points between the converted left-eye image and the converted infrared image, the second pixel point matching group includes three matching pixel points between the converted left-eye image, the converted right-eye image and the converted infrared image, and the third pixel point matching group includes two matching pixel points between the converted right-eye image and the converted infrared image;

[0086] In step S420, for any of the pixel matching groups, the color coding information of each matching pixel therein is fused to obtain a fused pixel;

[0087] In step S430, a target fused image is determined according to each of the fused pixel points and their coordinates.

[0088] It can be understood that the two matching pixels in the first pixel matching group have the same coordinates, one from the converted left eye image and the other from the converted infrared image. The three matching pixels in the second pixel matching group have the same coordinates, which are respectively from the converted left eye image, the converted right eye image and the converted infrared image. The two matching pixels in the third pixel matching group have the same coordinates, one from the converted right eye image and the other from the converted infrared image.

[0089] For a first pixel point matching group or a third pixel point matching group, the color coding information of two matching pixel points therein is fused while keeping the coordinates unchanged, so as to obtain a fused pixel point having the coordinates and fused color coding information.

[0090] For a second pixel matching group, it is necessary to first fuse the matching pixel points from the converted left eye image with the matching pixel points from the converted right eye image, and then fuse the fusion result with the matching pixel points from the converted infrared image to obtain a final fused pixel point.

[0091] Accordingly, in some embodiments, the second pixel matching group includes left-eye matching pixels, right-eye matching pixels, and infrared matching pixels. In the step of fusing the color coding information of each matching pixel in any of the pixel matching groups to obtain a fused pixel (i.e., step S420), as Figure 6 As shown, it may include:

[0092] In step S510, for any of the second pixel matching groups, the color coding information of the left-eye matching pixel therein is merged with the color coding information of the right-eye matching pixel therein to obtain a spliced ​​pixel;

[0093] In step S520, the color coding information of the infrared matching pixel points in the second pixel matching group is fused with the color coding information of the spliced ​​pixel points to obtain a fused pixel point.

[0094] Regardless of the first pixel matching group, the second pixel matching group or the third pixel matching group, the color coding information of every two matching pixel points is fused. In some embodiments, the color coding information of two matching pixel points is fused by the following formula:

[0095] y after =xy before1 +(1-x)y before2 (1);

[0096] In formula (1), y after represents the color coding information obtained by fusing the color coding information of two matching pixels, y before1 Indicates the color coding information of the first matching pixel, y before2 represents the color coding information of the second matching pixel, x represents the weight coefficient and is determined according to the distance between the first matching pixel and the midline of the overlapping area of ​​the corresponding image.

[0097] For example, for a first pixel matching group, y before1 From the transformed left image, y before2 From the transformed right image, then x can be calculated according to y before1 The distance between the left eye image and the center line of the overlapping area (of the converted left eye image and the converted right eye image) is determined.

[0098] Of course, the embodiments of the present application are not limited to this. For the second pixel matching group, the ratio relationship between the visible light image (including the converted left eye image and the converted right eye image) and the infrared image (i.e., the converted infrared image), i.e., x and (1--x), can also be set according to other pre-set strategies.

[0099] In some embodiments, the dimensions of the color encoding information include brightness values ​​and color component values.

[0100] The color coding information is image information in YUV format, the brightness value is represented by Y, and the color component value may include a color component U and a color component V. The brightness value and the color component value of two matching pixels may be fused respectively to obtain a fused brightness value and a fused color component value.

[0101] In some embodiments, the calibration information is an affine transformation matrix.

[0102] The binocular visible light image and infrared image fusion method provided in the embodiment of the present application can be applied to FPGA (Field-Programmable Gate Array), which means that after the method is programmed on the FPGA, only two channels of visible light data (left eye image and right eye image) and one channel of infrared data (infrared image) need to be input into the FPGA to automatically align and calibrate the images, splice the visible light data and fuse it with the infrared data, output the fused data in real time and display it, greatly improving the efficiency of image processing. Moreover, fusing the visible light image with the infrared image can also effectively expand the application scenarios of image processing.

[0103] As a second aspect of the embodiments of the present application, an electronic device is provided, wherein: Figure 7 As shown, the electronic device includes:

[0104] One or more processors 101;

[0105] The memory 102 stores one or more computer programs. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement the method for fusing binocular visible light images and infrared images provided in the first aspect of the embodiment of the present application.

[0106] The electronic device may further include one or more I / O interfaces 103 connected between the processor 101 and the memory 102 and configured to implement information interaction between the processor 101 and the memory 102 .

[0107] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the processor and the memory, including but not limited to a data bus (Bus), etc.

[0108] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0109] As a third aspect of the embodiment of the present application, Figure 8 As shown, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the method for fusing binocular visible light images and infrared images provided in the first aspect of the embodiment of the present application is implemented.

[0110] It can be understood by a person of ordinary skill in the art that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can implement the method of any of the above-mentioned embodiments when executed. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present application will be included in the scope of the claims.

Claims

1. A method for fusing binocular visible light images and infrared images, characterized in that: The method comprises: Acquire a left-eye image and a right-eye image captured by a binocular camera, and acquire an infrared image captured by an infrared camera; Based on a checkerboard corner point detection algorithm, a plurality of initial corner point arrays are extracted from the overlapping area of ​​the left-eye image and the right-eye image, and a plurality of initial corner point arrays are extracted from the overlapping area of ​​the infrared image and the left-eye image and the right-eye image; wherein the initial corner point array includes a plurality of corner points having initial corner point coordinates; For any of the initial corner point arrays, sorting the corner points in the initial corner point array according to the chessboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array; A target fused image is determined according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays.

2. The method according to claim 1, characterized in that The corner points in the initial corner point array are arranged in multiple rows and columns. For any of the initial corner point arrays, the corner points in the initial corner point array are sorted according to the chessboard side length information and the initial corner point coordinates of the corner points in the initial corner point array to obtain an ordered corner point array, including: For any of the initial corner point arrays, the corner point located in the second row and the second column is determined as the target left vertex, and the corner point located in the second row and the second to last column is determined as the target right vertex; Determine the equation of a straight line between the left vertex of the target and the right vertex of the target according to the initial corner point coordinates of the left vertex of the target and the initial corner point coordinates of the right vertex of the target; According to the initial corner point coordinates of other corner points in the initial corner point array, identifying the corner points located on the straight line identified by the straight line equation, and sorting the identified corner points; According to the currently sorted row of corner points and the chessboard side length information, the next row of corner points is identified and the identified next row of corner points is sorted.

3. The method according to claim 1, characterized in that Determining a target fused image according to the left-eye image and its ordered corner point arrays, the right-eye image and its ordered corner point arrays, and the infrared image and its ordered corner point arrays includes: According to the ordered corner point arrays of the left image and the ordered corner point arrays of the right image, the left image and the right image are registered to obtain visible road marking information; According to the visible road marking information, coordinates of each pixel point in the left eye image and coordinates of each pixel point in the right eye image are respectively converted to obtain a converted left eye image and a converted right eye image; According to the ordered corner point arrays of the converted left-eye image and the converted right-eye image and the infrared image, the infrared image is registered with the converted left-eye image and the converted right-eye image to obtain infrared road marking information; According to the infrared road marking information, the coordinates of each pixel in the infrared image are converted to obtain a converted infrared image; A target fused image is determined according to the converted left-eye image, the converted right-eye image and the converted infrared image.

4. The method according to claim 3, characterized in that The step of determining a target fused image according to the converted left-eye image, the converted right-eye image and the converted infrared image comprises: Matching the converted left-eye image, the converted right-eye image and the converted infrared image according to the coordinates of the pixels, and identifying a first pixel matching group, a second pixel matching group and a third pixel matching group; wherein the first pixel matching group includes two matching pixels between the converted left-eye image and the converted infrared image, the second pixel matching group includes three matching pixels between the converted left-eye image, the converted right-eye image and the converted infrared image, and the third pixel matching group includes two matching pixels between the converted right-eye image and the converted infrared image; For any of the pixel point matching groups, the color coding information of each matching pixel point therein is fused to obtain a fused pixel point; A target fused image is determined according to each of the fused pixel points and their coordinates.

5. The method according to claim 4, characterized in that The second pixel matching group includes left-eye matching pixel points, right-eye matching pixel points and infrared matching pixel points. For any of the pixel matching groups, the step of fusing the color coding information of each matching pixel point therein to obtain a fused pixel point includes: For any of the second pixel point matching groups, the color coding information of the left-eye matching pixel points therein is merged with the color coding information of the right-eye matching pixel points therein to obtain a spliced ​​pixel point; The color coding information of the infrared matching pixel points in the second pixel matching group is fused with the color coding information of the splicing pixel points to obtain a fused pixel point.

6. The method according to claim 4, characterized in that The color coding information of two matching pixels is fused using the following formula: y after =xy before1 +(1-x)y before2 (1); In formula (1), y after represents the color coding information obtained by fusing the color coding information of two matching pixels, y before1 Indicates the color coding information of the first matching pixel, y before2 represents the color coding information of the second matching pixel, x represents the weight coefficient and is determined according to the distance between the first matching pixel and the midline of the overlapping area of ​​the corresponding image.

7. The method according to any one of claims 2 to 6, characterized in that: The dimensions of the color coding information include brightness value and color component value.

8. The method according to any one of claims 2 to 6, characterized in that: The calibration information is an affine transformation matrix.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method for fusing a binocular visible light image with an infrared image according to any one of claims 1 to 8.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for fusing a binocular visible light image and an infrared image according to any one of claims 1 to 8 is implemented.

Images