Image fusion correction method for RGB camera and thermal imaging camera
By using a checkerboard correction plate and sub-pixel corner refinement processing, the problem of accurate correction for RGB cameras and thermal imaging cameras was solved, achieving high-precision image fusion and improving the efficiency of search and rescue, inspection and monitoring of UAV systems.
Patent Information
- Application Number
- CN202610083283.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-01
AI Technical Summary
In existing technologies, RGB cameras and thermal imaging cameras are difficult to accurately calibrate on aircraft such as drones, resulting in the inability to accurately overlay temperature information on the target in thermal imaging images, which affects the efficiency of search and rescue, inspection and monitoring.
A checkerboard correction plate is used. Checkerboard images with constant temperature heat sources are captured by an RGB camera and a thermal imaging camera respectively. The corner coordinates are obtained, and sub-pixel corner refinement and consistency checks are performed. A homography matrix is established to achieve image fusion of RGB and IR images.
It improves the mapping and fusion accuracy of RGB and IR images, ensuring that temperature information is accurately superimposed on the target in the thermal imaging image, thereby improving the efficiency of search and rescue, inspection and monitoring.
Smart Images
Figure CN121961866A_ABST
Abstract
Description
Image fusion correction method for RGB cameras and thermal imaging cameras Technical Field
[0001] This invention belongs to the field of image fusion, and in particular relates to an image fusion correction method for RGB cameras and thermal imaging cameras. Background Technology
[0002] Drones and other aircraft typically carry both RGB cameras and thermal imaging cameras, necessitating precise calibration to align them to the same coordinate system. This allows targets marked on the RGB image to be accurately located and have their temperature information overlaid on the thermal image, improving search and rescue, inspection, and monitoring efficiency. Therefore, a standardized calibration process and parameters are necessary to ensure product consistency and facilitate maintenance and testing. Summary of the Invention
[0003] To address the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide an image fusion correction method for RGB cameras and thermal imaging cameras.
[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an image fusion correction method using an RGB camera and a thermal imaging camera, comprising the following steps: S100, using an RGB camera and a thermal imaging camera respectively to photograph a checkerboard correction plate with a constant-temperature heat source, obtaining RGB images and IR images; the corner points of the checkerboard are the RGB corner points of the correction plate. The center point of the constant temperature heat source is the IR corner point of the calibration plate. The constant-temperature heat sources are arranged at equal intervals in both the horizontal and vertical directions, and each of the calibration plate IR corner points All are aligned with the RGB corner points of a calibration board. Overlap; S200, obtain the corner coordinates on the RGB image and IR image respectively to obtain the first RGB corner set. and the first IR corner set S300, based on the first RGB corner point set The first RGB corner point in the middle The coordinates and the corresponding RGB corner points of the calibration board in the board coordinate domain Coordinate estimation of the homography matrix mapping from the plate coordinate domain to the RGB image The parameters in the first IR corner point set are based on the parameters in the first IR corner point set. First IR corner points in each of the middle The coordinates and the corresponding IR corner points of the calibration plate in the plate coordinate domain Coordinate estimation of the homography matrix mapped from the plate coordinate domain to the IR image The parameters in; S400, for the first RGB corner point set and the first IR corner set Each corner point in the image undergoes sub-pixel corner point refinement, followed by a consistency check to remove corner points that do not meet the consistency requirements, forming a second RGB corner point set. Second IR corner set S500, Set the second RGB corner points Second IR corner set The missing corner points are filled in to obtain the third RGB corner point set and the third IR corner point set. The homography matrix is then re-estimated based on the corner points in the third RGB corner point set and the third IR corner point set. Homography matrix The parameters in.
[0005] S600, According to the homography matrix Homography matrix Establish the mapping relationship between the plate coordinate domain and the RGB image, as well as the mapping relationship between the plate coordinate domain and the IR image, to achieve image fusion of RGB and IR images.
[0006] Furthermore, step S100 includes the following sub-steps: S110, heating each constant-temperature heat source of the checkerboard to the target temperature and waiting for a period of time; S120, using an RGB camera and a thermal imaging camera to take one frame of image respectively with fixed exposure as RGB image and IR image; or using an RGB camera and a thermal imaging camera to take multiple frames of image respectively with fixed exposure, and performing time-domain averaging noise reduction processing to obtain RGB image and IR image.
[0007] Furthermore, in step S200, corner coordinates are obtained on the IR image to obtain a first IR corner set. The method includes the following sub-steps: S211, Calculate the isothermal boundaries of each isothermal heat source on the IR image; S212, According to the IR corner points of the calibration plate The location divides the chessboard into multiple 2×2 regions (i.e., each region has four IR corner points of the correction plate that form a 2×2 array). In each of the 2×2 regions, least squares straight line fitting is performed on two sets of approximately orthogonal isothermal boundaries, and their intersection points are used as initial temperature corner points; S213, each initial temperature corner point is fitted with a straight line and monotonically sorted according to its row and column respectively; S214, by forcing collinearity, outlier initial temperature corner points that are not collinear are removed, and the remaining initial temperature corner points are used as temperature identification corner points; S215, each temperature identification corner point is sub-pixel refined within a predetermined size patch by quadratic fitting or Gaussian kernel fitting to obtain the first IR corner point with sub-pixel level accuracy in coordinates. Thus, the first IR corner point set is obtained. .
[0008] Furthermore, in step S200, corner coordinates are obtained from the RGB image to obtain a first RGB corner set. The method includes the following sub-steps: S221, using the findChessboardCorners algorithm to coarsely locate RGB corner points on the RGB image; S222, using the cornerSubPix algorithm to refine the corner points obtained from the coarse location on the RGB image to subpixel precision, resulting in the first RGB corner point with subpixel-level coordinate accuracy. Thus, the first RGB corner point set is obtained. .
[0009] Furthermore, step S400 includes the following sub-steps: S410, processing the first RGB corner point set. The first RGB corner point in the middle The coordinates and the set of the first IR corner points First IR corner points in each of the middle Subpixel corner refinement is performed, and the corresponding second RGB corner is obtained after updating the corner coordinates. Second IR corner S420, for the second RGB corner point Second IR corner Perform a consistency check, remove corner points that do not meet the consistency requirements, and keep the remaining second RGB corner points. Second IR corner Form the second RGB corner point set respectively Second IR corner set .
[0010] Furthermore, in step S410, the first RGB corner point set is... The first RGB corner point in the middle The coordinates and the set of the first IR corner points First IR corner points in each of the middle The method for sub-pixel corner refinement is as follows: in the first RGB corner set The first RGB corner point in the middle In the image domain, and the first IR corner point set First IR corner points in each of the middle In the image domain, one-dimensional edge fitting is performed using the local coordinate system of the corner points to obtain the corresponding second RGB corner points. Second IR corner .
[0011] Furthermore, in step S420, the second RGB corner point... Second IR corner The consistency check includes the following sub-steps: Sa, calculate each second IR corner point separately. Second RGB corner point IR corners of the corresponding calibration plate and calibration board RGB corner points The local gradient direction angle between them; if it is greater than a predetermined direction angle threshold, then the second IR corner point is determined. Or the second RGB corner point The local gradient direction angle does not meet the requirements; Sb, calculate each second RGB corner point respectively. With the corresponding RGB corner points of the calibration board residuals between and each second IR corner point IR corners of the corresponding calibration plate The residuals between If residual or residual If the residual value is greater than a predetermined residual threshold, then the second RGB corner point is determined. Or the second IR corner The residual does not meet the requirements; Sc, the second RGB corner point where the local gradient direction angle does not meet the requirements. Second IR corner And the second RGB corner point where the residual does not meet the requirements Second IR corner All were determined to be corner points that did not meet the consistency requirements.
[0012] Furthermore, in the Sa step, each second IR corner point and the second RGB corner point are calculated. IR corners of the corresponding calibration plate and calibration board RGB corner points The local gradient direction angle between them includes the following steps: Sa01, taking a w×w window at the target corner point; the target corner point is the second IR corner point where the local gradient direction needs to be calculated. Second RGB corner point Sa02. Perform Gaussian smoothing on the image within the window; Sa03. Calculate the gradient Gx in the x-direction and the gradient Gy in the y-direction at the target corner point using the Scharr operator or the Sobel operator, respectively; Sa04. Calculate the local gradient direction θ at the target corner point using the following formula: θ = atan2(Gy, Gx), where atan2() is the arctangent function; Sa05. Use the local gradient direction θ of each target corner point as its relationship with the corresponding IR corner point of the calibration plate. Or calibration board RGB corner points The angle between the local gradient directions.
[0013] Furthermore, in step Sb, the second RGB corner point is calculated. With the corresponding RGB corner points of the calibration board The residuals between The formula is as follows:
[0014] Calculate the second IR corner point IR corners of the corresponding calibration plate The residuals between The formula is as follows:
[0015] in, Indicates the second RGB corner point x-axis coordinate; Indicates the RGB corner points of the calibration board x-axis coordinate; Indicates the second RGB corner point y-axis coordinate; Indicates the RGB corner points of the calibration board y-axis coordinate; Indicates the second IR corner point x-axis coordinate; Indicates the IR corner of the calibration plate x-axis coordinate; Indicates the second IR corner point y-axis coordinate; Indicates the IR corner of the calibration plate The y-axis coordinate.
[0016] Furthermore, the directional angle threshold is 10° to 15°, and the residual threshold is 1 to 1.5 pixels.
[0017] In this invention, the homography matrix mapped from the plate coordinate domain to the RGB image is estimated. The parameters in the image, and the homography matrix mapped from the plate coordinate domain to the IR image. The parameters in the image can be used to obtain the mapping relationship between the cross-modal overlay of RGB and IR images, thereby achieving image fusion of RGB and IR images. By obtaining isothermal boundaries and performing least-squares line fitting with two sets of approximately orthogonal isothermal boundaries, and by forcing collinearity to remove interfering corner points and performing sub-pixel refinement within the patch, the accuracy of IR corner detection and the precision of corner coordinates can be improved. Sub-pixel corner refinement can further improve the precision of corner coordinates. After removing corners with large deviations through consistency checks, the edges are re-filled, and the homography matrix is re-estimated. Homography matrix The parameters in the model can improve the mapping accuracy between the plate coordinate domain, RGB image and IR image, and achieve high-precision image fusion of RGB image and IR image. Attached Figure Description
[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 is a flowchart of an embodiment of the image fusion correction method for an RGB camera and a thermal imaging camera according to the present invention.
[0019] Figure 2 is a schematic diagram of the structure of the checkerboard correction plate.
[0020] Figure 3 is a flowchart for obtaining the first IR corner point set. Detailed Implementation
[0021] The following specific examples illustrate the implementation of the present invention. The illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0022] Please refer to Figure 1, which is a flowchart of an embodiment of the image fusion correction method using an RGB camera and a thermal imaging camera according to the present invention. The image fusion correction method using an RGB camera and a thermal imaging camera in this embodiment includes the following steps: S100, using an RGB camera and a thermal imaging camera respectively to photograph a checkerboard correction plate with a constant-temperature heat source, obtaining an RGB image and an IR image. The RGB image is the image captured by the RGB camera, which can be a zoom camera, a wide-angle camera, or other visible light camera. The IR image is the thermal imaging image, which is the image captured by the thermal imaging camera, which can be an infrared camera.
[0023] The corner points of the chessboard are the RGB corner points of the calibration board. The center point of the constant temperature heat source is the IR corner point of the calibration plate. The constant-temperature heat sources are arranged at equal intervals in both the horizontal and vertical directions, and each of the calibration plate IR corner points... All are aligned with the RGB corner points of a calibration board. The constant temperature heat source can be a heating coil whose temperature remains constant after stabilizing under a fixed current.
[0024] Please refer to Figure 2, which is a structural diagram of a checkerboard calibration plate. The checkerboard calibration plate has an 8×14 checkerboard pattern, and the corner points of the 8×14 checkerboard pattern are the RGB corner points of the calibration plate. A constant-temperature heat source is set up on an 8×14 checkerboard pattern in a 4×7 array (i.e., the positions marked "×" in Figure 2, including: A1~G1, A2~G2, A3~G3, A4~G4). The spacing between adjacent constant-temperature heat sources is two grids, and the center point of each constant-temperature heat source (i.e., the IR corner point of the calibration plate) is... All are aligned with the RGB corner points of a calibration board. coincide.
[0025] This step may include the following sub-steps: S110, heating each constant-temperature heat source of the checkerboard to the target temperature and waiting for a period of time. Generally, it takes 3 to 5 seconds, but if high precision is required, it can be waited until the temperature change is <1% / min.
[0026] S120: Use an RGB camera and a thermal imaging camera to take one frame of image with fixed exposure as RGB image and IR image respectively; or use an RGB camera and a thermal imaging camera to take multiple frames of image with fixed exposure, and perform time-domain averaging and noise reduction processing to take RGB image and IR image respectively.
[0027] An RGB image can be obtained by capturing a single frame with a fixed exposure using an RGB camera, and an IR image can be obtained by capturing a single frame with a fixed exposure using a thermal imaging camera. Alternatively, when conditions permit, multiple frames can be captured using an RGB camera with a fixed exposure, preferably 8 to 16 frames, and then subjected to temporal averaging noise reduction (calculating only the mean, without other complex preprocessing) before being used as the RGB image. Similarly, an IR image can be obtained by capturing multiple frames with a thermal imaging camera with a fixed exposure, preferably 8 to 16 frames, and then subjected to temporal averaging noise reduction (calculating only the mean, without other complex preprocessing).
[0028] S200. Obtain the corner coordinates from the RGB and IR images respectively to obtain the first RGB corner set. and the first IR corner set .
[0029] Please refer to Figure 3. In this step, the corner coordinates are obtained on the IR image to obtain the first IR corner set. The method may include the following sub-steps: S211, Calculate the isothermal boundaries of each isothermal heat source on the IR image. The specific method may be as follows: First, use the Otsu algorithm to automatically select a threshold on the IR image (the Otsu algorithm can obtain the optimal segmentation threshold based on the grayscale histogram), and then perform binarization processing on the IR image according to the threshold to obtain a binarized image.
[0030] Then, the Canny algorithm is used to extract the boundary of the binary temperature image, which is the pixel-level edge corresponding to the "isotherm", and this is used as the isothermal boundary of the constant temperature heat source.
[0031] S212, According to the IR corner points of the calibration plate The location divides the chessboard into multiple 2×2 regions (i.e., each region has four IR corner points of the correction plate that form a 2×2 array). In each of the 2×2 regions, least squares straight line fitting is performed using two sets of approximately orthogonal isothermal boundaries (i.e., one set of isothermal boundaries is the isothermal boundaries corresponding to two heat sources in the same row, and the other set of isothermal boundaries is the isothermal boundaries corresponding to two heat sources in the same column), and the intersection point is used as its initial temperature corner point.
[0032] S213. Perform linear fitting and monotonic sorting on each initial temperature corner point according to row and column respectively. Monotonic sorting means sorting the corner points in the same row by x-coordinate and the corner points in the same column by y-coordinate to ensure physical order. Since the initial temperature corner points may not be strictly distributed on a straight line, linear fitting is required. The specific method is as follows: group the corner points by row and column, that is: group the corner points in the same row (corner points with equal or close y-coordinates) as a group, and fit a straight line to them to obtain the equation of the line for that row; group the corner points in the same column (corner points with equal or close x-coordinates) as a group, and fit a straight line to them to obtain the equation of the line for that column.
[0033] S214. By forcing collinearity, outlier initial temperature corner points that are not collinear are eliminated, and the remaining initial temperature corner points are used as temperature identification corner points. The specific method is as follows: After fitting a straight line, the distance from each initial temperature corner point to the corresponding row and column lines is calculated. If the distance exceeds the distance threshold (e.g., 0.5 pixels), it is considered an outlier and is eliminated. Then, the remaining initial temperature corner points are used to refit the straight line, and the process is repeated iteratively until all remaining initial temperature corner points satisfy the collinearity constraint (i.e., the distance from the initial temperature corner point to the corresponding row and column lines does not exceed the distance threshold). This step is to remove mismatched corner points and ensure that the points strictly satisfy the chessboard structure of the straight line arrangement.
[0034] S215. By performing sub-pixel refinement on each temperature identification corner point within a predetermined size patch (e.g., 15×15) using quadratic fitting or Gaussian kernel fitting, the first IR corner point with sub-pixel accuracy in coordinates is obtained. Thus, the first IR corner point set is obtained. The corner positions obtained through step S212 are integers or preliminary sub-pixel positions, but they are still not precise enough. Therefore, it is necessary to improve the accuracy of the coordinates. The specific method is as follows: First, take a small window of 15×15 pixels around each corner point; within this small window, fit the gray-level distribution near the corner point using a quadratic surface, or use a Gaussian kernel for fitting (treating the corner point as the peak of the Gaussian surface). Then, find the sub-pixel coordinates of the peak points through fitting (usually using quadratic fitting to find the extrema or Gaussian fitting to find the center). This yields high-precision sub-pixel corner coordinates that satisfy collinearity constraints and have high accuracy.
[0035] In step S200, corner coordinates are obtained from the RGB image to obtain the first RGB corner set. The method may include the following sub-steps: S221, using the findChessboardCorners algorithm to coarsely locate RGB corner points on the RGB image. The findChessboardCorners algorithm automatically detects and locates the initial positions of corner points in the chessboard grid in the image, achieving pixel-level accuracy.
[0036] S222. The cornerSubPix algorithm is used to refine the corner points obtained by coarse localization on the RGB image to subpixel level, resulting in the first RGB corner point with subpixel level coordinate precision. Thus, the first RGB corner point set is obtained. The specific principles of the findChessboardCorners algorithm and the cornerSubPix algorithm are existing technologies and will not be elaborated here.
[0037] S300, based on the first RGB corner point set The first RGB corner point in the middle The corresponding RGB corner points of the calibration board in the coordinates and board coordinate domain (i.e., the coordinate domain of the checkerboard calibration board). Coordinate estimation of the homography matrix mapping from the plate coordinate domain to the RGB image The parameters in the data. Based on the first IR corner point set. First IR corner points in each of the middle The coordinates and the corresponding IR corner points of the calibration plate in the plate coordinate domain Coordinate estimation of the homography matrix mapped from the plate coordinate domain to the IR image The parameters in.
[0038] Points on two different planes can be mapped using a homography matrix. Therefore, a mapping relationship can also be established between the plate coordinate domain and images (including RGB and IR images), as shown in the following expression:
[0039] Where x and y are plate coordinates (Z=0 can be ignored); u and v are pixel coordinates of the image (including RGB and IR images); s is the scaling factor; and H is the homography matrix, which is a 3×3 matrix with 8 degrees of freedom (because it can be scaled as a whole). When u and v are pixel coordinates of an RGB image, H is the homography matrix. When u and v are pixel coordinates of an IR image, H is the homography matrix. Therefore, through the first RGB corner point set The first RGB corner point in the middle The coordinates and the corresponding RGB corner points of the calibration board in the board coordinate domain The homography matrix can be estimated from the coordinates. The parameters. Passed through the first IR corner point set. First IR corner points in each of the middle The coordinates and the corresponding IR corner points of the calibration plate in the plate coordinate domain The homography matrix can be estimated from the coordinates. The parameters.
[0040] S400, for the first RGB corner point set and the first IR corner set Each corner point in the image undergoes sub-pixel corner point refinement, followed by a consistency check to remove corner points that do not meet the consistency requirements, forming a second RGB corner point set. Second IR corner set .
[0041] This step may include the following sub-steps: S410, for the first RGB corner point set The first RGB corner point in the middle The coordinates and the set of the first IR corner points First IR corner points in each of the middle Subpixel corner refinement is performed, and the corresponding second RGB corner is obtained after updating the corner coordinates. Second IR corner .
[0042] In this step, the first RGB corner point set The first RGB corner point in the middle The coordinates and the set of the first IR corner points First IR corner points in each of the middle The method for sub-pixel corner refinement is as follows: in the first RGB corner set The first RGB corner point in the middle In the image domain, and the first IR corner point set First IR corner points in each of the middle In the image domain, one-dimensional edge fitting is performed using the local coordinate system of the corner points to obtain the corresponding second RGB corner points. Second IR corner .
[0043] One-dimensional edge fitting may specifically include the following sub-steps: S411, for each first RGB corner point In the RGB image, samples are taken along the u and v directions of its local coordinate system to obtain grayscale profiles along the u direction and along the v direction.
[0044] S412. Fit a one-dimensional edge model based on the grayscale profiles along the u direction and the v direction respectively; at the corner points of the checkerboard, there should be step edges (black and white boundaries) in both directions. Commonly used one-dimensional edge fitting models include: error function (erf) model, hyperbolic tangent model and linear combination model, etc.
[0045] S413. The sub-pixel offsets Δu and Δv of the corner point in the u and v directions are obtained by least squares fitting. The coordinates of the corner point are updated according to the sub-pixel offsets in the u and v directions to obtain the corresponding second RGB corner point. .
[0046] For the first IR corner point The second IR corner point is obtained by performing one-dimensional edge fitting. The method and the first RGB corner point The same applies, so I will not elaborate further here.
[0047] One-dimensional edge fitting achieves high accuracy, reaching 0.05 pixels or even higher. It utilizes the structural prior of the checkerboard pattern (known edge directions) and fits along the edge directions, conforming to physical reality; it uses a more accurate edge model, with the edge model (ERF / sigmoid) more closely resembling the true edge than quadratic fitting; it reduces the number of fitting parameters by one dimension, resulting in greater stability; and it uses least-squares fitting for each profile to suppress noise.
[0048] This method is particularly suitable for checkerboard patterns in infrared images, because infrared images may be blurry and have low contrast, but structural priors can greatly improve positioning accuracy.
[0049] S420, for the second RGB corner point Second IR corner Perform a consistency check, remove corner points that do not meet the consistency requirements, and keep the remaining second RGB corner points. Second IR corner Form the second RGB corner point set respectively Second IR corner set .
[0050] In this step, the second RGB corner point Second IR corner The consistency check includes the following sub-steps: Sa, calculate each second IR corner point separately. Second RGB corner point IR corners of the corresponding calibration plate and calibration board RGB corner points The local gradient direction between them is determined, and the IR corner points are calculated. With corresponding RGB corner points The angle between the local gradient directions. If it is greater than a predetermined angle threshold, then the second IR corner is determined. Or the second RGB corner point The local gradient direction angle does not meet the requirements. The threshold for the direction angle is 10° to 15°, preferably 15°.
[0051] In the Sa step, each second IR corner point and the second RGB corner point are calculated. IR corners of the corresponding calibration plate and calibration board RGB corner points The local gradient direction angle between them includes the following steps: Sa01, taking a w×w window at the target corner point; the target corner point is the second IR corner point where the local gradient direction needs to be calculated. Second RGB corner point The optimal value range for w is 11 to 17, as this window size balances local accuracy and noise resistance.
[0052] Sa02. Perform Gaussian smoothing (filtering) on the image within the window to suppress noise, reduce detail interference, and avoid gradient calculation being too sensitive to noise.
[0053] Sa03. Use the Scharr operator or the Sobel operator to calculate the gradient Gx in the x-direction and the gradient Gy in the y-direction at the target corner point, respectively; the calculation method is current technology and will not be elaborated here.
[0054] Sa04. Calculate the local gradient direction θ of the target corner point. The calculation formula is as follows: θ = atan2(Gy, Gx) where atan2() is the arctangent function.
[0055] Sa05, Calculate each second IR corner point Second RGB corner point IR corners of the corresponding calibration plate and calibration board RGB corner points The angle between the local gradient directions is calculated using the following formula:
[0056]
[0057] Where the subscript i represents the second IR corner point Index; Represents the i-th second IR corner point IR corners of the corresponding calibration plate The local gradient direction between them; Represents the i-th second IR corner point The local gradient direction; Indicates the IR corner point of the i-th calibration plate The local gradient direction.
[0058] The subscript j indicates the second RGB corner point. Index; Represents the j-th second RGB corner point With the corresponding RGB corner points of the calibration board The local gradient direction between them; Represents the j-th second RGB corner point The local gradient direction; Represents the RGB corner point of the j-th calibration board. The local gradient direction.
[0059] Due to the IR corner of the calibration plate Local gradient direction and RGB corner points of the calibration plate The local gradient directions are all 0; therefore, it can be directly... The local gradient direction θ of each target corner point is used as its relationship with the corresponding IR corner point of the calibration plate. Or calibration board RGB corner points The angle between the local gradient directions. That is:
[0060]
[0061] Sb, calculate each of the second RGB corner points respectively. With the corresponding RGB corner points of the calibration board The residuals between and each second IR corner point IR corners of the corresponding calibration plate The residuals between If residual or residual If the residual value is greater than a predetermined residual threshold, then the second RGB corner point is determined. Or the second IR corner The residual does not meet the requirements. The residual threshold is 1 to 1.5 pixels.
[0062] In step Sb, the second RGB corner point is calculated. With the corresponding RGB corner points of the calibration board The residuals between The formula is as follows:
[0063] Calculate the second IR corner point IR corners of the corresponding calibration plate The residuals between The formula is as follows:
[0064] in, Indicates the second RGB corner point x-axis coordinate; Indicates the RGB corner points of the calibration board x-axis coordinate; Indicates the second RGB corner point y-axis coordinate; Indicates the RGB corner points of the calibration board y-axis coordinate; Indicates the second IR corner point x-axis coordinate; Indicates the IR corner of the calibration plate x-axis coordinate; Indicates the second IR corner point y-axis coordinate; Indicates the IR corner of the calibration plate The y-axis coordinate.
[0065] Sc, the second RGB corner point where the angle between the local gradient directions does not meet the requirements. Second IR corner And the second RGB corner point where the residual does not meet the requirements Second IR corner All were determined to be corner points that do not meet the consistency requirements. That is, a corner point is determined to be a corner point that does not meet the consistency requirements if either the angle between the local gradient directions or the residual does not meet the requirements.
[0066] S500, Set the second RGB corner points Second IR corner set The missing corner points are filled in to obtain the third RGB corner point set and the third IR corner point set. The homography matrix is then re-estimated based on the corner points in the third RGB corner point set and the third IR corner point set. Homography matrix The parameters in.
[0067] In this step, interpolation can be used to set the second RGB corner points. Second IR corner set To complete the missing corner points, the specific method is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Second IR corner set The corner points in the same row and column are subjected to a second fitting. Based on the distribution ratio of the corner points, the coordinates of the missing corner points are calculated, thereby filling in the missing corner points.
[0068] The homography matrix was re-estimated based on the corner points in the third RGB corner point set and the third IR corner point set. Homography matrix The method for calculating the parameters is the same as in step S300. The homography matrix can be obtained through two estimations. Homography matrix The parameters in the calibration process achieve high accuracy, meeting the requirements for cross-modal overlay of RGB and IR images, thus allowing the calibration to be completed.
[0069] S600, According to the homography matrix Homography matrix Establish the mapping relationship from the plate coordinate domain to the RGB image and the mapping relationship from the plate coordinate domain to the IR image, obtain the mapping relationship of the cross-modal overlay of the RGB image and the IR image, realize the image fusion of the RGB image and the IR image, generate a synthetic image that contains both thermal information and texture details, and realize pseudo-color fusion.
[0070] In this embodiment, the homography matrix mapped from the plate coordinate domain to the RGB image is estimated. The parameters in the image, and the homography matrix mapped from the plate coordinate domain to the IR image. The parameters in the image can be used to obtain the mapping relationship between the cross-modal overlay of RGB and IR images, thereby achieving image fusion of RGB and IR images. By obtaining isothermal boundaries and performing least-squares line fitting with two sets of approximately orthogonal isothermal boundaries, and by forcing collinearity to remove interfering corner points and performing sub-pixel refinement within the patch, the accuracy of IR corner detection and the precision of corner coordinates can be improved. Sub-pixel corner refinement can further improve the precision of corner coordinates. After removing corners with large deviations through consistency checks, the edges are re-filled, and the homography matrix is re-estimated. Homography matrix The parameters in the model can improve the mapping accuracy between the plate coordinate domain, RGB image and IR image, and achieve high-precision image fusion of RGB image and IR image.
[0071] The above embodiments merely illustrate preferred implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention should be determined by the appended claims.
Claims
1. An image fusion correction method for an RGB camera and a thermal imaging camera, characterized in that, The process includes the following steps: S100, using an RGB camera and a thermal imaging camera to photograph the checkerboard calibration board with a constant-temperature heat source, respectively, to obtain RGB images and IR images; the corner points of the checkerboard are the RGB corner points of the calibration board. The center point of the constant temperature heat source is the IR corner point of the calibration plate. The constant-temperature heat sources are arranged at equal intervals in both the horizontal and vertical directions, and each of the calibration plate IR corner points All are aligned with the RGB corner points of a calibration board. Overlap; S200, obtain the corner coordinates on the RGB image and IR image respectively to obtain the first RGB corner set. and the first IR corner set S300, based on the first RGB corner point set The first RGB corner point in the middle The coordinates and the corresponding RGB corner points of the calibration board in the board coordinate domain Coordinate estimation of the homography matrix mapping from the plate coordinate domain to the RGB image The parameters in the first IR corner point set are based on the parameters in the first IR corner point set. First IR corner points in each of the middle The coordinates and the corresponding IR corner points of the calibration plate in the plate coordinate domain The coordinate estimation is the homography matrix mapped from the plate coordinate domain to the IR image. The parameters in; S400, for the first RGB corner point set and the first IR corner set Each corner point in the image undergoes sub-pixel corner point refinement, followed by a consistency check to remove corner points that do not meet the consistency requirements, forming a second RGB corner point set. Second IR corner set S500, Set the second RGB corner points Second IR corner set The missing corner points are filled in to obtain the third RGB corner point set and the third IR corner point set. The homography matrix is then re-estimated based on the corner points in the third RGB corner point set and the third IR corner point set. Homography matrix The parameters in S600, based on the homography matrix. Homography matrix Establish the mapping relationship between the plate coordinate domain and the RGB image, as well as the mapping relationship between the plate coordinate domain and the IR image, to achieve image fusion of RGB and IR images.
2. The image fusion correction method for RGB cameras and thermal imaging cameras as described in claim 1, characterized in that, The S100 step includes the following sub-steps: S110, heating each constant temperature heat source of the checkerboard to the target temperature and waiting for a period of time; S120, using an RGB camera and a thermal imaging camera to take one frame of image respectively with fixed exposure as RGB image and IR image; or using an RGB camera and a thermal imaging camera to take multiple frames of image respectively with fixed exposure, and performing time-domain averaging noise reduction processing to obtain RGB image and IR image.
3. The image fusion correction method for RGB cameras and thermal imaging cameras as described in claim 1, characterized in that, In step S200, corner coordinates are obtained on the IR image to obtain the first IR corner set. The method includes the following sub-steps: S211, Calculate the isothermal boundaries of each isothermal heat source on the IR image; S212, According to the IR corner points of the calibration plate The location divides the chessboard into multiple 2×2 regions (i.e., each region has four IR corner points of the correction plate that form a 2×2 array). In each of the 2×2 regions, least squares straight line fitting is performed on two sets of approximately orthogonal isothermal boundaries, and their intersection points are used as initial temperature corner points; S213, each initial temperature corner point is fitted with a straight line and monotonically sorted according to its row and column respectively; S214, by forcing collinearity, outlier initial temperature corner points that are not collinear are removed, and the remaining initial temperature corner points are used as temperature identification corner points; S215, each temperature identification corner point is sub-pixel refined within a predetermined size patch by quadratic fitting or Gaussian kernel fitting to obtain the first IR corner point with sub-pixel level accuracy in coordinates. Thus, the first IR corner point set is obtained. 。 4. The image fusion correction method for RGB cameras and thermal imaging cameras as described in claim 1, characterized in that, In step S200, corner coordinates are obtained from the RGB image to obtain the first RGB corner set. The method includes the following sub-steps: S221, using the findChessboardCorners algorithm to coarsely locate RGB corner points on the RGB image; S222, using the cornerSubPix algorithm to refine the corner points obtained from the coarse location on the RGB image to subpixel precision, resulting in the first RGB corner point with subpixel-level coordinate accuracy. Thus, the first RGB corner point set is obtained. 。 5. The image fusion correction method for RGB cameras and thermal imaging cameras as described in claim 1, characterized in that, The S400 step includes the following sub-steps: S410, processing the first RGB corner point set The first RGB corner point in the middle The coordinates and the set of the first IR corner points First IR corner points in each of the middle Subpixel corner refinement is performed, and the corresponding second RGB corner is obtained after updating the corner coordinates. Second IR corner S420, for the second RGB corner point Second IR corner Perform a consistency check, remove corner points that do not meet the consistency requirements, and leave the remaining second RGB corner points. Second IR corner Form the second RGB corner point set respectively Second IR corner set 。 6. The image fusion correction method for RGB cameras and thermal imaging cameras as described in claim 5, characterized in that, In step S410, the first RGB corner point set is... The first RGB corner point in the middle The coordinates and the set of the first IR corner points First IR corner points in each of the middle The method for sub-pixel corner refinement is as follows: in the first RGB corner set The first RGB corner point in the middle In the image domain, and the first IR corner point set First IR corner points in each of the middle In the image domain, one-dimensional edge fitting is performed using the local coordinate system of the corner points to obtain the corresponding second RGB corner points. Second IR corner 。 7. The image fusion correction method for RGB cameras and thermal imaging cameras as described in claim 5, characterized in that, In step S420, the second RGB corner point Second IR corner The consistency check includes the following sub-steps: Sa, calculate each second IR corner point separately. Second RGB corner point IR corners of the corresponding calibration plate and calibration board RGB corner points The local gradient direction angle between them; if it is greater than a predetermined direction angle threshold, then the second IR corner point is determined. Or the second RGB corner point The local gradient direction angle does not meet the requirements; Sb, calculate each second RGB corner point respectively. With the corresponding RGB corner points of the calibration board residuals between and each second IR corner point IR corners of the corresponding calibration plate residuals between If residual or residual If the residual value is greater than a predetermined residual threshold, then the second RGB corner point is determined. Or the second IR corner The residual does not meet the requirements; Sc, the second RGB corner point where the local gradient direction angle does not meet the requirements. Second IR corner And the second RGB corner point where the residual does not meet the requirements Second IR corner All were determined to be corner points that did not meet the consistency requirements.
8. The image fusion correction method for an RGB camera and a thermal imaging camera as described in claim 7, characterized in that, In the Sa step, each second IR corner point and the second RGB corner point are calculated. IR corners of the corresponding calibration plate and calibration board RGB corner points The local gradient direction angle between them includes the following steps: Sa01, taking a w×w window at the target corner point; the target corner point is the second IR corner point where the local gradient direction needs to be calculated. Second RGB corner point Sa02. Perform Gaussian smoothing on the image within the window; Sa03. Calculate the gradient Gx in the x-direction and the gradient Gy in the y-direction at the target corner point using the Scharr operator or the Sobel operator, respectively; Sa04. Calculate the local gradient direction θ at the target corner point using the following formula: θ = atan2(Gy, Gx), where atan2() is the arctangent function; Sa05. Use the local gradient direction θ of each target corner point as its relationship with the corresponding IR corner point of the calibration plate. Or calibration board RGB corner points The angle between the local gradient directions.
9. The image fusion correction method for an RGB camera and a thermal imaging camera as described in claim 7, characterized in that, In step Sb, the second RGB corner point is calculated. With the corresponding RGB corner points of the calibration board residuals between The formula is as follows: Calculate the second IR corner point IR corners of the corresponding calibration plate residuals between The formula is as follows: in, Indicates the second RGB corner point x-axis coordinate; Indicates the RGB corner points of the calibration board x-axis coordinate; Indicates the second RGB corner point y-axis coordinate; Indicates the RGB corner points of the calibration board y-axis coordinate; Indicates the second IR corner point x-axis coordinate; Indicates the IR corner of the calibration plate x-axis coordinate; Indicates the second IR corner point y-axis coordinate; Indicates the IR corner of the calibration plate The y-axis coordinate.
10. The image fusion correction method for an RGB camera and a thermal imaging camera as described in any one of claims 7 to 9, characterized in that, The directional angle threshold is 10° to 15°, and the residual threshold is 1 to 1.5 pixels.