Visible light and infrared telephoto image registration method

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a visible light and infrared long-focus image registration method, which comprises the following steps: acquiring a visible light image and an infrared image; generating a first angular point coordinate according to the visible light image, and generating a second angular point coordinate according to the infrared image; calculating a horizontal magnification factor and a vertical magnification factor according to the first angular point coordinate and the second angular point coordinate; zooming the infrared image to generate a zoomed image; extracting the visible light image to generate a visible light edge image, and extracting the reduced image to generate an infrared edge image; matching the visible light edge image and the infrared edge image to generate matching coordinates, and generating a to-be-processed image based on the matching coordinates; amplifying the to-be-processed image by using the super-resolution model to generate a super-resolution image; and scaling the super-resolution image to generate a registration image. According to the method, the problem of large difference between the view field and the scale of two types of images under the infrared telephoto lens is effectively solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly relates to a visible light and infrared long - focal - length image registration method. Background Art

[0002] Visible - light and infrared image registration is to match the corresponding points between two images so that their image contents are aligned. In related technologies, visible - light image and infrared - image registration methods are mainly based on region - based registration and feature - based registration. The region - based registration method estimates the translation relationship between two images by minimizing the total distance between pixel - correspondence relationships under a specific metric to achieve image alignment. The feature - based registration method extracts feature points from two images, then determines the corresponding coordinate relationships between them, and estimates the spatial transformation accordingly to achieve image alignment. When using an infrared long - focal - length lens, the field of view and scale differences between the infrared image and the visible - light image are large. The region - based registration method cannot estimate the scale relationship and is only applicable to the case where the scale differences between the two images are not large. In the case of large scale differences between the two images, the feature - based registration method will have the problem of image blurring after the image undergoes a large - scale spatial transformation.

[0003] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art.

[0004] Content of the Application

[0005] In view of at least one of the above technical problems, this application provides a visible - light and infrared long - focal - length image registration method.

[0006] This application provides a visible - light and infrared long - focal - length image registration method, including:

[0007] Obtain a visible - light image and an infrared image, where both the visible - light image and the infrared image are images taken of the same calibration board;

[0008] Generate first corner coordinates according to the visible - light image, where the first corner coordinates are the coordinates on the calibration board in the visible - light image, and generate second corner coordinates according to the infrared image, where the second corner coordinates are the coordinates on the calibration board in the infrared image;

[0009] Calculate the horizontal magnification and the vertical magnification according to the first corner coordinates and the second corner coordinates;

[0010] Shrink the infrared image according to the horizontal magnification and the vertical magnification to generate a shrunk image;

[0011] Extract the visible - light image to generate a visible - light edge image, and extract the shrunk image to generate an infrared edge image;

[0012] Match the visible light edge image and the infrared edge image to generate matching coordinates, and generate the image to be processed based on the matching coordinates;

[0013] Use the super-resolution model to magnify the image to be processed to generate a super-resolution image;

[0014] Scale the super-resolution image to generate a registered image, and the resolution of the registered image is the same as that of the infrared image.

[0015] The visible light and infrared long focal length image registration method of the present application effectively solves the problem of large differences in the field of view and scale between the two types of images under an infrared long focal length lens by calculating the horizontal and vertical magnification factors through the calibration plate corner coordinates and scaling the infrared image, and makes up for the defect that the regional registration method cannot estimate the scale relationship; generates accurate matching coordinates through edge image matching, ensuring the accuracy of the feature correspondence relationship during the registration process; uses the super-resolution model to magnify the processed image, avoiding the problem of image blurring caused by large-scale spatial transformation, and improving the image details and clarity; finally generates a registered image with the same resolution as the infrared image, realizing high-precision alignment of cross-modal long focal length images.

[0016] In some possible implementation manners, generating the first corner coordinates includes: detecting the visible light image by using a corner detection algorithm to generate the first corner coordinates;

[0017] Generating the second corner coordinates includes: detecting the infrared image by using a corner detection algorithm to generate the second corner coordinates.

[0018] In some possible implementation manners, the calculation formula for the horizontal magnification factor is:

[0019]

[0020] where S X is the horizontal magnification factor, (x1, y1) is the coordinate of the upper left corner point of the calibration plate in the visible light image, (x2, y2) is the coordinate of the lower right corner point of the calibration plate in the visible light image, (x'1, y'1) is the coordinate of the upper left corner point of the calibration plate in the infrared image, and (x'2, y'2) is the coordinate of the lower right corner point of the calibration plate in the infrared image.

[0021] In some possible implementation manners, the calculation formula for the vertical magnification factor is:

[0022]

[0023] where S yLet \(M\) be the vertical magnification factor, \((x_1, y_1)\) be the coordinates of the upper left corner point of the calibration board in the visible light image, \((x_2, y_2)\) be the coordinates of the lower right corner point of the calibration board in the visible light image, \((x_1', y_1')\) be the coordinates of the upper left corner point of the calibration board in the infrared image, and \((x_2', y_2')\) be the coordinates of the lower right corner point of the calibration board in the infrared image.

[0024] In some possible implementation manners, generating a reduced image includes:

[0025] Performing double linear interpolation algorithm on the infrared image to reduce its size and generate a reduced image.

[0026] In some possible implementation manners, generating a visible light edge image includes: Extracting the visible light image using the canny algorithm to generate a visible light edge image;

[0027] Generating an infrared edge image includes: Extracting the reduced image using the canny algorithm to generate an infrared edge image.

[0028] In some possible implementation manners, generating an image to be processed includes:

[0029] According to the horizontal magnification factor, vertical magnification factor, width of the infrared image, and height of the infrared image, generating the width of the image to be processed and the height of the image to be processed;

[0030] Using the matching coordinates as the upper left vertex and generating the image to be processed according to the width of the image to be processed and the height of the image to be processed.

[0031] In some possible implementation manners, generating a super-resolution image includes:

[0032] Normalizing and performing channel transformation on the image to be processed to generate a low-resolution visible light image;

[0033] Inputting the low-resolution visible light image into a trained super-resolution model to generate a super-resolution image; wherein, the super-resolution model has a preset loss function.

[0034] In some possible implementation manners, the super-resolution model includes several alternately arranged convolutional layers and activation functions.

[0035] In some possible implementation manners, generating a registered image includes:

[0036] Scaling the super-resolution image using the double linear interpolation algorithm to generate a registered image.

[0037] The present application will be further described below in conjunction with the accompanying drawings and embodiments. Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 is a schematic flowchart of a visible light and infrared long - focal - length image registration method provided by an embodiment of the present application; Detailed implementation manners

[0040] To make the above - mentioned objects, features, and advantages of the present application more obvious and understandable, the following will give a detailed description of the specific implementation manners of the present application with reference to the drawings. Many specific details are set forth in the following description to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein. Those skilled in the art can make similar improvements without departing from the connotation of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0041] As Figure 1 shown, this embodiment provides a visible light and infrared long - focal - length image registration method, including: step S100 to step S800.

[0042] Step S100, obtain a visible light image and an infrared image, where both the visible light image and the infrared image are images of the same calibration board.

[0043] In step S100, a calibration board is fixedly placed, and this calibration board is a checkerboard calibration board. The calibration board is photographed by an infrared thermal imager to obtain a visible light image and an infrared image.

[0044] Step S200, generate first corner coordinates according to the visible light image, where the first corner coordinates are the coordinates on the calibration board in the visible light image, and generate second corner coordinates according to the infrared image, where the second corner coordinates are the coordinates on the calibration board in the infrared image.

[0045] In this step S200, use a corner detection algorithm to detect the visible light image to generate first corner coordinates; use a corner detection algorithm to detect the infrared image to generate second corner coordinates.

[0046] It can be understood that the corner detection algorithm can include but is not limited to: Harris corner detection algorithm, Shi - Tomasi corner detection algorithm, FAST corner detection algorithm, and SIFT (Scale - Invariant Feature Transform) corner detection algorithm.

[0047] It can be understood that the first corner point coordinates may include, but are not limited to, the coordinates of the upper left corner point of the calibration plate in the visible light image and the coordinates of the lower right corner point of the calibration plate in the visible light image. The second corner point coordinates may include, but are not limited to, the coordinates of the upper left corner point of the calibration plate in the infrared image and the coordinates of the lower right corner point of the calibration plate in the infrared image.

[0048] Step S300: Calculate the horizontal magnification and the vertical magnification according to the first corner point coordinates and the second corner point coordinates.

[0049] The calculation formula for the horizontal magnification is:

[0050]

[0051] where S X is the horizontal magnification, (x1, y1) are the coordinates of the upper left corner point of the calibration plate in the visible light image, (x2, y2) are the coordinates of the lower right corner point of the calibration plate in the visible light image, (x'1, y'1) are the coordinates of the upper left corner point of the calibration plate in the infrared image, and (x'2, y'2) are the coordinates of the lower right corner point of the calibration plate in the infrared image.

[0052] The calculation formula for the vertical magnification is:

[0053]

[0054] where S y is the vertical magnification, (x1, y1) are the coordinates of the upper left corner point of the calibration plate in the visible light image, (x2, y2) are the coordinates of the lower right corner point of the calibration plate in the visible light image, (x'1, y'1) are the coordinates of the upper left corner point of the calibration plate in the infrared image, and (x'2, y'2) are the coordinates of the lower right corner point of the calibration plate in the infrared image.

[0055] Step S400: Scale down the infrared image according to the horizontal magnification and the vertical magnification to generate a scaled-down image.

[0056] In this step S400, the bilinear interpolation algorithm is used to scale down the infrared image to generate a scaled-down image. Specifically, the horizontal direction of the infrared image is scaled down by the value of the horizontal magnification using the bilinear interpolation algorithm, and the vertical direction of the infrared image is scaled down by the value of the vertical magnification.

[0057] It can be understood that scaling down the infrared image according to the calculated horizontal magnification and vertical magnification makes the infrared image closer to the visible light image in scale. This can reduce the influence of scale differences in the subsequent registration process, improve the accuracy and efficiency of registration. At the same time, the scaled-down image also helps to reduce the computational complexity of subsequent processing.

[0058] Step S500: Extract the visible light image to generate a visible light edge image, and extract the scaled-down image to generate an infrared edge image.

[0059] In this step S500, use the canny algorithm to extract the visible light image to generate a visible light edge image; use the canny algorithm to extract the scaled-down image to generate an infrared edge image.

[0060] It can be understood that extracting the edge images of the visible light image and the scaled-down infrared image respectively can highlight the key features in the images and reduce the interference of irrelevant information. The edge images are more likely to find corresponding relationships in the subsequent matching process, improving the accuracy and reliability of the matching.

[0061] Step S600: Match the visible light edge image and the infrared edge image to generate matching coordinates, and generate a to-be-processed image based on the matching coordinates.

[0062] It can be understood that matching the visible light edge image and the infrared edge image to generate matching coordinates, and generating a to-be-processed image based on these coordinates. By matching the edge features, the positional relationship of the corresponding objects in the two images can be found, thus realizing the preliminary alignment of the images. The to-be-processed image provides an accurate basis for the subsequent super-resolution processing, ensuring that the super-resolution model can operate on the correct image content.

[0063] Step S700: Use the super-resolution model to magnify the to-be-processed image to generate a super-resolution image.

[0064] It can be understood that when processing images with large-scale differences, traditional registration methods are prone to cause image blurring. Using the super-resolution model to magnify the to-be-processed image can restore the detailed information of the image while magnifying the image, improving the clarity and quality of the image. The super-resolution model can learn the high-frequency information in the image, make up for the details lost during the large-scale transformation, and avoid the problem of image blurring that occurs in the feature-based registration method after the large-scale transformation.

[0065] Step S800: Scale the super-resolution image to generate a registered image, and the resolution of the registered image is the same as that of the infrared image.

[0066] In this step S800, use the bilinear interpolation algorithm to scale the super-resolution image to generate a registered image.

[0067] It can be understood that scaling the super-resolution image to the same resolution as the infrared image makes the final registered image consistent with the original infrared image in terms of resolution. This ensures the practicality and compatibility of the registered image.

[0068] The visible light and infrared long - focal - length image registration method of this embodiment calculates the horizontal and vertical magnification factors through the corner coordinates of the calibration board and scales down the infrared image, effectively solving the problem of large differences in the fields of view and scales between the two types of images under an infrared long - focal - length lens, and making up for the defect that the region - based registration method cannot estimate the scale relationship; generates accurate matching coordinates through edge - image matching, ensuring the accuracy of the feature correspondence relationship during the registration process; uses a super - resolution model to magnify the processed image, avoiding the image blurring problem caused by large - scale spatial transformation and improving the image details and clarity; finally generates a registered image with the same resolution as the infrared image, achieving high - precision alignment of cross - modal long - focal - length images.

[0069] In some embodiments, generating the image to be processed includes:

[0070] Generate the width and height of the image to be processed according to the horizontal magnification factor, the vertical magnification factor, the width of the infrared image, and the height of the infrared image;

[0071] Use the following formula to calculate the width of the image to be processed:

[0072]

[0073] where, w represents the width of the image to be processed, ir w represents the width of the infrared image, and S X is the horizontal magnification factor.

[0074] Use the following formula to calculate the height of the image to be processed:

[0075]

[0076] where, h represents the height of the image to be processed, ir h represents the height of the infrared image, and S y is the vertical magnification factor.

[0077] Use the matching coordinates as the top - left vertex, and generate the image to be processed according to the width and height of the image to be processed.

[0078] It should be noted that after the infrared camera and the visible - light camera are installed in hardware, their relative positions remain unchanged. Therefore, after calibration, during actual use, there is no need for a checkerboard calibration board. Only according to the matching coordinates, the width and height of the image to be processed, the registration of visible - light images and infrared images in any scenario can be achieved according to steps S600 to S800.

[0079] In some embodiments, generating the super - resolution image includes:

[0080] Normalize the image to be processed and perform channel transformation to generate a low-resolution visible light image;

[0081] Input the low-resolution visible light image into the trained super-resolution model to generate a super-resolution image; among them, the super-resolution model has a preset loss function.

[0082] It can be understood that when the low-resolution visible light image is input into the trained super-resolution model, the low-resolution visible light image feature is represented as I low , and its input space can be represented as I low ∈R H×W×1 , where R represents the defined three-dimensional space, H and W are the height and width of the low-resolution visible light image respectively, and 1 represents the number of channels of the feature image. In addition, the super-resolution model includes several alternately arranged convolutional layers and activation functions.

[0083] Perform multiple convolution and activation operations on I low to extract image detail features, and map the input feature I low to F n . This process can be formalized as: F n =prelu(Conv 3×3 (F n-1 ))), where Conv 3×3 (·) represents a 3x3 convolution operation, and prelu(·) represents an activation function. Add the image detail features F n obtained from the low-resolution visible light image and I low pixel by pixel, and then obtain the final super-resolution image I sr through upsampling. This process can be formalized as: I sr =upsampler(I low +F n ), where upsampler represents an upsampling operation.

[0084] In addition, introduce the pixel-level loss L1 loss as the loss function of the super-resolution model. The L1 loss can be expressed as:

[0085]

[0086] Among them, N is the total number of pixels of the low-resolution visible light image, and L1 represents the average pixel interpolation.

[0087] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0088] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0089] Any process or method description shown in a flowchart or described in other ways herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0090] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definable sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then stored in a computer memory.

[0091] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0092] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments. The above-mentioned storage medium can be a read-only memory, a disk, or an optical disc, etc.

[0093] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0094] The above are only the preferred embodiments of the present application, and do not impose any formal restrictions on the present application. Any person skilled in the art can make many possible changes and modifications to the technical solution of the present application, or modify it into an equivalent embodiment with equivalent changes, without departing from the scope of the technical solution of the present application. Therefore, all equivalent changes made according to the shape, structure and principle of the present application without departing from the content of the technical solution of the present application shall be covered by the protection scope of the present application.

Claims

1. A visible light and infrared long - focal - length image registration method, characterized in that, Including: Obtain a visible light image and an infrared image, where both the visible light image and the infrared image are images taken of the same calibration board; Generate first corner coordinates according to the visible light image, where the first corner coordinates are the coordinates on the calibration board in the visible light image, and generate second corner coordinates according to the infrared image, where the second corner coordinates are the coordinates on the calibration board in the infrared image; Calculate the horizontal magnification and the vertical magnification according to the first corner coordinates and the second corner coordinates; Shrink the infrared image according to the horizontal magnification and the vertical magnification to generate a shrunk image; Extract the visible light image to generate a visible light edge image, and extract the shrunk image to generate an infrared edge image; Match the visible light edge image and the infrared edge image to generate matching coordinates, and generate a to-be-processed image based on the matching coordinates; Enlarge the to-be-processed image using a super-resolution model to generate a super-resolution image; Scale the super-resolution image to generate a registered image, where the resolution of the registered image is the same as the resolution of the infrared image.

2. The visible light and infrared long focal length image registration method according to claim 1, wherein The generating of the first corner coordinates includes: detecting the visible light image using a corner detection algorithm to generate the first corner coordinates; The generating of the second corner coordinates includes: detecting the infrared image using a corner detection algorithm to generate the second corner coordinates.

3. The visible light and infrared long focal length image registration method according to claim 1, characterized in that The calculation formula for the horizontal magnification is: where S X is the horizontal magnification factor, (x1, y1) are the coordinates of the upper left corner point of the calibration plate in the visible light image, (x2, y2) are the coordinates of the lower right corner point of the calibration plate in the visible light image, (x1', y1') are the coordinates of the upper left corner point of the calibration plate in the infrared image, and (x'2, y'2) are the coordinates of the lower right corner point of the calibration plate in the infrared image.

4. The visible light and infrared long focal length image registration method according to claim 1, characterized in that, The calculation formula for the vertical magnification is: Among them, S y is the vertical magnification factor, (x1, y1) is the coordinates of the upper left corner point of the calibration plate in the visible light image, (x2, y2) is the coordinates of the lower right corner point of the calibration plate in the visible light image, (x1', y1') is the coordinates of the upper left corner point of the calibration plate in the infrared image, and (x'2, y'2) is the coordinates of the lower right corner point of the calibration plate in the infrared image.

5. The visible light and infrared long focal length image registration method according to claim 1, wherein The generating of the shrunk image includes: Shrink the infrared image using bilinear interpolation algorithm to generate a shrunk image.

6. The visible light and infrared long focal length image registration method according to claim 1, characterized in that The generating of the visible light edge image includes: extracting the visible light image using the canny algorithm to generate a visible light edge image; The generating of the infrared edge image includes: extracting the shrunk image using the canny algorithm to generate an infrared edge image.

7. The visible light and infrared long focal length image registration method according to claim 1, wherein The generating of the to-be-processed image includes: Generate the width and height of the to-be-processed image according to the horizontal magnification, the vertical magnification, the width of the infrared image, and the height of the infrared image; Use the matching coordinates as the upper left vertex, and generate the to-be-processed image according to the width and height of the to-be-processed image.

8. The visible light and infrared long focal length image registration method according to claim 1, characterized in that The generating of the super-resolution image includes: Normalize and perform channel transformation on the to-be-processed image to generate a low-resolution visible light image; Input the low-resolution visible light image into a trained super-resolution model to generate a super-resolution image; where the super-resolution model has a preset loss function.

9. The visible light and infrared long focal length image registration method according to claim 8, wherein The super-resolution model includes several alternately arranged convolutional layers and activation functions.

10. The visible light and infrared long focal length image registration method according to claim 1, characterized in that The generating of the registered image includes: Scale the super-resolution image using bilinear interpolation algorithm to generate a registered image.

Images