Image Registration Method, Device, User Terminal and Medium Based on Edge Features

Abstract

The present invention belongs to the technical field of infrared imager image processing, and particularly relates to an image registration method, device, user terminal and medium based on edge features, including: acquiring a visible light image and an infrared image of a target object; performing primary registration on the visible light image and the infrared image to generate an initial visible light registration image; generating an initial visible light registration extended image based on the initial visible light registration image; extracting first edge features from the initial visible light registration extended image; extracting second edge features from the infrared image; determining registration position coordinates based on the first edge features and the second edge features, and completing the final registration of the visible light image and the infrared image based on the registration position coordinates. This image registration method can extract relatively accurate features in the case of low resolution of the infrared image, thereby realizing accurate registration of the infrared image and the visible light image, and effectively ensuring the subsequent smooth fusion of the infrared image and the visible light image.

Description

Technical Field

[0001] This application relates to the technical field of infrared thermal imager image registration, and particularly relates to an image registration method, device, user terminal, and medium based on edge features. Background Art

[0002] With the continuous development of infrared thermal imaging technology, infrared thermal imagers have been widely used in many fields such as military and civilian. Due to the common defects of infrared images such as blurred edges, low contrast, and high noise, infrared image enhancement algorithms have been widely studied to further improve the imaging effect of infrared thermal imaging technology.

[0003] In the existing infrared image enhancement algorithm technology, when automatically registering visible light images and infrared images, a feature point matching method is adopted, that is, feature points on the visible light image and feature points on the infrared image are extracted respectively, and then they are matched one by one.

[0004] The applicant found during use that due to the low resolution of infrared images, it is difficult to extract accurate feature points, which in turn leads to poor registration effect and low accuracy between visible light images and infrared images, affecting the subsequent fusion of visible light images and infrared images.

[0005] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] In view of at least one of the above technical problems, this application provides an image registration method, device, user terminal, and medium based on edge features, which solves the problem that due to the low resolution of infrared images, it is difficult to extract accurate feature points, resulting in poor registration effect and low accuracy between visible light images and infrared images, and affecting the subsequent fusion of visible light images and infrared images.

[0007] An embodiment of the first aspect of this application provides an image registration method based on edge features, which is applied to an infrared thermal imager. The infrared thermal imager includes an infrared thermal imaging camera and a visible light camera, and includes:

[0008] Obtain a visible light image and an infrared image of a target object;

[0009] Perform primary registration on the visible light image and the infrared image to generate an initial visible light registration image;

[0010] Generate an initial visible light registration extended image based on the initial visible light registration image;

[0011] Extract first edge features from the initial visible light registration extended image;

[0012] Extract the second edge feature from the infrared image;

[0013] Based on the first edge feature and the second edge feature, determine the registration position coordinates, and based on the registration position coordinates, complete the final registration of the visible light image and the infrared image.

[0014] The present application has the following technical effects: This image registration method can extract relatively accurate features in the case of low resolution of the infrared image, so as to realize the precise registration of the infrared image and the visible light image, and effectively ensure the subsequent smooth fusion of the infrared image and the visible light image.

[0015] In one implementation, perform an initial registration on the visible light image and the infrared image to generate an initial visible light registration image, including:

[0016] Obtain the visible light rotation matrix and the visible light translation matrix of the visible light camera;

[0017] Obtain the infrared rotation matrix and the infrared translation matrix of the infrared thermal imaging camera;

[0018] According to the visible light rotation matrix and the infrared rotation matrix, and based on the first formula, generate a rotation mapping matrix;

[0019] According to the visible light translation matrix, the infrared translation matrix and the rotation mapping matrix, and based on the second formula, generate a translation mapping matrix;

[0020] Obtain the visible light internal reference matrix of the visible light camera, the infrared internal reference matrix of the infrared thermal imaging camera, and the horizontal and vertical coordinates of the infrared image;

[0021] According to the rotation mapping matrix, the translation mapping matrix, the visible light internal reference matrix, the infrared internal reference matrix and the horizontal and vertical coordinates of the infrared image, and based on the third formula, generate the horizontal and vertical coordinates of the visible light image;

[0022] Generate an initial visible light registration image according to the horizontal and vertical coordinates of the visible light image.

[0023] In one implementation, after generating the initial visible light registration image according to the horizontal and vertical coordinates of the visible light image, it includes:

[0024] Obtain the horizontal and vertical coordinates, width value and height value of the upper left corner of the initial visible light registration image;

[0025] Generate an initial visible light registration extended image according to the horizontal and vertical coordinates, width value, height value of the upper left corner of the initial visible light registration image and the expansion value.

[0026] In one implementation, the extracting the first edge feature from the initial visible light registration extended image further includes:

[0027] Gray-scale the initial registered extended visible light image to generate a gray-scale image of the initial registered extended visible light image;

[0028] Calculate the visible light X-direction gradient value and the visible light Y-direction gradient value based on the gray-scale image of the initial registered extended visible light image;

[0029] Generate visible light edge features based on the visible light X-direction gradient value and the visible light Y-direction gradient value;

[0030] Generate the first edge feature based on the visible light edge features and perform binarization and filtering processing.

[0031] In one implementation, the extracting the second edge feature from the infrared image further includes:

[0032] Gray-scale the infrared image to generate a gray-scale image of the infrared image;

[0033] Calculate the infrared X-direction gradient value and the infrared Y-direction gradient value based on the gray-scale image of the infrared image;

[0034] Generate infrared edge features based on the infrared X-direction gradient value and the infrared Y-direction gradient value;

[0035] Generate the second edge feature based on the infrared edge features and perform binarization and filtering processing.

[0036] In one implementation, based on the first edge feature and the second edge feature, determining the registration position coordinates includes:

[0037] Use the correlation coefficient matching method to match the first edge feature and the second edge feature;

[0038] Select the position of the maximum matching value of the first edge feature and the second edge feature to determine the registration position coordinates.

[0039] In one implementation, using the correlation coefficient matching method to match the first edge feature and the second edge feature; selecting the position of the maximum matching value of the first edge feature and the second edge feature to determine the registration position coordinates further includes:

[0040] Obtain the original value of the second edge feature and the mean value of the second edge feature of the second edge feature, and generate an infrared edge feature relationship parameter based on the fourth formula;

[0041] Obtain the original value of the first edge feature and the mean value of the first edge feature of the first edge feature, and generate a visible light edge feature relationship parameter based on the fifth formula;

[0042] Generate multiple correlation coefficient matching values based on the infrared edge feature relationship parameter and the visible light edge feature relationship parameter and based on the sixth formula;

[0043] Select the maximum matching value among the correlation coefficient matching values, and the position corresponding to the maximum matching value is the registration position coordinate.

[0044] An embodiment of the second aspect of the present application provides an image registration device based on edge features, including:

[0045] An acquisition module, configured to acquire a visible light image and an infrared image of a target object;

[0046] A first registration module, configured to perform primary registration on the visible light image and the infrared image to generate an initial visible light registration image;

[0047] An expansion module, configured to generate an initial visible light registration expansion image based on the initial visible light registration image;

[0048] A first extraction module, configured to extract first edge features from the initial visible light registration expansion image;

[0049] A second extraction module, configured to extract second edge features from the infrared image;

[0050] A second registration module, configured to determine the registration position coordinate based on the first edge feature and the second edge feature, and complete the final registration of the visible light image and the infrared image based on the registration position coordinate.

[0051] An embodiment of the third aspect of the present application provides a user terminal, including a storage and a processor. The storage stores a computer program, and when the processor executes the computer program, the steps of the image registration method according to the embodiment of the first aspect of the present application are implemented.

[0052] An embodiment of the fourth aspect of the present application provides a storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the image registration method according to the embodiment of the first aspect of the present application are implemented.

[0053] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. Description of the Drawings

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0055] Figure 1 It is a schematic diagram of the image registration method;

[0056] Figure 2 It is a flowchart of the image registration method;

[0057] Figure 3 is a flowchart for initially registering a visible light image and an infrared image to generate an initial visible light registered image;

[0058] Figure 4 is a flowchart for generating an initial visible light registered extended image based on the initial visible light registered image;

[0059] Figure 5 is a flowchart for extracting first edge features from the initial visible light registered extended image;

[0060] Figure 6 is a flowchart for extracting second edge features from the infrared image;

[0061] Figure 7 is the first flowchart for final registration;

[0062] Figure 8 is the second flowchart for final registration;

[0063] Figure 9 is the structural diagram of the image registration device;

[0064] Figure 10 is the structural diagram of the user terminal. Detailed implementation manners

[0065] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will describe the detailed implementation manners of the present application with reference to the accompanying drawings. Many specific details are set forth in the following description in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0066] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present application.

[0067] In the description of this application, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of this application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0068] In the embodiments of this application, unless otherwise clearly specified and limited, terms such as "installed", "connected", "connected to", "fixed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components or the interaction relationship between two components. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of this application can be understood according to specific circumstances.

[0069] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the aspects can be practiced without these specific details.

[0070] In the existing infrared image enhancement algorithm technology, when automatically registering visible light images and infrared images, a feature point matching method is adopted, that is, feature points on the visible light image and feature points on the infrared image are extracted respectively, and then they are matched one by one. The applicant found during use that due to the low resolution of the infrared image, it is difficult to extract accurate feature points, which in turn leads to poor registration effect and low accuracy between the visible light image and the infrared image, affecting the subsequent fusion of the visible light image and the infrared image. Therefore, the applicant invented an image registration method based on edge features to improve the registration accuracy and success rate of visible light images and infrared images.

[0071] As Figure 1 and Figure 2 shown, an embodiment of the first aspect of this application provides an image registration method based on edge features, which is applied to an infrared thermal imager. The infrared thermal imager includes an infrared thermal imaging camera and a visible light camera, and includes:

[0072] Step 100: Obtain the visible light image and the infrared image of the target object;

[0073] In step 100, the visible light image is obtained through the visible light camera, and the infrared image is obtained through the infrared thermal imaging camera. Among them, the array size of the visible light camera is 640*480, and the array size of the infrared thermal imaging camera is 384*288.

[0074] Step 200: Perform initial registration on the visible light image and the infrared image to generate an initial registered visible light image;

[0075] In Step 200, the visible light image and the infrared image are initially registered, mapping the visible light image onto the infrared image, that is, mapping the horizontal and vertical coordinates of the visible light image onto the horizontal and vertical coordinates of the infrared image, and generating an initial registered visible light image as a preliminary preparation for extracting the first edge feature on the initial registered visible light image later. Since the visible light camera and the infrared thermal imaging camera are not set at the same height of the infrared thermal imager, the visible light image obtained by the visible light camera and the infrared image obtained by the infrared thermal imaging camera, although taken of the same target at the same time, have certain differences due to angle deviation. The initial registration is to reduce the difference between the visible light image and the infrared image.

[0076] Step 300: Generate an initial registered and extended visible light image based on the initial registered visible light image;

[0077] In Step 300, since the area after initial registration is determined by the distance range during the offline calibration of the visible light camera and the infrared thermal imaging camera, if the shooting distances of the visible light camera and the infrared thermal imaging camera exceed the distance range during offline calibration, it will cause the visible light image and the infrared image to be unable to match accurately and deviate. And the deviation position will only be near the initial registration area. Therefore, after expanding the initial registered visible light image, search and match are performed to find the accurate registration position.

[0078] Step 400: Extract the first edge feature from the initial registered and extended visible light image;

[0079] In Step 400, in order to minimize the large matching error caused by the inconsistent brightness between the first edge feature and the second edge feature, before extracting the first edge feature, the initial registered and extended visible light image is grayscaled to eliminate errors to the greatest extent. In addition, after binarizing the extracted edge feature, median filtering is performed to remove noise and eliminate image noise interference, generating the first edge feature. Median filtering for noise removal is a common image processing step and will not be elaborated in detail here.

[0080] And the binarization of the image is to set the grayscale value of the pixel points on the image to 0 or 255, that is, to present the entire image with an obvious visual effect of only black and white.

[0081] Step 500: Extract the second edge feature from the infrared image;

[0082] In step 500, in order to minimize the luminance inconsistency between the second edge feature and the first edge feature, which may lead to a large matching error, before extracting the second edge feature, the infrared image is grayscaled to eliminate the error to the greatest extent. In addition, after the edge features extracted from the infrared image are binarized, median filtering is performed to remove noise and eliminate the interference of image noise, generating the second edge feature. Median filtering for noise reduction is a common image processing step and will not be elaborated in detail here.

[0083] Step 600: Based on the first edge feature and the second edge feature, determine the registration position coordinates, and based on these registration position coordinates, complete the final registration of the visible light image and the infrared image.

[0084] In step 600, based on the extracted first edge feature and second edge feature, quickly match the first edge feature and the second edge feature, find the maximum matching value between the two, and set the position of this maximum matching value as the final registration position.

[0085] This image registration method can extract relatively accurate features in the case of low resolution of the infrared image, so as to realize the accurate registration of the infrared image and the visible light image, and effectively ensure the subsequent smooth fusion of the infrared image and the visible light image.

[0086] Figure 3 As shown, perform initial registration on the visible light image and the infrared image to generate the initial visible light registration image, including:

[0087] Step 210: Obtain the visible light rotation matrix and the visible light translation matrix of the visible light camera;

[0088] In step 210, both the visible light rotation matrix and the visible light translation matrix are calculated by calibrating the visible light camera and the infrared camera.

[0089] Step 220: Obtain the infrared rotation matrix and the infrared translation matrix of the infrared thermal imaging camera;

[0090] In step 220, both the infrared rotation matrix and the infrared translation matrix are calculated by calibrating the infrared thermal imaging camera and the visible light camera.

[0091] Step 230: According to the visible light rotation matrix and the infrared rotation matrix, and based on the first formula, generate the rotation mapping matrix;

[0092] In step 230, the first formula is: R = vi_R × ir_R -1

[0093] where R is the rotation mapping matrix, vi_R is the visible light rotation matrix, and ir_R is the infrared rotation matrix.

[0094] Step 240: Generate a translation mapping matrix according to the visible light translation matrix, the infrared translation matrix, and the rotation mapping matrix, and based on the second formula.

[0095] In step 240, the second formula is: T = vi_T - R × ir_T

[0096] Where, T is the translation mapping matrix, R is the rotation mapping matrix, vi_T is the visible light translation matrix, and ir_R is the infrared translation matrix.

[0097] Step 250: Obtain the visible light internal parameter matrix of the visible light camera, the infrared internal parameter matrix of the infrared thermal imaging camera, and the horizontal and vertical coordinates of the infrared image.

[0098] In step 250, both the visible light internal parameter matrix and the infrared internal parameter matrix are calculated by calibrating the infrared thermal imaging camera and the visible light camera.

[0099] Step 260: Generate the horizontal and vertical coordinates of the visible light image according to the rotation mapping matrix, the translation mapping matrix, the visible light internal parameter matrix, the infrared internal parameter matrix, and the horizontal and vertical coordinates of the infrared image, and based on the third formula.

[0100] In step 260, the third formula is:

[0101]

[0102] Where, vi_u is the horizontal coordinate of the visible light image, vi_v is the vertical coordinate of the visible light image, vi_K is the visible light internal parameter matrix, R is the rotation mapping matrix, T is the translation mapping matrix, ir_u is the horizontal coordinate of the infrared image, ir_v is the vertical coordinate of the infrared image, and ir_K is the infrared internal parameter matrix.

[0103] Step 270: Generate an initial registered visible light image according to the horizontal and vertical coordinates of the visible light image.

[0104] In step 270, the specific generation process of the initial registered visible light image is as follows from the first relation to the fourth relation:

[0105] First relation:

[0106] vi_mappingX = min(vi_u)

[0107] Where, vi_mappingX is the horizontal coordinate of the upper left corner of the initial registered visible light image, vi_u is the horizontal coordinate of the visible light image, and min(vi_u) is the minimum value of the horizontal coordinates of the visible light image.

[0108] Second relation:

[0109] vi_mappingY = min(vi_v)

[0110] Among them, vi_mappingY is the vertical coordinate of the upper left corner of the initial visible light registration image, vi_v is the vertical coordinate of the visible light image, and min(vi_v) is the minimum value of the vertical coordinate of the visible light image.

[0111] The third relational expression:

[0112] vi_mappingW = max(vi_u) - vi_mappingX

[0113] Among them, vi_mappingW is the width of the initial visible light registration image, max(vi_u) is the maximum value of the horizontal coordinate of the visible light image, and vi_mappingX is the horizontal coordinate of the upper left corner of the initial visible light registration image.

[0114] The fourth relational expression:

[0115] vi_mappingH = max(vi_v) - vi_mappingY

[0116] Among them, vi_mappingH is the height of the initial visible light registration image, max(vi_v) is the maximum value of the vertical coordinate of the visible light image, and vi_mappingY is the vertical coordinate of the upper left corner of the initial visible light registration image.

[0117] The calculated vi_mappingX as the horizontal coordinate of the upper left corner of the initial visible light registration image, vi_mappingY as the vertical coordinate of the upper left corner of the initial visible light registration image, vi_mappingW as the width of the initial visible light registration image, and vi_mappingH as the height of the initial visible light registration image together form the initial visible light registration image.

[0118] Figure 4 As shown, after generating the initial visible light registration image according to the horizontal and vertical coordinates of the visible light image, it includes:

[0119] Step 310: Obtain the horizontal and vertical coordinates, width value, and height value of the upper left corner of the initial visible light registration image;

[0120] Step 320: Generate an initial visible light registration extended image according to the horizontal and vertical coordinates, width value, height value of the upper left corner of the initial visible light registration image, and the expansion value.

[0121] In step 320, the expansion value is a fixed value and is preset in the infrared thermal imager. According to the horizontal and vertical coordinates, width value, height value of the initial visible light registration image, and the expansion value, and using the fifth relational expression, an initial visible light registration extended image is generated.

[0122] The fifth relational expression is:

[0123] vi_expand = (vi_mappingX - d, vi_mappingY - d, vi_mappingW + 2 * d, vi_mappingH + 2 * d), where vi_expand is the visible light initial registration expanded image, vi_mappingX is the abscissa of the upper left corner of the visible light initial registration image, vi_mappingY is the ordinate of the upper left corner of the visible light initial registration image, vi_mappingH is the height of the visible light initial registration image, vi_mappingW is the width of the visible light initial registration image, and d is the expansion value.

[0124] Figure 5 As shown, extracting the first edge feature from the visible light initial registration expanded image further includes:

[0125] Step 410: Grayscale the visible light initial registration expanded image to generate a grayscale image of the visible light initial registration expanded image;

[0126] Step 420: Calculate the visible light X-direction gradient value and the visible light Y-direction gradient value according to the grayscale image of the visible light initial registration expanded image;

[0127] In step 420, according to the grayscale image of the visible light initial registration expanded image, the sixth relational expression, and the seventh relational expression, calculate the visible light X-direction gradient value and the visible light Y-direction gradient value. The sixth relational expression is:

[0128]

[0129] where vi_exG x is the visible light X-direction gradient value, and vi_exGray is the grayscale image of the visible light initial registration expanded image.

[0130] The seventh relational expression:

[0131]

[0132] where vi_exG y is the visible light Y-direction gradient value, and vi_exGray is the grayscale image of the visible light initial registration expanded image.

[0133] Step 430: Generate the visible light edge feature according to the visible light X-direction gradient value and the visible light Y-direction gradient value.

[0134] In step 430, according to the visible light X-direction gradient value and the visible light Y-direction gradient value, and using the eighth relational expression, generate the visible light edge feature. The eighth relational expression is:

[0135] vi_exG = |vi_exG x | + |vi_exG y |

[0136] where vi_exG x is the visible light X - direction gradient value, and vi_exG y is the visible light Y - direction gradient value, and vi_exG is the visible light edge feature.

[0137] Step 440: According to the visible light edge feature, perform binarization and filtering processing to generate the first edge feature.

[0138] Figure 6 As shown, extracting the second edge feature from the infrared image further includes:

[0139] Step 510: Grayscale the infrared image to generate a grayscale infrared image;

[0140] Step 520: Calculate the infrared X - direction gradient value and the infrared Y - direction gradient value according to the grayscale infrared image;

[0141] In step 520, according to the grayscale infrared image, the ninth relation and the tenth relation, calculate the infrared X - direction gradient value and the infrared Y - direction gradient value. The ninth relation is:

[0142]

[0143] where ir_exG x is the infrared X - direction gradient value, and ir_exGray is the grayscale image of the infrared registration extended image.

[0144] The tenth relation is:

[0145]

[0146] where ir_exG y is the infrared Y - direction gradient value, and ir_exGray is the grayscale image of the infrared registration extended image.

[0147] Step 530: Generate the infrared edge feature according to the infrared X - direction gradient value and the infrared Y - direction gradient value.

[0148] In step 530, according to the infrared X - direction gradient value and the infrared Y - direction gradient value, and using the eleventh relation, generate the second edge feature. The eleventh relation is:

[0149] ir_exG = |ir_exG x | + |ir_exG y |

[0150] where ir_exGx is the visible light X-direction gradient value, ir_exG y is the visible light Y-direction gradient value, and ir_exG is the infrared edge feature.

[0151] Step 540: Generate a second edge feature based on the infrared edge feature and perform binarization and filtering processing.

[0152] Figure 7 As shown, based on the first edge feature and the second edge feature, determine the registration position coordinates, including:

[0153] Step 610: Use the correlation coefficient matching method to match the first edge feature and the second edge feature;

[0154] Step 620: Select the position of the maximum matching value of the first edge feature and the second edge feature to determine the registration position coordinates.

[0155] Figure 8 As shown, use the correlation coefficient matching method to match the first edge feature and the second edge feature; select the position of the maximum matching value of the first edge feature and the second edge feature to determine the registration position coordinates, and also include:

[0156] Step 611: Obtain the second edge feature original value and the second edge feature mean value of the second edge feature, and generate an infrared edge feature relationship parameter based on the fourth formula;

[0157] In step 611, the fourth formula is:

[0158]

[0159] where ir_G is the second edge feature original value, is the second edge feature mean value, ir_G' is the infrared edge feature relationship parameter, and (x', y') are the horizontal and vertical coordinates of the second edge feature.

[0160] Step 612: Obtain the first edge feature original value and the first edge feature mean value of the first edge feature, and generate a visible light edge feature relationship parameter using the fifth formula;

[0161] In step 612, the fifth formula is:

[0162]

[0163] where vi_exG is the first edge feature original value, is the first edge feature mean value, vi_exG' is the visible light edge feature relationship parameter, and (x + x', y + y') are the horizontal and vertical coordinates of the first edge feature.

[0164] Step 613: Generate multiple correlation coefficient matching values according to the infrared edge feature relationship parameters and the visible light edge feature relationship parameters, and based on the sixth formula;

[0165] In step 613, the sixth formula is:

[0166] R(x,y) = ∑(ir_G'(x',y') * vi_exG'(x + x',y + y'))

[0167] wherein, ir_G'(x',y') is the infrared edge feature relationship parameter, vi_exG'(x + x',y + y') is the visible light edge feature relationship parameter, and R(x,y) is the correlation coefficient matching value.

[0168] Step 614: Select the maximum matching value among the correlation coefficient matching values, and the position corresponding to the maximum matching value is the registration position coordinate.

[0169] As Figure 9 shown, an embodiment of the second aspect of the present application provides an image registration device based on edge features, including:

[0170] An acquisition module S100, configured to acquire a visible light image and an infrared image of a target object;

[0171] A first registration module S200, configured to perform primary registration on the visible light image and the infrared image to generate a visible light initial registration image;

[0172] An expansion module S300, configured to generate a visible light initial registration expansion image based on the visible light initial registration image;

[0173] A first extraction module S400, configured to extract first edge features from the visible light initial registration expansion image;

[0174] A second extraction module S500, configured to extract second edge features from the infrared image;

[0175] A second registration module S600, configured to determine the registration position coordinate based on the first edge feature and the second edge feature, and complete the final registration of the visible light image and the infrared image based on the registration position coordinate.

[0176] As Figure 10 shown, an embodiment of the third aspect of the present application provides a user terminal, including a storage and a processor, the storage stores a computer program, and when the processor executes the computer program, the steps of the image registration method according to the embodiment of the first aspect of the present application are implemented.

[0177] The user terminal according to the embodiment of the third aspect of the present application can be implemented with reference to the content specifically described in the embodiment of the first aspect of the present application, and has similar beneficial effects to the image registration method according to the embodiment of the first aspect of the present application, which will not be elaborated herein.

[0178] The user terminal 10 can be implemented in the form of a general-purpose computing device. The components of the user terminal 10 may include, but are not limited to: one or more processors or processing units 11, a system memory 12, and a bus 13 connecting different system components (including the system memory 12 and the processing unit 11).

[0179] The bus 13 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnection (PCI) bus.

[0180] The user terminal 10 typically includes a variety of computer system-readable media. These media can be any available media accessible by the user terminal 10, including volatile and non-volatile media, removable and non-removable media.

[0181] The memory 12 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 14 and / or cache memory 15. The user terminal 10 may further include other removable / non-removable, volatile / non-volatile computer-readable storage media. By way of example only, the storage system 16 can be used to read and write non-removable, non-volatile magnetic media (not shown in the figure, commonly referred to as a "hard disk drive"). Although Figure 5Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a Compact Disc ReadOnlyMemory (hereinafter referred to as: CD-ROM), a Digital Video Disc ReadOnlyMemory (hereinafter referred to as: DVD-ROM), or other optical media) can be provided. In these cases, each drive can be connected to the bus 13 through one or more data medium interfaces. The memory can include at least one program product, which has a set (such as at least one) of program modules, and these program modules are configured to execute the functions of the various embodiments of the present disclosure.

[0182] A program / utility 18 having a set (at least one) of program modules 17 can be stored, for example, in the memory. Such program modules 17 include, but are not limited to, an operating system, one or more application programs, other program modules 17, and program data. The implementation of a network environment may be included in each or some combination of these examples. The program modules 17 generally execute the functions and / or methods in the embodiments described in the present disclosure.

[0183] The user terminal 10 can also communicate with one or more external devices 19 (such as a keyboard, a pointing device, a display 20, etc.), and can also communicate with one or more devices that enable the user to interact with the computer system / server, and / or communicate with any device that enables the computer system / server to communicate with one or more other user terminals 10 (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 21. And, the user terminal 10 can also communicate with one or more networks (such as a Local Area Network (hereinafter referred to as: LAN), a Wide Area Network (hereinafter referred to as: WAN), and / or a public network, such as the Internet) through the network adapter 22. As shown in the figure, the network adapter 22 communicates with other modules of the user terminal 10 through the bus 13. It should be noted that, although not shown in the figure, other hardware and / or software modules can be used in combination with the user terminal 10, including but not limited to: microcode, device drivers, redundant processing units 11, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 16, etc.

[0184] The processing unit 11 executes various functional applications and data processing by running the programs stored in the system memory 12, such as implementing the methods mentioned in the foregoing embodiments.

[0185] The user terminal 10 in the embodiments of the present application may be a server or a terminal device with limited computing power.

[0186] An embodiment of the fourth aspect of the present application provides a storage medium storing a computer program, which when executed by a processor implements the steps of the image registration method in the embodiment of the first aspect of the present application.

[0187] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. A non-transitory computer-readable storage medium may include any computer-readable medium except for the signals propagating temporarily themselves.

[0188] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM14), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0189] The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. In particular, the Python language suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0190] The above are only the preferred embodiments of the present application and do not impose any formal restrictions on the present application. Any person skilled in the art can, without departing from the scope of the technical solution of the present application, make many possible changes and modifications to the technical solution of the present application by using the methods and technical contents disclosed above, or modify it into equivalent embodiments with equivalent changes. Therefore, all equivalent changes made according to the shape, structure and principle of the present application without departing from the content of the technical solution of the present application shall be covered by the protection scope of the present application.

Claims

1. An image registration method based on edge features, which is applied to an infrared thermal imager. The infrared thermal imager includes an infrared thermal imaging camera and a visible light camera, and is characterized in that Including: Obtain the visible light image and the infrared image of the target object; Perform primary registration on the visible light image and the infrared image to generate an initial visible light registration image; Based on the initial visible light registration image, generate an initial visible light registration extended image, where the horizontal and vertical coordinates, width value, and height value of the upper left corner of the initial visible light registration image are obtained; according to the horizontal and vertical coordinates, width value, height value, and expansion value of the upper left corner of the initial visible light registration image, generate an initial visible light registration extended image; Extract the first edge feature from the initial visible light registration extended image; Extract the second edge feature from the infrared image; Based on the first edge feature and the second edge feature, determine the registration position coordinates, and based on the registration position coordinates, complete the final registration of the visible light image and the infrared image.

2. The image registration method based on edge features according to claim 1, wherein, The performing primary registration on the visible light image and the infrared image to generate an initial visible light registration image includes: Obtain the visible light rotation matrix and the visible light translation matrix of the visible light camera; Obtain the infrared rotation matrix and the infrared translation matrix of the infrared thermal imaging camera; According to the visible light rotation matrix and the infrared rotation matrix, and based on the first formula, generate a rotation mapping matrix; According to the visible light translation matrix, the infrared translation matrix, and the rotation mapping matrix, and based on the second formula, generate a translation mapping matrix; Obtain the visible light internal parameter matrix of the visible light camera, the infrared internal parameter matrix of the infrared thermal imaging camera, and the horizontal and vertical coordinates of the infrared image; According to the rotation mapping matrix, the translation mapping matrix, the visible light internal parameter matrix, the infrared internal parameter matrix, and the horizontal and vertical coordinates of the infrared image, and based on the third formula, generate the horizontal and vertical coordinates of the visible light image; According to the horizontal and vertical coordinates of the visible light image, generate an initial visible light registration image.

3. The image registration method based on edge features according to claim 1, wherein The extracting the first edge feature from the initial visible light registration extended image further includes: Grayscale the initial visible light registration extended image to generate a grayscale image of the initial visible light registration extended image; Calculate the visible light X-direction gradient value and the visible light Y-direction gradient value according to the grayscale image of the initial visible light registration extended image; Generate a visible light edge feature according to the visible light X-direction gradient value and the visible light Y-direction gradient value; According to the visible light edge feature, and perform binarization and filtering processing to generate the first edge feature.

4. The image registration method based on edge features according to claim 1, wherein The extracting the second edge feature from the infrared image further includes: Grayscale the infrared image to generate a grayscale image of the infrared image; Calculate the infrared X-direction gradient value and the infrared Y-direction gradient value according to the grayscale image of the infrared image; Generate an infrared edge feature according to the infrared X-direction gradient value and the infrared Y-direction gradient value; According to the infrared edge feature, and perform binarization and filtering processing to generate the second edge feature.

5. The image registration method based on edge features according to claim 1, wherein The determining the registration position coordinates based on the first edge feature and the second edge feature includes: Use the correlation coefficient matching method to match the first edge feature and the second edge feature; Select the position of the maximum matching value of the first edge feature and the second edge feature to determine the registration position coordinates.

6. The image registration method based on edge features according to claim 5, wherein The using the correlation coefficient matching method to match the first edge feature and the second edge feature; Selecting the position of the maximum matching value of the first edge feature and the second edge feature to determine the registration position coordinates further includes: Obtaining the original value and the mean value of the second edge feature of the second edge feature, and generating an infrared edge feature relationship parameter based on the fourth formula; Obtaining the original value and the mean value of the first edge feature of the first edge feature, and generating a visible light edge feature relationship parameter based on the fifth formula; Generating a plurality of correlation coefficient matching values according to the infrared edge feature relationship parameter and the visible light edge feature relationship parameter, and based on the sixth formula; Selecting the maximum matching value among the correlation coefficient matching values, and the position corresponding to the maximum matching value is the registration position coordinates.

7. An image registration device based on edge features, characterized in that Including: An acquisition module, configured to acquire a visible light image and an infrared image of a target object; A first registration module, configured to perform primary registration on the visible light image and the infrared image to generate an initial visible light registration image; An expansion module, configured to generate an initial visible light registration expanded image based on the initial visible light registration image, where the horizontal and vertical coordinates, width value, and height value of the upper left corner of the initial visible light registration image are obtained; an initial visible light registration expanded image is generated according to the horizontal and vertical coordinates, width value, height value of the upper left corner of the initial visible light registration image, and the expansion value; A first extraction module, configured to extract a first edge feature from the initial visible light registration expanded image; A second extraction module, configured to extract a second edge feature from the infrared image; A second registration module, configured to determine the registration position coordinates based on the first edge feature and the second edge feature, and complete the final registration of the visible light image and the infrared image based on the registration position coordinates.

8. A user terminal, comprising a storage and a processor, wherein the storage stores a computer program, characterized in that, When the processor executes the computer program, the steps of the image registration method according to any one of claims 1 to 6 are implemented.

9. A storage medium stores a computer program, characterized in that, When the computer program is executed by the processor, the steps of the image registration method according to any one of claims 1 to 6 are implemented.

Images