Infrared image registration method and device, medium and parallel dual-light handheld infrared device

Abstract

The application discloses an infrared image registration method and device, a medium and a parallel double-light handheld infrared equipment, and relates to the field of infrared temperature monitoring. The first infrared image and the first visible light image are registered according to imaging rules, then the second visible light image and the first visible light image are registered according to marks on a target object, and finally the second visible light image and the second infrared image are registered again according to imaging rules, so that the registration of the infrared image is realized. In the method, key points, feature vectors of the key points and the like do not need to be extracted from the infrared image, and the method is not affected by the resolution of the infrared image, so that the stability and accuracy are higher, and the success rate of infrared image matching is improved. Furthermore, the method is not limited by the color texture of the pseudo-color image, so that the registration effect is better. In addition, key point feature extraction and key point feature matching are omitted, so that the calculation amount is greatly reduced, and the registration efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of infrared temperature monitoring, and particularly relates to an infrared image registration method and device, a medium and a parallel dual-light handheld infrared device. BACKGROUND

[0002] When using a handheld infrared imager to monitor the temperature of equipment, two pictures are generated each time, which are an infrared image and a visible light image. However, due to differences in the angle and distance of each time of shooting, the current infrared image and the previously shot infrared image cannot be accurately aligned, that is, the coordinates of the same object on the infrared image 1 and the infrared image 2 are different. The goal of image registration is to find a mapping relationship to map the coordinates on the infrared image 1 to the infrared image 2, so that many related business functions can be realized, for example, the coordinates of a target object are obtained on the infrared image 1, and then the coordinates are mapped to the historical infrared image, so that the historical temperature change curve of the target object can be obtained.

[0003] To solve this problem, there are many implementation schemes, and a common method is to realize it based on the detection and matching of key feature points. Since the resolution of the infrared image is generally low, there may be many invalid key points extracted from the infrared image, thereby causing the registration algorithm to fail.

[0004] Therefore, how to improve the success rate of infrared image registration is a technical problem that needs to be solved by the person skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide an infrared image registration method, device, medium and parallel dual-light handheld infrared device, which can improve the success rate of infrared image registration.

[0006] To solve the above technical problem, the present application provides an infrared image registration method, which comprises the following steps:

[0007] obtaining a first group of images and a second group of images of a target object with a mark collected by a parallel dual-light handheld infrared device; wherein the first group of images comprises a first infrared image and a first visible light image, and the second group of images comprises a second infrared image and a second visible light image;

[0008] registering the first infrared image and the first visible light image according to the imaging rule of the parallel dual-light handheld infrared device;

[0009] registering the second visible light image and the first visible light image according to the mark in the first visible light image and the mark in the second visible light image;

[0010] The second visible light image and the first visible light image are registered according to the imaging rule.

[0011] Preferably, the target object has a preset number of markers, and the registration of the second visible light image and the first visible light image according to the markers in the first visible light image and the markers in the second visible light image comprises:

[0012] The first binary image corresponding to the first visible light image is generated by acquiring three channels of RGB in the first visible light image, and the second binary image corresponding to the second visible light image is generated by acquiring three channels of RGB in the second visible light image.

[0013] First target marker points are screened from the first binary image and second target marker points are screened from the second binary image, respectively.

[0014] A transformation matrix is determined according to the coordinates of each first target marker point and the coordinates of each second target marker point, wherein the number of the first target marker points and the number of the second target marker points are determined according to the number of the markers on the target object.

[0015] The second visible light image and the first visible light image are registered according to the transformation matrix.

[0016] Preferably, the screening of the first target marker points from the first binary image and the screening of the second target marker points from the second binary image respectively comprises:

[0017] An opening operation is performed on the first binary image and the second binary image using a preset structural element, respectively.

[0018] Each first connected domain in the first binary image and each second connected domain in the second binary image are extracted, respectively.

[0019] Each first edge point corresponding to each first connected domain and each second edge point corresponding to each second connected domain are acquired.

[0020] The distance from each first edge point of each first connected domain to the center of a target circumscribed rectangle corresponding to the first connected domain is acquired, and the distance from each second edge point of each second connected domain to the center of a target circumscribed rectangle corresponding to the second connected domain is acquired.

[0021] a variance of a sequence of distances from all the first edge points of each of the first connected domains to a center of a target circumscribed rectangle of the corresponding first connected domain, and a variance of a sequence of distances from all the second edge points of each of the second connected domains to a center of a target circumscribed rectangle of the corresponding second connected domain;

[0022] selecting a preset number of first connected domains with the smallest variance from all the first connected domains and a preset number of second connected domains with the smallest variance from all the second connected domains;

[0023] taking the center points of the preset number of first connected domains with the smallest variance as the first target marker points and taking the center points of the preset number of second connected domains with the smallest variance as the second target marker points.

[0024] Preferably, after the first and second connected domains are extracted respectively from the first and second binary images, before the first and second edge points are obtained, the method further comprises:

[0025] grouping all the first connected domains into a first connected domain set and grouping all the second connected domains into a second connected domain set;

[0026] deleting first connected domains not meeting a preset requirement from the first connected domain set and deleting second connected domains not meeting a preset requirement from the second connected domain set;

[0027] performing the step of obtaining the first and second edge points on the first and second connected domains meeting the preset requirement in the first and second connected domain sets.

[0028] Preferably, the preset condition comprises at least one of the following:

[0029] Condition one: the number of pixels with a value of 1 is greater than or equal to a first threshold value;

[0030] Condition two: a ratio of a minimum value of the width and height of the target circumscribed rectangle to a maximum value of the width and height is greater than or equal to a second threshold value;

[0031] Condition three: a ratio of the number of pixels with a value of 1 to the area of the corresponding target circumscribed rectangle is within a preset range.

[0032] Preferably, in the case where the target object is square, the markers are four circular target marker points located at the four corners of the target object.

[0033] Correspondingly, the number of the first target mark points and the number of the second target mark points are both 4;

[0034] In the case of the mark being a red mark, the pixel value of the R channel in the first visible light image and the pixel value of the R channel in the second visible light image are both greater than the sum of the pixel value of the G channel and the pixel value of the B channel;

[0035] In the case of the mark being a green mark, the pixel value of the G channel in the first visible light image and the pixel value of the G channel in the second visible light image are both greater than the sum of the pixel value of the R channel and the pixel value of the B channel;

[0036] In the case of the mark being a blue mark, the pixel value of the B channel in the first visible light image and the pixel value of the B channel in the second visible light image are both greater than the sum of the pixel value of the R channel and the pixel value of the G channel.

[0037] Preferably, before the step of determining the transformation matrix according to the coordinates of each of the first target mark points and the coordinates of each of the second target mark points, the method further comprises:

[0038] obtaining all first rectangles circumscribed by each of the first target mark points, and obtaining all second rectangles circumscribed by each of the second target mark points;

[0039] selecting a first rectangle with the smallest area from all the first rectangles, and selecting a second rectangle with the smallest area from all the second rectangles;

[0040] obtaining the distance from each of the first target mark points to the vertex of the first rectangle with the smallest area, and obtaining the distance from each of the second target mark points to the vertex of the second rectangle with the smallest area;

[0041] assigning each of the first target mark points to the vertex of the first rectangle with the smallest area closest in distance, and assigning each of the second target mark points to the vertex of the second rectangle with the smallest area closest in distance;

[0042] in the case that each of the vertices of the first rectangle with the smallest area is assigned with the first target mark point, and each of the vertices of the second rectangle with the smallest area is assigned with the second target mark point, entering the step of determining the transformation matrix according to the coordinates of each of the first target mark points and the coordinates of each of the second target mark points.

[0043] To solve the above technical problems, the application also provides an infrared image registration device, comprising:

[0044] An acquisition module is configured to acquire a first set of images and a second set of images of a target object with a mark collected by a parallel dual-optical handheld infrared device, wherein the first set of images comprises a first infrared image and a first visible light image, and the second set of images comprises a second infrared image and a second visible light image.

[0045] A first registration module is configured to register the first infrared image and the first visible light image according to an imaging rule of the parallel dual-optical handheld infrared device.

[0046] A second registration module is configured to register the second visible light image and the first visible light image according to the mark in the first visible light image and the mark in the second visible light image.

[0047] A third registration module is configured to register the second visible light image and the second infrared image according to the imaging rule, so as to realize registration of the first infrared image and the second infrared image.

[0048] To solve the above technical problems, the present application further provides a parallel dual-optical handheld infrared device, comprising:

[0049] A memory is configured to store a computer program.

[0050] A processor is configured to execute the computer program to realize the steps of the above infrared image registration method.

[0051] To solve the above technical problems, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the above infrared image registration method.

[0052] In the method for infrared image registration provided in the application, the first infrared image and the first visible light image are registered according to the imaging rule first, then the second visible light image and the first visible light image are registered according to the mark on the target object, and finally the second visible light image and the second infrared image are registered according to the imaging rule again, so that the registration from the first infrared image to the second infrared image is realized. It can be seen that in the method, the key points and the feature vectors of the key points are not extracted from the infrared image in the process of registering the infrared image, therefore, compared with the previous method of directly extracting the key points from two infrared images and performing feature matching, the method of the application has higher stability and accuracy because it is not affected by the resolution of the infrared image, and the success rate of the infrared image registration is improved; secondly, the infrared image is a pseudo-color image rendered according to the temperature value, if the temperature difference of the object to be photographed is obvious, the color and texture of the infrared image may be obviously different, and if the previous registration method is used, the feature description vector difference between the key points may be large, thereby affecting the matching effect, while in the method provided in the application, the color and texture of the pseudo-color image are not limited, and therefore the registration effect is better; in addition, if the previous registration method is used, the best matching key points are extracted from a large number of key points, for example, four matching key points are selected through perspective transformation, and a large number of loops are required to determine whether the key point combination meets the requirements, therefore, the calculation efficiency is low, while in the method provided in the application, the key point feature extraction and key point feature matching are omitted, thereby greatly reducing the calculation amount and improving the registration efficiency.

[0053] In addition, the application also provides an infrared image registration device, a parallel dual-optical handheld infrared device and a computer readable storage medium, which have the same or corresponding technical features as the above-mentioned infrared image registration method, and the effects are the same. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the application, the drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 A flowchart of a method for infrared image registration provided in an embodiment of the application;

[0056] Figure 2 A schematic diagram of the overall process of the infrared image registration provided in an embodiment of the application;

[0057] Figure 3 An imaging schematic diagram of parallel optical axes provided in an embodiment of the application;

[0058] Figure 4 The structural diagram of the device for infrared image registration provided by an embodiment of the present application is shown in the figure;

[0059] Figure 5 The structural diagram of the parallel dual-optical handheld infrared device provided by another embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0061] The core of the present application is to provide an infrared image registration method, device, medium and parallel dual-optical handheld infrared device, for improving the success rate of infrared image registration.

[0062] A handheld infrared imager is used for equipment temperature monitoring, and two pictures are generated each time, which are infrared images and visible light images respectively. However, due to the difference in angle and distance of each shooting, the current infrared image and the previously shot infrared image cannot be accurately aligned, that is, the coordinates of the same object on the infrared image 1 and the infrared image 2 are different. The goal of image registration is to find a mapping relationship to map the coordinates on the infrared image 1 to the infrared image 2, so that many related business functions can be realized, for example, the coordinates of the target object are obtained on the infrared image 1, and then the coordinates are mapped to the historical infrared image, so that the historical temperature change curve of the target object can be obtained.

[0063] In view of this problem, there are many implementation schemes. A common method is to realize it based on the detection and matching of key feature points. The general process includes extracting some key points and feature description vectors of the key points on the infrared image through Scale-Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF) and other algorithms, then matching the key point feature description vectors on the two infrared images through a matching algorithm, and calculating the transformation matrix of coordinate transformation after the matching is completed. The transformation matrix can be used to realize the pixel-level coordinate transformation of the two infrared images.

[0064] The above method mainly has the following problems:

[0065] 1. Since the resolution of the infrared image is generally low, there may be many invalid key points extracted from the infrared image, which may lead to algorithm failure;

[0066] 2. Since infrared images are pseudo-color images rendered based on temperature values, if the temperature of the object being photographed varies significantly, the color and texture of the infrared image may differ significantly. Therefore, the feature description vectors between key points may differ considerably, affecting the overall effect.

[0067] 3. Extracting the best matching key points from a large number of key points. For example, if four matching key points need to be selected through perspective transformation, a large number of loops are needed to determine whether the combination of key points meets the requirements, which is computationally inefficient.

[0068] Therefore, this application proposes an image registration method applicable to infrared handheld devices with parallel optical axes. It can quickly perform image registration, is not limited by the resolution of infrared images and the color texture of pseudo-color images, and does not require a large number of key point combination judgments. It has a high success rate and efficiency.

[0069] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 A flowchart of an infrared image registration method provided in this application embodiment is shown below. Figure 1 As shown, the method includes:

[0070] S10: Acquire a first set of images and a second set of images of a marked target object collected by a parallel dual-light handheld infrared device; wherein, the first set of images contains a first infrared image and a first visible light image, and the second set of images contains a second infrared image and a second visible light image;

[0071] S11: Register the first infrared image and the first visible light image according to the imaging rules of the parallel dual-light handheld infrared device;

[0072] S12: Register the second visible light image and the first visible light image according to the markers in the first visible light image and the markers in the second visible light image;

[0073] S13: Register the second visible light image and the second infrared image according to the imaging rules to achieve the registration of the first infrared image and the second infrared image.

[0074] The target object is the object that needs to be monitored for temperature. There are no restrictions on the target object; it will be determined based on the actual situation.

[0075] The embodiments of the present application paste a mark on the target object, and the shape, position, color and the like of the mark are not limited. The target object with the mark is photographed by the parallel light handheld infrared device to obtain a visible light image and an infrared image of the target object. In order to obtain a target temperature change curve, the image of the target object needs to be collected by the parallel light handheld infrared device for multiple times. The frequency of photographing the target object is not limited, and preferably, the visible light image and the corresponding infrared image of the target object at different times are photographed at a preset time interval.

[0076] Due to the difference in the angle and distance of each photographing, the current infrared image and the previously photographed infrared image cannot be accurately aligned, and therefore, the infrared image registration is needed, that is, the mapping relationship between different infrared images is found. In the embodiments of the present application, the process of infrared image registration is described by taking two groups of images photographed by the parallel light handheld infrared device as an example. First, a first group of images and a second group of images of the target object with the mark collected by the parallel light handheld infrared device are obtained, and the parallel light handheld infrared device generates two images at each photographing, which are an infrared image and a visible light image. The first group of images includes a first infrared image and a first visible light image, and the second group of images includes a second infrared image and a second visible light image.

[0077] The purpose of the infrared image registration of the embodiments of the present application is to map the coordinates on the first infrared image to the second infrared image. Figure 2 A schematic diagram of the overall process of the infrared image registration provided by the embodiments of the present application is shown in Figure 2 First, the coordinates on the first infrared image are mapped to the first visible light image through infrared and visible light registration; then, the coordinates on the first visible light image are mapped to the second visible light image through visible light registration; and finally, the coordinates on the second visible light image are mapped to the second infrared image through infrared and visible light registration.

[0078] The method for infrared and visible light registration is not limited, as long as the coordinates on the infrared image can be mapped to the visible light image. In the embodiments, the infrared and visible light registration (including the registration of the first infrared image and the first visible light image, and the registration of the second infrared image and the second visible light image) is realized based on the imaging rule of the parallel light handheld infrared device.

[0079] Specifically, for the infrared image and the visible light image photographed by the parallel light infrared handheld device, the distance between the photographed object and the handheld device is measured by the laser ranging sensor on the handheld device, and the overlapping part between the infrared image and the visible light image can be calculated by combining the field of view angle of the infrared camera and the visible light camera. The overlapping part is cropped and scaled to realize the pixel-level registration between the infrared image and the visible light image. Figure 3The imaging schematic diagram of parallel light axis provided by the embodiment of the present application. The imaging principle in horizontal and vertical directions is the same. As shown in Figure 3 In the formula (1), the distance between the visible light lens and the infrared lens is b, the distance between the object and the imaging plane is u, the field of view angle is θ, and the pixel number of the image in the horizontal or vertical direction is L. The formula (1) is an expression of the pixel number P that needs to be cropped, and the pixel number P that needs to be cropped can be calculated through the formula (1).

[0080]

[0081] It can be seen from the formula (1) that the pixel number that needs to be reserved in the visible light image in the direction is [0, L-1], and the pixel number that needs to be reserved in the infrared image in the direction is [0, L-P-1]. Figure 3 After the pixel cropping in the horizontal and vertical directions according to the above method, the cropped infrared image and the visible light image are scaled to the same size as the original size of the visible light image, so as to facilitate the subsequent coordinate transformation. Through the above process, the pixel-level registration of the infrared image and the visible light image is realized, and the coordinates on the infrared image are given. The coordinates on the corresponding visible light image can be obtained, or the coordinates on the visible light image are given, and the coordinates on the corresponding infrared image can be obtained.

[0082] After the registration of the first infrared image and the first visible light image, the registration of the first visible light image and the second infrared image is performed. In the embodiment, the second visible light image and the first visible light image are registered according to the mark in the first visible light image and the mark in the second visible light image. Specifically, the coordinates of the mark points on the first visible light image and the coordinates of the mark points on the second visible light image are obtained, the transformation matrix is solved according to the coordinates of the mark points on the first visible light image and the coordinates of the mark points on the second visible light image, and then the coordinates on the first visible light image are mapped to the second visible light image.

[0083] After the registration of the first visible light image to the second visible light image, the registration of the second visible light image and the second infrared image is performed. When the registration of the second visible light image and the second infrared image is performed, the same method as the registration of the first infrared image and the first visible light image is adopted, and the process of the registration of the first infrared image and the first visible light image has been described in detail in the foregoing, and the process of the registration of the second visible light image and the second infrared image will not be described herein.

[0084] It can be seen that the registration of the first infrared image and the second infrared image is realized through the above process, and then the historical temperature change curve of the target object can be obtained, so as to realize the monitoring of the temperature change trend of the target object.

[0085]

[0086] ​The method for infrared image registration provided in the embodiment comprises: acquiring a first group of images and a second group of images of a target object with marks collected by a parallel dual-optical handheld infrared device; wherein the first group of images comprises a first infrared image and a first visible light image, and the second group of images comprises a second infrared image and a second visible light image; performing registration on the first infrared image and the first visible light image according to the imaging rule of the parallel dual-optical handheld infrared device; performing registration on the second visible light image and the first visible light image according to the marks in the first visible light image and the marks in the second visible light image; and performing registration on the second visible light image and the second infrared image according to the imaging rule, so as to realize registration of the first infrared image and the second infrared image. In the method provided in the embodiment, the first infrared image is first registered with the first visible light image according to the imaging rule, then the second visible light image is registered with the first visible light image according to the marks on the target object, and finally the second visible light image is registered with the second infrared image according to the imaging rule, so as to realize registration of the first infrared image and the second infrared image. It can be seen that, in the method, key points and feature vectors of the key points are not extracted from the infrared image in the process of registration of the infrared image, so compared with the previous method of directly extracting key points from two infrared images and performing feature matching, the method of the application has higher stability and accuracy due to the fact that it is not affected by the resolution of the infrared image, and the success rate of registration of the infrared image is improved. Secondly, the infrared image is a pseudo-color image rendered according to temperature values, and if the temperature difference of the object to be photographed is obvious, the color and texture of the infrared image may have obvious differences. If the previous registration method is used, the feature description vector difference between the key points may be large, thereby affecting the matching effect. However, in the method provided in the application, the color and texture of the pseudo-color image are not limited, so the registration effect is better. In addition, if the previous registration method is used, the best matching key points are extracted from a large number of key points, for example, four matching key points are selected through perspective transformation, and a large number of loops are required to determine whether the key point combination meets the requirements, so the calculation efficiency is low. However, in the method provided in the application, key point feature extraction and key point feature matching are omitted, so the calculation amount is greatly reduced, and the registration efficiency is improved.

[0087] When the first visible light image and the second visible light image are registered, preferably, the target object has a preset number of marks, and registration of the second visible light image and the first visible light image according to the marks in the first visible light image and the marks in the second visible light image comprises:

[0088] The first binary image corresponding to the first visible light image is generated by acquiring three channels of RGB in the first visible light image, and the second binary image corresponding to the second visible light image is generated by acquiring three channels of RGB in the second visible light image.

[0089] Select the first target marker point from the first binary image and the second target marker point from the second binary image, respectively;

[0090] The transformation matrix is ​​determined based on the coordinates of each first target marker point and each second target marker point; the number of first target marker points and the number of second target marker points are determined based on the number of markers on the target object.

[0091] The second visible light image and the first visible light image are registered according to the transformation matrix.

[0092] In practice, when the target object is square, a preferred implementation is to mark it with four circular target markers located at the four corners of the target object.

[0093] Correspondingly, there are 4 first target markers and 4 second target markers.

[0094] In the case marked in red, the pixel values ​​of the R channel in the first visible light image and the pixel values ​​of the R channel in the second visible light image are both greater than the sum of the pixel values ​​of the G channel and the pixel values ​​of the B channel;

[0095] In the case marked in green, the pixel value of the G channel in the first visible light image and the pixel value of the G channel in the second visible light image are both greater than the sum of the pixel values ​​of the R channel and the B channel;

[0096] In the case marked in blue, the pixel values ​​of the B channel in the first visible light image and the pixel values ​​of the B channel in the second visible light image are both greater than the sum of the pixel values ​​of the R channel and the G channel.

[0097] Here, we take four red circular target markers on the target object as an example to illustrate the process of registering the first visible light image and the second visible light image. First, the RGB channels are extracted from the two visible light images respectively, and then a binary image is generated according to formula (2).

[0098]

[0099] Among them, I bin It is the pixel value on the binary image, I r It is the pixel value of the R channel, I b It is the B channel pixel value, I g This refers to the pixel value of the G channel. It should be noted that when the marker on the target object is green, the value in the I channel... bin When = 1, the constraint condition is I. g >I b +I r When the mark on the target object is blue, in Ibin = 1, the constraint condition is I b > I g + I r .

[0100] After obtaining the first binary image and the second binary image, a first target marker point is screened from the first binary image and a second target marker point is screened from the second binary image, that is, four red circular target marker points are screened from the first binary image and the second binary image respectively; after obtaining four corresponding marker point coordinates on the two visible light images respectively, a 3x3 perspective transformation matrix H can be solved, and formula (3) is an expression for mapping the coordinates on the first visible light image to the second visible light image.

[0101]

[0102] In formula (3), x1 and y1 represent the horizontal coordinate and the vertical coordinate of a pixel point in the first visible light image respectively, and x2 and y2 represent the horizontal coordinate and the vertical coordinate of a corresponding pixel point in the second visible light image respectively.

[0103] In the method provided in the embodiment, the possible positions of the markers can be determined by extracting the first binary image and the second binary image, the first target marker point in the first visible light image and the second target marker point in the second visible light image can be determined by screening, and the registration of the first visible light image and the second visible light image can be realized by coordinate transformation of the first target marker point and the second target marker point.

[0104] In order to determine the first target marker point and the second target marker point, the preferred embodiment is that screening the first target marker point from the first binary image and screening the second target marker point from the second binary image respectively comprises:

[0105] Performing an opening operation on the first binary image and the second binary image using a preset structure element respectively;

[0106] Extracting each first connected domain in the first binary image and each second connected domain in the second binary image respectively;

[0107] Obtaining each first edge point corresponding to each first connected domain and each second edge point corresponding to each second connected domain;

[0108] Obtaining the distance from each first edge point of each first connected domain to the center of the target circumscribed rectangle of the corresponding first connected domain, and obtaining the distance from each second edge point of each second connected domain to the center of the target circumscribed rectangle of the corresponding second connected domain;

[0109] a variance of a sequence composed of distances from all first edge points of each first connected domain to a center of a target circumscribed rectangle of the corresponding first connected domain, and a variance of a sequence composed of distances from all second edge points of each second connected domain to a center of a target circumscribed rectangle of the corresponding second connected domain;

[0110] selecting a preset number of first connected domains with the smallest variance from all the first connected domains and a preset number of second connected domains with the smallest variance from all the second connected domains;

[0111] taking the center points of the preset number of first connected domains with the smallest variance as the first target marker points and taking the center points of the preset number of second connected domains with the smallest variance as the second target marker points.

[0112] The target marker points are determined on the first binary image and the second binary image using the same method, which is described directly below. An open operation is performed on the binary image using a preset structural element, for example, a 5*5 rectangular structural element.

[0113] All connected domains on the binary image are extracted, which can be four-connected domains or eight-connected domains, and are not limited in this embodiment. Four-connected domains are taken as an example in this embodiment. All four-connected domains on the binary image are extracted. Four-connected means that if a center pixel point and an element above, below, left or right of the center pixel point have the same value, the center pixel point and the element are connected, and all connected pixel points form a connected domain.

[0114] The connected domains are traversed, a 3*3 matrix structural element is used to perform an erosion operation on each connected domain, and then an exclusive OR operation is performed on the connected domain before the erosion operation and the connected domain after the erosion operation to obtain edge points in the connected domain. The distance from each edge point to the center of a target circumscribed rectangle of the connected domain is calculated (it should be noted that the target circumscribed rectangle refers to a vertical circumscribed rectangle of the connected domain), and a variance of a sequence composed of all distances is calculated. The four connected domains with the smallest variance among all the connected domains are reserved according to the variance value as a standard, and the center points of the four connected domains are taken as target marker points.

[0115] In the method provided in this embodiment, the target marker points are obtained in the first visible light image and the second visible light image respectively.

[0116] On the basis of the above-mentioned embodiments, in order to quickly determine the first target marker points on the first visible light image and the second target marker points on the second visible light image, a preferred implementation manner is that, after the first connected domains in the first binary image and the second connected domains in the second binary image are extracted respectively, before the corresponding first edge points of each first connected domain and the corresponding second edge points of each second connected domain are obtained, the method for infrared image registration further includes:

[0117] all the first connected domains form a first connected domain set and all the second connected domains form a second connected domain set;

[0118] delete the first connected domain not satisfying the preset requirement from the first connected domain set and delete the second connected domain not satisfying the preset requirement from the second connected domain set;

[0119] perform the step of obtaining each first edge point corresponding to each first connected domain and each second edge point corresponding to each second connected domain on the first connected domain satisfying the preset requirement in the first connected domain set and the second connected domain satisfying the preset requirement in the second connected domain set.

[0120] In implementation, the preset condition at least includes one of the following:

[0121] Condition one: the number of pixel values being 1 is greater than or equal to a first threshold value;

[0122] Condition two: the ratio of the minimum value between the width and the height of the target circumscribed rectangle to the maximum value between the width and the height of the target circumscribed rectangle is greater than or equal to a second threshold value;

[0123] Condition three: the ratio of the number of pixel values being 1 to the area of the corresponding target circumscribed rectangle is within a preset range.

[0124] Similarly, the target mark point is also directly determined from the binary image. All the connected domains form a connected domain set. The above three preset conditions can be considered as three ways of screening the connected domains. It should be noted that in implementation, one or multiple ways can be used to screen the connected domains. When three ways are used to screen the connected domains, preferably, the screening is performed in the order of condition one, condition two and condition three.

[0125] Specifically, for condition one, the connected domain set is traversed to obtain the connected domain whose number of pixel values being 1 is greater than or equal to the first threshold value. In implementation, the connected domain whose number of pixel values being 1 is less than or equal to the first threshold value is deleted from the connected domain set, so that the connected domain whose number of pixel values being 1 is greater than or equal to the first threshold value is obtained. The purpose of this step is to remove small connected domains (which can be considered as removing small noise).

[0126] For condition two, the target circumscribed rectangle also refers to the vertical circumscribed rectangle. The connected domain set is traversed to calculate the vertical circumscribed rectangle of the current connected domain, the width of the rectangle is denoted as w and the height of the rectangle is denoted as h, and formula (4) is the expression of the ratio of the minimum value min(w, h) between the width and the height of the target circumscribed rectangle to the maximum value max(w, h) between the width and the height of the target circumscribed rectangle.

[0127] ratio = min(w, h) / max(w, h) (4)

[0128] The second threshold value is not limited, and in general, for a circular mark, the second threshold value is 0.85. The connected domains with the ratio less than the second threshold value are deleted, and the remaining connected domains are the connected domains with the ratio greater than or equal to the second threshold value. The purpose of this step is to determine the shape of the vertical circumscribed rectangle of the connected domain, and delete the connected domain with a circumscribed rectangle similar to a long strip shape.

[0129] For condition three, the number of pixels with a value of 1 in the current connected domain is recorded as N, the area of the vertical circumscribed rectangle of the current connected domain is calculated as Area, and the ratio of N / Area is calculated. If the value of N / Area is not in a preset range, for example, [0.7, 0.9], the current connected domain is deleted from the connected domain set. The purpose of this step is to delete the connected domain with a shape that does not meet the requirements (for example, the connected domain with a ratio of N / Area greater than 0.9) or the connected domain with a special shape composed of multiple arcs (for example, the connected domain with a ratio of N / Area less than 0.7).

[0130] In the method provided in the embodiment, the target standard points are determined from the connected domains that meet the preset requirements by deleting the connected domains that do not meet the preset requirements from the connected domain set, so that the positions of the target mark points can be determined more quickly and accurately.

[0131] After the target mark points are determined in the above embodiment, in order to verify the accuracy of the selected target mark points, a preferred implementation manner is that, before the transformation matrix is determined according to the coordinates of the first target mark points and the coordinates of the second target mark points, the infrared image registration method further includes:

[0132] obtaining all first rectangles circumscribed by the first target mark points, and obtaining all second rectangles circumscribed by the second mark points;

[0133] selecting a first rectangle with the smallest area from all the first rectangles, and selecting a second rectangle with the smallest area from all the second rectangles;

[0134] obtaining distances from the first target mark points to the vertices of the first rectangle with the smallest area, and obtaining distances from the second target mark points to the vertices of the second rectangle with the smallest area;

[0135] assigning the first target mark points to the vertices of the first rectangle with the smallest area closest in distance, and assigning the second target mark points to the vertices of the second rectangle with the smallest area closest in distance;

[0136] in a case where each of the vertices of the first rectangle with the smallest area is assigned a first target mark point, and each of the vertices of the second rectangle with the smallest area is assigned a second target mark point, entering the step of determining the transformation matrix according to the coordinates of the first target mark points and the coordinates of the second target mark points.

[0137] The step of the embodiment is to determine whether the target mark point recognition is correct. Taking the obtained four target mark points as an example, a minimum area circumscribed rectangle (which can be a vertical circumscribed rectangle or an oblique circumscribed rectangle) is obtained, and then the distances from the four target mark points to the four vertices of the circumscribed rectangle are calculated in sequence, and the mark points are assigned to the vertices of the circumscribed rectangle closest to the mark points. After the assignment is completed, if each vertex of the circumscribed rectangle contains one point, the upper left corner of the circumscribed rectangle corresponds to the first point, and the coordinates of the four points are returned in a clockwise order. If a vertex of the circumscribed rectangle does not assign a mark point, it means that the selected target mark point is incorrect, and the process of determining the target mark point needs to be returned to the above process.

[0138] In order to enable those skilled in the art to better understand the overall process of the visible light image, the process of visible light registration will be described again in combination with the embodiment.

[0139] (1) Extract the RGB three channels of the visible light picture, and generate a binary image according to the above formula (2);

[0140] (2) Perform an opening operation on the binary image using a preset structural element, for example, using a 5x5 rectangular structural element to perform an opening operation;

[0141] (3) Extract all four-connected regions on the binary image, and all connected pixel points form a connected domain. All connected domains form a connected domain set;

[0142] (4) Traverse the connected domain set, calculate the number of pixel values of 1 in the current connected domain, and delete the connected domain with the number of pixel values of 1 less than the preset threshold from the connected domain set;

[0143] (5) Traverse the connected domain set, calculate the vertical circumscribed rectangle of the current connected domain, and the rectangle width and height are denoted as w and h, and the value of ratio in formula (4) is calculated. If the ratio is less than the preset threshold, for example, 0.85, the current connected domain is deleted from the connected domain set;

[0144] (6) Traverse the connected domain set, record the number of pixel values of 1 in the current connected domain as N, calculate the area of the vertical circumscribed rectangle of the current region as Area, and if the value of N / Area is not in the preset interval range, for example, [0.7, 0.9], the current connected domain is deleted from the connected domain set;

[0145] (7) Traverse the connected component set, and perform erosion operation on each connected component using a 3*3 rectangular structure element, and then perform XOR operation with the connected component before the erosion operation to obtain edge points in the connected component, calculate the distance from the edge points to the center of the vertical outer rectangle of the connected component, calculate the variance of the sequence composed of all distances, and take the variance value as a standard, and retain the four connected components with the smallest variance in all connected components, and take the center points of the four connected components as marker points.

[0146] (8) The step judges whether the marker point recognition is correct. A minimum area outer rectangle is obtained from the four marker points obtained in the step (7), then the distances between the four marker points and the four vertices of the outer rectangle are calculated in turn, and the marker points are distributed to the vertices of the outer rectangle with the closest distance. After the distribution is completed, if each vertex of the outer rectangle contains one point, the upper left corner of the outer rectangle corresponds to the first point, and the coordinates of the four points are returned in a clockwise order. If a vertex of the outer rectangle does not distribute a marker point, the algorithm fails.

[0147] After obtaining the four corresponding marker point coordinates on the two visible light images respectively, a 3*3 perspective transformation matrix H can be solved, and the coordinates on the first visible light image are mapped to the second visible light image through the above formula (3).

[0148] In the method provided in the embodiment, the accuracy of the obtained target marker point is verified, so that the registration of the first visible light image to the second visible light image is realized according to the more accurate target marker point obtained.

[0149] In the above embodiment, the method for infrared image registration is described in detail, and the application also provides an infrared image registration device and a corresponding embodiment of a parallel dual-light handheld infrared device. It should be noted that the embodiments of the device part are described from two angles, one is based on the functional module angle, and the other is based on the hardware angle.

[0150] Figure 4 The structural diagram of the infrared image registration device provided in an embodiment of the application. The embodiment is based on the functional module angle, and includes:

[0151] The acquisition module 10 is configured to acquire a first group of images and a second group of images of a target object with markers collected by the parallel dual-light handheld infrared device; the first group of images includes a first infrared image and a first visible light image, and the second group of images includes a second infrared image and a second visible light image.

[0152] The first registration module 11 is configured to register the first infrared image and the first visible light image according to the imaging rule of the parallel dual-light handheld infrared device.

[0153] The second registration module 12 is configured to register the second visible light image and the first visible light image according to the mark in the first visible light image and the mark in the second visible light image.

[0154] The third registration module 13 is configured to register the second visible light image and the second infrared image according to the imaging rule, so as to realize registration of the first infrared image and the second infrared image.

[0155] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, which will not be described here again, and the effects are the same as above.

[0156] Figure 5 A structure diagram of a parallel dual-light handheld infrared device is provided for another embodiment of the present application. The embodiment is based on a hardware perspective, as shown in the figure, the parallel dual-light handheld infrared device comprises: Figure 5

[0157] The memory 20 is configured to store a computer program.

[0158] The processor 21 is configured to implement the steps of the method for infrared image registration mentioned in the above embodiments when executing the computer program.

[0159] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 21 can also include a main processor and a coprocessor, the main processor is a processor for processing data in the wake-up state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 can be integrated with a graphics processor (GPU) for rendering and drawing the content to be displayed by the display screen. In some embodiments, the processor 21 can also include an artificial intelligence (AI) processor for processing machine learning-related computing operations.

[0160] ​The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the infrared image registration method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the infrared image registration method mentioned above.

[0161] In some embodiments, the parallel dual-light handheld infrared device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0162] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on parallel dual-beam handheld infrared devices and may include more or fewer components than illustrated.

[0163] The parallel dual-light handheld infrared device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: an infrared image registration method, with the same effect as above.

[0164] This application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0165] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] The computer readable storage medium provided in the application includes the infrared image registration method mentioned above, and the effects are the same.

[0167] The infrared image registration method, device, medium and parallel dual-optical handheld infrared equipment provided by the application are described in detail above. Each embodiment in the specification is described in a progressive manner, and each embodiment mainly explains the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the application, the application can be improved and modified, and these improvements and modifications also fall within the protection scope of the claims of the application.

[0168] It should be further noted that in the specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method of infrared image registration, characterized by, The method comprises: acquiring a first set of images and a second set of images of a target object with markers collected by a parallel dual-optical handheld infrared device; wherein the first set of images comprises a first infrared image and a first visible light image, and the second set of images comprises a second infrared image and a second visible light image; registering the first infrared image and the first visible light image according to the imaging rule of the parallel dual-optical handheld infrared device; registering the second visible light image and the first visible light image according to the markers in the first visible light image and the markers in the second visible light image; registering the second visible light image and the second infrared image according to the imaging rule to realize registration of the first infrared image and the second infrared image.

2. The method of infrared image registration according to claim 1, wherein, The target object has a preset number of markers, and the registration of the second visible light image and the first visible light image according to the markers in the first visible light image and the markers in the second visible light image comprises: acquiring RGB three channels in the first visible light image to generate a first binary image corresponding to the first visible light image, and acquiring RGB three channels in the second visible light image to generate a second binary image corresponding to the second visible light image; selecting first target marker points from the first binary image and selecting second target marker points from the second binary image, respectively; determining a transformation matrix according to the coordinates of the first target marker points and the coordinates of the second target marker points; wherein the number of the first target marker points and the number of the second target marker points are determined according to the number of the markers on the target object; registering the second visible light image and the first visible light image according to the transformation matrix.

3. The method of infrared image registration according to claim 2, wherein, The selection of the first target marker points from the first binary image and the selection of the second target marker points from the second binary image comprises: performing an opening operation on the first binary image and the second binary image using a preset structural element, respectively; extracting each first connected domain in the first binary image and each second connected domain in the second binary image, respectively; acquiring each first edge point corresponding to each first connected domain and each second edge point corresponding to each second connected domain; acquiring the distance from each first edge point of each first connected domain to the center of the target circumscribed rectangle of the corresponding first connected domain, and acquiring the distance from each second edge point of each second connected domain to the center of the target circumscribed rectangle of the corresponding second connected domain; acquiring the variance of the sequence composed of the distances from all first edge points of each first connected domain to the center of the target circumscribed rectangle of the corresponding first connected domain, and the variance of the sequence composed of the distances from all second edge points of each second connected domain to the center of the target circumscribed rectangle of the corresponding second connected domain; selecting a preset number of first connected domains with the smallest variance from all first connected domains, and selecting a preset number of second connected domains with the smallest variance from all second connected domains; The preset number of first connected domain center points with the minimum variance are taken as the first target marker points, and the preset number of second connected domain center points with the minimum variance are taken as the second target marker points.

4. The method of infrared image registration according to claim 3, wherein, After the respective first connected domains in the first binary image and the respective second connected domains in the second binary image are extracted, before the respective first edge points corresponding to the respective first connected domains and the respective second edge points corresponding to the respective second connected domains are obtained, the method further comprises: all the first connected domains are grouped into a first connected domain set, and all the second connected domains are grouped into a second connected domain set; first connected domains not satisfying a preset requirement are deleted from the first connected domain set, and second connected domains not satisfying the preset requirement are deleted from the second connected domain set; the first connected domains satisfying the preset requirement in the first connected domain set and the second connected domains satisfying the preset requirement in the second connected domain set are subjected to the step of obtaining the respective first edge points corresponding to the respective first connected domains and the respective second edge points corresponding to the respective second connected domains.

5. The method of infrared image registration according to claim 4, wherein, The preset condition at least comprises one of the following: condition one: the number of pixel values being 1 is greater than or equal to a first threshold value; condition two: a ratio of a minimum value of a width and a height of the target circumscribed rectangle to a maximum value of the width and the height is greater than or equal to a second threshold value; condition three: a ratio of the number of pixel values being 1 to an area of the target circumscribed rectangle is within a preset range.

6. The method of infrared image registration according to any one of claims 1 to 5, characterized in that, In the case that the target object is a square, the marker is four circular target marker points, and is located at four corners of the target object. Correspondingly, the number of the first target marker points and the number of the second target marker points are both four. In the case that the marker is a red marker, pixel values of an R channel in the first visible light image and pixel values of the R channel in the second visible light image are both greater than a sum of pixel values of a G channel and pixel values of a B channel. In the case that the marker is a green marker, pixel values of a G channel in the first visible light image and pixel values of the G channel in the second visible light image are both greater than a sum of pixel values of an R channel and pixel values of a B channel. In the case that the marker is a blue marker, pixel values of a B channel in the first visible light image and pixel values of the B channel in the second visible light image are both greater than a sum of pixel values of an R channel and pixel values of a G channel.

7. The method of infrared image registration according to claim 6, wherein, Before the transformation matrix is determined according to the coordinates of the respective first target marker points and the coordinates of the respective second target marker points, the method further comprises: all first rectangles circumscribed by the respective first target marker points are obtained, and all second rectangles circumscribed by the respective second marker points are obtained; the first rectangle with the minimum area is screened from all the first rectangles, and the second rectangle with the minimum area is screened from all the second rectangles; distances from the respective first target marker points to vertices of the first rectangle with the minimum area are obtained, and distances from the respective second target marker points to vertices of the second rectangle with the minimum area are obtained; assign each of the first target mark points to a vertex of the smallest-area first rectangle closest thereto, and assign each of the second target mark points to a vertex of the smallest-area second rectangle closest thereto; if each vertex of the smallest-area first rectangle is assigned with the first target mark point and each vertex of the smallest-area second rectangle is assigned with the second target mark point, proceed to the step of determining a transformation matrix according to the coordinates of each of the first target mark points and the coordinates of each of the second target mark points.

8. An apparatus for infrared image registration, characterized by The method comprises the steps of: obtaining a first group of images and a second group of images of a target object with marks collected by a parallel dual-optical handheld infrared device; wherein the first group of images comprises a first infrared image and a first visible light image, and the second group of images comprises a second infrared image and a second visible light image; performing registration on the first infrared image and the first visible light image according to an imaging rule of the parallel dual-optical handheld infrared device; performing registration on the second visible light image and the first visible light image according to the marks in the first visible light image and the marks in the second visible light image; performing registration on the second visible light image and the second infrared image according to the imaging rule, so as to realize registration of the first infrared image and the second infrared image.

9. A parallel dual optical hand-held infrared device, characterized by The method comprises the steps of: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the method for infrared image registration according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to realize the steps of the method for infrared image registration according to any one of claims 1 to 7.

Images