A thermal imaging and visible light registration method based on hand features

Abstract

The present invention discloses a thermal imaging and visible light registration method based on hand features, which belongs to the field of image processing technology, and includes: obtaining hand images taken by a visible light camera and a thermal imaging camera at the same time and the same scene, respectively, to obtain a visible light image corresponding to the visible light camera and a thermal imaging image corresponding to the thermal imaging camera; extracting key areas of the hand from the visible light image and the thermal imaging image respectively; extracting key point features from the visible light image and the thermal imaging image extracted through the key areas of the hand respectively; calculating the homography matrix corresponding to the visible light image through the key point features of the visible light image, calculating the homography matrix corresponding to the thermal imaging image through the key point features of the thermal imaging image, and registering using the homography matrix corresponding to the visible light image and the homography matrix corresponding to the thermal imaging image. The thermal imaging and visible light registration method based on hand features solves the problem that the stability of the existing registration methods is difficult to ensure.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a thermal imaging and visible light registration method based on hand features. Background Art

[0002] At present, the registration methods of thermal imagers can be roughly divided into two categories, one is to use calibration plates for registration, and the other is to use contour features for registration. The calibration plate method has high accuracy, but it requires the collection of multiple sets of images and the operation is more complicated. The method based on contour features requires that the edge shapes in the thermal imaging and visible light images are roughly the same. This method is greatly affected by the environment. If the object contour is clear, but the object temperature is low, the edge of the RGB image is clear, but the edge in the thermal imaging image is difficult to extract. In response to this problem, most of the improvements are now directed to improving edge extraction methods, such as the Chinese invention patent "A method for registering infrared and visible light images" with application number 202211115537.3 and the Chinese invention patent "A method for synchronous registration of thermal infrared and visible light images based on time and space unification" with application number 202210366866.9, but the more complex the calculation in the algorithm, the more difficult it is to ensure stability. Summary of the invention

[0003] In order to overcome the defects of the prior art, the present invention provides a thermal imaging and visible light registration method based on hand features to solve the above problems.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a thermal imaging and visible light registration method based on hand features, comprising the following steps:

[0005] S1: Acquire hand images taken by a visible light camera and a thermal imaging camera at the same time and in the same scene, respectively, to obtain a visible light image corresponding to the visible light camera and a thermal imaging image corresponding to the thermal imaging camera;

[0006] S2: Extract the key areas of the hand from the visible light image and thermal imaging image respectively;

[0007] S3: extract key point features from the visible light image and thermal imaging image respectively through the key areas of the hand;

[0008] S4: Calculate the homography matrix corresponding to the visible light image through the key point features of the visible light image, calculate the homography matrix corresponding to the thermal imaging image through the key point features of the thermal imaging image, and use the homography matrix corresponding to the visible light image and the homography matrix corresponding to the thermal imaging image for registration.

[0009] Preferably, in step S2, for the thermal imaging image TH, a threshold is set to extract the hand mask TH in the thermal imaging image.hand ; Mask the hand TH hand Perform an AND operation with the thermal imaging image TH to extract the key hand area TH of the thermal imaging image TH roi .

[0010] Optionally, in step S2, when the thermal imaging acquisition mode is the white-hot mode, the thermal imaging image TH is first converted into a grayscale image TH gray ; Set the grayscale threshold to 150 and perform TH gray Binarize the image and get the binary image TH thresh ; For the binary image TH thresh Perform hole filling operation to obtain image TH thresh_holes_filled , and keep the image TH thresh_holes_filled The maximum connected domain in the image is used to remove noise and obtain the hand mask TH corresponding to the thermal imaging image TH. hand .

[0011] Specifically, in step S2, for the visible light image IMG, a hand mask IMG in the visible light image is extracted according to the skin color feature. hand ; Set the hand mask IMG hand Perform AND operation with the thermal imaging image IMG to extract the hand key area IMG of the visible light image IMG roi .

[0012] It is worth noting that in step S2, the step of extracting the hand mask in the visible light image includes: converting the visible light image IMG into a YCrCb image, setting the grayscale threshold to 135, binarizing the Cr image in the YCrCb image, and obtaining a binarized image IMG thresh ; For the binary image IMG thresh Perform hole filling operation to obtain image IMG thresh_holes_filled , and keep the image IMG thresh_holes_filled The maximum connected domain in is used to remove noise and obtain the hand mask IMG corresponding to the visible light image IMG. hand .

[0013] Preferably, in step S2, the binary image P thresh The steps of performing the hole filling operation based on the flood filling algorithm include: thresh Negate and get P thresh_inv , filter out P thresh_inv The largest connected region in the thresh_inv Any point in the maximum connected domain of P is used as the seed point SEED; SEED is used to thresh Perform a flood filling operation, set the filled pixel value to 255, and obtain image Pthresh_flood_filled ; Then for image P thresh_flood_filled Take the inverse and get P thresh_flood_filled_inv ; P thresh_flood_filled_inv With the binary image P thresh The result after addition is P thresh_holes_filled As for the binary image P thresh The output of hole filling; where the binary image P thresh The binary image TH corresponding to the thermal imaging image thresh Or the binary image IMG corresponding to the visible light image thresh .

[0014] Specifically, in step S2, before extracting the key area of ​​the hand, the size of the visible light image is adjusted to be consistent with the size of the thermal imaging image.

[0015] It is worth noting that in step S3, the hand key point detection algorithm in OpenCV is used to identify the hand key area TH roi and hand key area IMG roi And extract the key point features of the fingertips corresponding to the five fingers of the hand.

[0016] Preferably, in step S4, the key hand area TH roi The key point features of the corresponding five fingertips construct the feature point group Points src , the hand key area IMG roi The key point features of the corresponding five fingertips construct the feature point group Points target ; For feature point groups Points src And feature point group Points target Use the minimum mean square error to obtain the feature point group Points src The corresponding homography matrix H TH And feature point group Points target The corresponding homography matrix H IMG ; Finally, the homography matrix H corresponding to the thermal imaging image TH The matrix after projection transformation is projected to the homography matrix H corresponding to the visible light image IMG for registration.

[0017] Optionally, in step S1, hand ornaments and metal jewelry are first removed, and then the five fingers of the hand are spread out, and the hand is photographed simultaneously using a visible light camera and a thermal imaging camera to obtain a visible light image and a thermal imaging image.

[0018] The beneficial effect of the present invention is that in the thermal imaging and visible light registration method based on hand features, thermal imaging and visible light are used to simultaneously collect hand images in the same scene, and registration is performed according to hand feature points. After long-term use, users can calibrate themselves according to the hand images. The hand, as a heat source, has clearer edges in both visible light images and thermal imaging images. At the same time, a registration process is formed according to the characteristics of the hand, which greatly reduces the requirements for the registration environment, simplifies the calculation process, and thus ensures the stability of the registration method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a thermal imaging and visible light registration method based on hand features in one embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of key point features of a key area of ​​a hand in one embodiment of the present invention;

[0021] Figure 3 Schematic diagram of a process of extracting a hand mask from a thermal imaging image in one embodiment of the present invention;

[0022] Figure 4 Schematic diagram of a process for extracting a hand mask from a visible light image in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] like Figure 1-4 As shown, a thermal imaging and visible light registration method based on hand features includes the following steps:

[0025] S1: Acquire hand images taken by a visible light camera and a thermal imaging camera at the same time and in the same scene, respectively, to obtain a visible light image corresponding to the visible light camera and a thermal imaging image corresponding to the thermal imaging camera;

[0026] In step S1, hand decorations and metal jewelry are first removed, and then in a scene with a relatively simple background, the five fingers of the hand are spread out, and the hand is photographed simultaneously using a visible light camera and a thermal imaging camera to obtain a visible light image and a thermal imaging image.

[0027] S2: Extract the key areas of the hand from the visible light image and the thermal imaging image respectively;

[0028] In step S2, after extracting the key areas of the hands in the thermal imaging image and the visible light image respectively by combining a deep learning method or a traditional image processing method, preprocessing is performed to make the hand sizes in the preprocessed thermal imaging image and the visible light image consistent; the preprocessing includes eliminating background noise, clothing noise and completing the hand area.

[0029] In step S2, before extracting the key area of ​​the hand, the size of the visible light image is adjusted to be consistent with the size of the thermal imaging image.

[0030] In step S2, for the thermal imaging image TH, since the hand temperature is higher than the ambient temperature, a threshold is set to extract the hand mask TH in the thermal imaging image. hand ; For example, when the thermal imaging acquisition mode is white-hot mode, the higher the temperature, the whiter the image block color, that is, the temperature is proportional to the grayscale value; if the image grayscale value is represented by 8 bits, the grayscale value range is 0-255, and the grayscale threshold is set to 150 to extract the hand image from the thermal imaging image; the hand mask TH hand Perform an AND operation with the thermal imaging image TH to extract the key hand area TH of the thermal imaging image TH roi .

[0031] In step S2, Figure 3 As shown, when the thermal imaging acquisition mode is the white-hot mode, the thermal imaging image TH is first converted into a grayscale image TH gray ; Set the grayscale threshold to 150 and perform TH gray Binarize the image and get the binary image TH thresh ; For the binary image TH thresh Perform hole filling operation to obtain image TH thresh_holes_filled , and keep the image TH thresh_holes_filled The maximum connected domain in the image is used to remove noise and obtain the hand mask TH corresponding to the thermal imaging image TH. hand In this embodiment, the image TH thresh_holes_filled The largest connected domain in the image should be the hand, so the noise can be removed by statistically comparing the sizes of the connected domains, further retaining the largest connected domain and removing other connected domains.

[0032] In step S2, for the visible light image IMG, a hand mask IMG in the visible light image is extracted according to the skin color feature. hand ; Set the hand mask IMG hand Perform AND operation with the thermal imaging image IMG to extract the hand key area IMG of the visible light image IMG roi .

[0033] In step S2, Figure 4 As shown, the step of extracting the hand mask in the visible light image includes: converting the visible light image IMG into a YCrCb image, setting the grayscale threshold to 135, performing image binarization on the Cr image in the YCrCb image, and obtaining a binarized image IMG thresh , where the YCrCb image is a three-channel image, the three channels of the YCrCb image are Y, Cr and Cb, and the Cr image is the second image in the three channels of the YCrCb image; for the binary image IMG thresh Perform hole filling operation to obtain image IMG thresh_holes_filled , and keep the image IMG thresh_holes_filled The maximum connected domain in is used to remove noise and obtain the hand mask IMG corresponding to the visible light image IMG. hand In this embodiment, the image IMG thresh_holes_filled The largest connected domain in the image should be the hand, so the noise can be removed by statistically comparing the sizes of the connected domains, further retaining the largest connected domain and removing other connected domains.

[0034] In step S2, the binary image P thresh The steps for performing a hole filling operation based on the flood filling algorithm include:

[0035] For the binary image P thresh Negate and get P thresh_inv , filter out P thresh_inv The largest connected region in the thresh_inv Any point in the maximum connected domain of P is used as the seed point SEED; SEED is used to thresh Perform a flood filling operation, set the filled pixel value to 255, and obtain image P thresh_flood_filled ; Then for image P thresh_flood_filled Take the inverse and get P thresh_flood_filled_inv ; P thresh_flood_filled_inv With the binary image P thresh The result after addition is P thresh_holes_filled As for the binary image P thresh The output of hole filling; where the binary image P thresh The binary image TH corresponding to the thermal imaging image thresh Or the binary image IMG corresponding to the visible light image thresh , P thresh_holes_filled The image TH corresponding to the thermal imaging image thresh_holes_filled Or the image IMG corresponding to the visible light image thresh_holes_filled .

[0036] That is, in the binary image TH corresponding to the thermal imaging image threshThe steps for performing a hole filling operation based on the flood filling algorithm are as follows:

[0037] For the binary image TH thresh Negate to get TH thresh_inv , filter out TH thresh_inv The largest connected region in the thresh_inv Any point in the maximum connected domain of TH is used as the seed point SEED; SEED is used to thresh Perform a flood filling operation, set the filled pixel value to 255, and get the image TH thresh_flood_filled ; Then the image TH thresh_flood_filled Take the inverse and get TH thresh_flood_filled_inv ; TH thresh_flood_filled_inv With the binary image TH thresh The result after addition is TH thresh_flood_filled As for the binary image TH thresh Output of hole filling.

[0038] Binarized image IMG corresponding to the visible light image thresh The steps for performing a hole filling operation based on the flood filling algorithm are as follows:

[0039] Binarized image IMG thresh Negate and get IMG thresh_inv , filter out IMG thresh_inv The largest connected region in IMG thresh_inv Any point in the maximum connected domain of IMG is used as the seed point SEED; SEED is used to thresh Perform a flood fill operation, set the filled pixel value to 255, and get the image IMG thresh_flood_filled ; Then for the image IMG thresh_flood_filled Negate and get IMG thresh_flood_filled_inv ; IMG thresh_flood_filled_inv With the binary image IMG thresh The result after addition is IMG thresh_flood_filled As for the binary image IMG thresh Output of hole filling.

[0040] S3: extract key point features from the visible light image and thermal imaging image respectively through the key areas of the hand;

[0041] In step S3, the hand key point detection algorithm in OpenCV is used to identify the hand key area TH roi and hand key area IMG roi The key point features in Figure 2As shown, the key point features of the fingertips corresponding to the five fingers of the hand are extracted. In the step S3, the edges of the hand mask in the visible light image and the thermal imaging image extracted through the key area of ​​the hand are first extracted, and then the key point features in the edges of the hand mask in the visible light image and the thermal imaging image extracted through the key area of ​​the hand are calculated respectively.

[0042] S4: Calculate the homography matrix corresponding to the visible light image through the key point features of the visible light image, calculate the homography matrix corresponding to the thermal imaging image through the key point features of the thermal imaging image, and use the homography matrix corresponding to the visible light image and the homography matrix corresponding to the thermal imaging image for registration.

[0043] In step S4, the key hand area TH roi The key point features of the corresponding five fingertips construct the feature point group Points src , the hand key area IMG roi The key point features of the corresponding five fingertips construct the feature point group Points target ; For feature point groups Points src And feature point group Points target Use the minimum mean square error to obtain the feature point group Points src The corresponding homography matrix H TH And feature point group Points target The corresponding homography matrix H IMG , where the homography matrix H TH And the homography matrix H IMG are all matrices of 3 rows and 3 columns; finally, the homography matrix H corresponding to the thermal imaging image TH The matrix after projection transformation is projected to the homography matrix H corresponding to the visible light image IMG for registration.

[0044] In this embodiment, a point on the thermal imaging image is set as P0 (x0, y0), and the coordinates after registration are P1 (x1, y1 ) The calculation formula is: Where H in the formula is the homography matrix corresponding to the thermal imaging image.

[0045] In the thermal imaging and visible light registration method based on hand features, not many registration tools, such as calibration plates, are required; since accurate extraction of key point features is a prerequisite for registration, and fingertip features are relatively easy to extract and not easily affected by other noises, the registration effect is stable. Use thermal imaging and visible light to simultaneously capture hand images in the same scene, and perform registration based on hand feature points. This registration scheme is applicable to handheld thermal imagers, that is, the positions of the thermal imaging camera and the visible light camera are relatively fixed. After long-term use, users can calibrate by themselves based on the hand image. As a heat source, the hand has clearer edges in both thermal imaging images and thermal imaging images. At the same time, a registration process is formed based on the characteristics of the hand, which greatly reduces the requirements for the registration environment, simplifies the calculation process, and thus ensures the stability of the registration method.

[0046] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A thermal imaging and visible light registration method based on hand features, characterized in that: The following steps are involved: S1: Acquire hand images taken by a visible light camera and a thermal imaging camera at the same time and in the same scene, respectively, to obtain a visible light image corresponding to the visible light camera and a thermal imaging image corresponding to the thermal imaging camera; S2: Extract the key areas of the hand from the visible light image and thermal imaging image respectively; S3: extract key point features from the visible light image and thermal imaging image respectively through the key areas of the hand; S4: Calculate the homography matrix corresponding to the visible light image through the key point features of the visible light image, calculate the homography matrix corresponding to the thermal imaging image through the key point features of the thermal imaging image, and use the homography matrix corresponding to the visible light image and the homography matrix corresponding to the thermal imaging image for registration.

2. The method for thermal imaging and visible light registration based on hand features according to claim 1, characterized in that: In step S2, for the thermal imaging image TH, a threshold is set to extract the hand mask TH in the thermal imaging image. hand ; Mask the hand TH hand Perform AND operation with the thermal imaging image TH to extract the key hand area TH of the thermal imaging image TH roi .

3. The method for thermal imaging and visible light registration based on hand features according to claim 2, characterized in that: In step S2, when the thermal imaging acquisition mode is the white-hot mode, the thermal imaging image TH is first converted into a grayscale image TH gray ; Set the grayscale threshold to 150 and perform TH gray Binarize the image and get the binary image TH thresh ; For the binary image TH thresh Perform hole filling operation to obtain image TH thresh_holes_filled , and keep the image TH thresh_holes_filled The maximum connected domain in the image is used to remove noise and obtain the hand mask TH corresponding to the thermal imaging image TH. hand .

4. The method for thermal imaging and visible light registration based on hand features according to claim 3, characterized in that: In step S2, for the visible light image IMG, a hand mask IMG in the visible light image is extracted according to the skin color feature. hand ; Set the hand mask IMG hand Perform AND operation with the thermal imaging image IMG to extract the hand key area IMG of the visible light image IMG roi .

5. The method for thermal imaging and visible light registration based on hand features according to claim 4, characterized in that: In step S2, the step of extracting the hand mask in the visible light image includes: The visible light image IMG is converted into a YCrCb image, the grayscale threshold is set to 135, and the Cr image in the YCrCb image is binarized to obtain the binary image IMG thresh ; For the binary image IMG thresh Perform hole filling operation to obtain image IMG thresh_holes_filled , and keep the image IMG thresh_holes_filled The maximum connected domain in is used to remove noise and obtain the hand mask IMG corresponding to the visible light image IMG. hand .

6. The method for thermal imaging and visible light registration based on hand features according to claim 5, characterized in that: In step S2, the binary image P thresh The steps of performing the hole filling operation based on the flood filling algorithm include: thresh Negate and get P thresh_inv , filter out P thresh_inv The largest connected region in the thresh_inv Any point in the maximum connected domain of P is used as the seed point SEED; SEED is used to thresh Perform a flood filling operation, set the filled pixel value to 255, and obtain image P thresh_flood_filled ; Then for image P thresh_flood_filled Take the inverse and get P thresh_flood_filled_inv ; P thresh_flood_filled_inv With the binary image P thresh The result after addition is P thresh_flood_filled As for the binary image P thresh The output of hole filling; where the binary image P thresh The binary image TH corresponding to the thermal imaging image thresh Or the binary image IMG corresponding to the visible light image thresh .

7. The method for thermal imaging and visible light registration based on hand features according to claim 6, characterized in that: In step S2, before extracting the key area of ​​the hand, the size of the visible light image is adjusted to be consistent with the size of the thermal imaging image.

8. The method for thermal imaging and visible light registration based on hand features according to claim 7, characterized in that: In step S3, the hand key point detection algorithm in OpenCV is used to identify the hand key area TH roi and hand key area IMG roi And extract the key point features of the fingertips corresponding to the five fingers of the hand.

9. The method for thermal imaging and visible light registration based on hand features according to claim 8, characterized in that: In step S4, the key hand area TH roi The key point features of the corresponding five fingertips construct the feature point group Points src , the hand key area IMG roi The key point features of the corresponding five fingertips construct the feature point group Points target ; For feature point groups Points src And feature point group Points target Use the minimum mean square error to obtain the feature point group Points src The corresponding homography matrix H TH And feature point group Points target The corresponding homography matrix H IMG ; Finally, the homography matrix H corresponding to the thermal imaging image TH The matrix after projection transformation is projected to the homography matrix H corresponding to the visible light image IMG for registration.

10. The method for thermal imaging and visible light registration based on hand features according to claim 9, characterized in that: In step S1, hand ornaments and metal jewelry are first removed, and then the five fingers of the hand are spread out, and the hand is photographed simultaneously using a visible light camera and a thermal imaging camera to obtain a visible light image and a thermal imaging image.

Images