Visible light image-thermal infrared image matching method and system for rice phenotype

Abstract

The invention discloses a visible light image-thermal infrared image matching method and system for rice phenotypes. The method comprises the following steps: acquiring a plurality of visible light images and thermal infrared images of rice; determining fuzzy areas and non-mask areas of the visible light image and the thermal infrared image; extracting feature points and descriptors of a non-mask area, obtaining matched feature point pairs on the visible light image and the thermal infrared image, and connecting the matched feature point pairs to obtain a connecting line; screening out connecting lines which simultaneously meet the length and angle threshold intervals of the connecting lines; according to fitting points of the feature points of the visible light image and the thermal infrared image mapped to the visible light image after passing through a homography matrix, generating a residual error to screen the feature points of the visible light image close to a correct position; and finally, connecting the feature points on the visible light image with the corresponding feature points on the thermal infrared image to obtain a connecting line which is a final matching result. According to the invention, the problem of mismatching caused by blurring of a part of thermal infrared images due to insufficient illumination can be avoided.

Description

Technical Field

[0001] The invention relates to the technical field of rice phenotype image registration, and in particular to a rice phenotype visible light image-thermal infrared image matching method and system. Background Art

[0002] Accurately acquiring rice phenotypic images can not only improve agricultural production efficiency, but also effectively ensure food security and quality. Among these tasks, drones have become one of the main methods for obtaining rice phenotypic data due to their high efficiency, accuracy and wide coverage. In drone remote sensing monitoring, the visible light image and thermal infrared image of the drone of the same field are usually registered first, and then image analysis is performed to obtain agricultural information. Image registration can improve the accuracy of disease identification and rice constant inversion, while enhancing image information for easy tracking and monitoring. It has broad application prospects and practical value. However, due to differences in resolution, spectrum and viewpoint, as well as the complexity of field rice phenotypic images, current registration methods often find it difficult to successfully register visible light images and thermal infrared images of rice fields, which urgently needs to be solved. Summary of the invention

[0003] The first purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a visible light image-thermal infrared image matching method for rice phenotype, which can efficiently complete the precise alignment of the visible light image and thermal infrared image of rice, so that the rice image can effectively complete subsequent image stitching operations.

[0004] The second object of the present invention is to provide a visible light image-thermal infrared image matching system for rice phenotype.

[0005] The first object of the present invention is achieved by the following technical solution: A visible light image-thermal infrared image matching method for rice phenotype, comprising the following steps: S1, obtaining several visible light images of rice and their corresponding thermal infrared images; S2, by binarization and threshold matching, the blurred area A1 of the visible light image and the blurred area A2 of the thermal infrared image are respectively screened out, the two blurred areas A1 and A2 are blackened and then added to obtain area A as the mask area of ​​the visible light image and the thermal infrared image, and the areas of the visible light image and the thermal infrared image that are not blackened are the non-mask areas; S3, using a cross-correlation two-stream network model to extract feature points of the non-mask area and their corresponding descriptors, and obtaining matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, connecting the matching feature point pairs to obtain connecting lines; S4, screening out connection lines that meet both the connection line length threshold interval and the connection line angle threshold interval; S5. Generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the connecting lines obtained in step S4. Generate residuals based on the feature points of the visible light image and the thermal infrared image mapped to the fitting points on the visible light image through the homography matrix to screen the feature points of the visible light image that are close to the correct position. The fitting points with larger residuals are most likely to be points close to the correct position. At this time, connect the feature points on the visible light image with the corresponding feature points on the thermal infrared image, and the connecting lines obtained are the final matching results.

[0006] Further, the step S1 comprises the following steps: S101, on the UAV ground station, setting a designated rice field as an aerial photography area, setting a heading overlap rate and a lateral overlap rate of the UAV aerial photography, and the UAV ground station automatically generates a route according to the operation information; S102, the UAV autonomously flies according to the planned route, and automatically collects visible light images and thermal infrared images of rice according to a set overlap rate; S103, the drone uploads the collected visible light image and thermal infrared image remote sensing data of rice to the SD card storage chip of the gimbal in real time; wherein the visible light image and its corresponding thermal infrared image are uploaded in pairs.

[0007] Further, the step S2 comprises the following steps: S201, preprocessing of visible light images and thermal infrared images, including the following steps: S2011, converting the formats of the visible light image and the thermal infrared image into grayscale, the vertical resolution of the visible light image is adjusted to be the same as that of the thermal infrared image, and the horizontal resolution is adjusted according to the original image ratio; S2012, using contrast enhancement to process the visible light image and the thermal infrared image after adjusting the proportions, so that the visible light image and the thermal infrared image have clearer contours and edges, thereby making it easier to detect and locate key points; wherein the contrast enhancement method is as follows: ; In the formula, Input represents the input visible light image or thermal infrared image, and Output represents the output enhanced visible light image or thermal infrared image. and Respectively represent the minimum and maximum pixel values ​​of the input image, minmaxscale is a scaling factor used to control the degree of contrast adjustment; S202, binarize and threshold match the preprocessed visible light image and thermal infrared image, set the pixel points greater than the set value H to 256, and set the pixel points less than or equal to the set value H to 0, count the pixel average value of each binarized area based on the sliding window, the pixel size of each area is 10 in length and width, and the moving step is 2 pixels, set the area contour threshold M, compare with the area contour average value, and take the area less than the area contour threshold M as the blurred area, set the blurred area A1 of the visible light image and the blurred area A1 of the thermal infrared image to black, and then add them to obtain area A, A is the mask area of ​​the visible light image and the thermal infrared image, and the area of ​​the visible light image and the thermal infrared image that is not set to black is the non-mask area; wherein, the mask area and the non-mask area of ​​the visible light image and the thermal infrared image are the same size.

[0008] Further, in step S3, the cross-correlated dual-stream network model includes two symmetrical branches, each branch contains 9 network layers, including 3 convolutional layers, 3 rectified linear unit ReLU functions, 2 maximum pooling layers and 1 average pooling layer; wherein, the first network layer is a convolutional layer, using a convolution kernel with a length of 3, a width of 3, and a step size of 2, the input of the network layer is 3×w×h image data, and the output is c1×w / 2×h / 2 feature data, c1 is the number of convolution kernel channels, w is the image width, and h is the image height; the second network layer is a rectified linear unit ReLU function; the third network layer is a convolutional layer, using a convolution kernel with a length of 3, a width of 3, and a step size of 2, the input of the network layer is 3×w×h image data, and the output is c1×w / 2×h / 2 feature data, c1 is the number of convolution kernel channels, w is the image width, and h is the image height; The first network layer is a maximum pooling layer with a stride of 2. The input of this network layer is the feature data of c1×w / 2×h / 2, and the output is the feature map C1 of c1×w / 4×h / 4; the fourth network layer is a convolution layer, which uses a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C1 of c1×w / 4×h / 4, and the output is the feature data of c2×w / 8×h / 8, where c2 is the number of convolution kernel channels; the fifth network layer is the rectified linear unit ReLU function; the sixth network layer is a maximum pooling layer with a stride of 2. The input of this network layer is the feature map C1, and the output is c2 ×w / 8×h / 8 feature map C2; the 7th network layer is a convolution layer, which uses a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C2 of c2×w / 8×h / 8, and the output is the feature data of c3×w / 16×h / 16, where c3 is the number of convolution kernel channels; the 8th network layer is a rectified linear unit ReLU function; the 9th network layer is an average pooling layer with a stride of 2. The input of this network layer is the feature map C2, and the output is the feature map C3 of c3×w / 16×h / 16; in order to obtain the correlation information between the features of the visible light image and the features of the thermal infrared image The feature maps C3 obtained by the visible light image and the thermal infrared image through their respective branches are compressed into one-dimensional data of 1×(2×c3×w / 16×h / 16), and weighted summation is performed based on the cross-correlation of the one-dimensional data to further convert them into 2×(w / 16×h / 16) feature points and c3×(w / 16×h / 16) descriptors; wherein the inputs at both ends of the cross-correlation two-stream network model are the visible light image and the thermal infrared image of the same size and range, respectively. After the visible light image and the thermal infrared image pass through the cross-correlation two-stream network model, the feature points of their respective non-mask areas and their corresponding descriptors are obtained.

[0009] Further, in step S3, a nearest neighbor search algorithm is used according to the obtained feature points and descriptors to search for matching feature point pairs on the visible light image and the thermal infrared image, wherein feature points with similar descriptors are matching feature point pairs, the descriptors describe the feature vectors of the feature points, and the similarity of the descriptors is measured by cosine similarity, which is the angle between the feature vectors of two feature points. The closer the cosine similarity value is to 1, the more similar the descriptors of the two feature points are. When matching, firstly, a feature point is randomly selected from all the feature points in the thermal infrared image as a reference feature point, and then a target feature point matching the reference feature point and a feature point closest to the target feature point are found from all the feature points in the visible light image as target neighbor points. The feature point matching method is measured by distance. The distance between the reference feature point and the target feature point is called the first distance, and the distance between the reference feature point and the target adjacent feature point is called the second distance. All matching feature point pairs on the visible light image and the thermal infrared image are traversed, and the first distance average avg1 and the second distance average avg2 of all matching feature point pairs are calculated to obtain the difference avg=avg1-avg2; then all matching feature point pairs are screened, and the screening condition for each pair of feature point pairs is: first distance < (second distance-avg). If the condition of first distance < (second distance-avg) is met, the corresponding feature point pair is retained. After the screening is completed, all the matching feature point pairs retained are connected to obtain connecting lines.

[0010] Further, in step S4, the lengths and angles of all connecting lines are calculated, threshold intervals are set for the lengths and angles of the connecting lines, and connecting lines that meet both the connecting line length threshold interval and the connecting line angle threshold interval are screened out; wherein, L is defined as the longest connecting line length minus the shortest connecting line length, the length threshold interval is the minimum connecting line length among all connecting lines plus 1 / 3L to the longest connecting line length among all connecting lines minus 1 / 3L, D is defined as the maximum absolute value of the acute angles formed by all connecting lines with the horizontal line minus the minimum absolute value of the acute angles formed by all connecting lines with the horizontal line, and the angle threshold interval is the minimum angle among all connecting lines plus 1 / 3D to the maximum angle among all connecting lines minus 1 / 3D.

[0011] Further, the step S5 comprises the following steps: S501, generating a homography matrix according to the feature point coordinate matrix of the connecting line on the visible light image and the feature point coordinate matrix on the thermal infrared image obtained in step S4, wherein the homography matrix describes the mapping relationship between the feature points of the visible light image and the feature points of the thermal infrared image; S502, filtering feature points of the visible light image close to the correct position by generating residuals, the residuals are expressed as: ; In the formula, Represents the feature points of the visible light image, Represents the feature points of thermal infrared images After being mapped to the fitting points on the visible light image through the homography matrix, represents the residual, i represents the i-th correct feature point pair, is the number of correct feature point pairs; among them, the fitting point with the larger residual is most likely to be the point close to the correct position. The thermal infrared image is used as the reference image, and the feature points of the thermal infrared image do not need to be transformed; S503, the residual Sort in descending order, find the first n feature points corresponding to the first n residuals in the visible light image, and replace the first n feature points with the corresponding n fitting points on the visible light image obtained after the homography matrix transformation; S504, repeat steps S501 to S503 for updating until the sum of the residuals is equal to zero. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting line is the final matching result.

[0012] The second object of the present invention is achieved by the following technical solution: a rice phenotype visible light image-thermal infrared image matching system, used to implement the above-mentioned rice phenotype visible light image-thermal infrared image matching method, comprising: An image acquisition unit, used for acquiring a plurality of visible light images of rice and their corresponding thermal infrared images; An image mask processing unit, used for preprocessing the visible light image and the thermal infrared image, generating mask areas and non-mask areas of the visible light image and the thermal infrared image; wherein the mask areas and non-mask areas of the visible light image and the thermal infrared image are the same in size; The image feature extraction unit uses a cross-correlation two-stream network model to extract feature points and their corresponding descriptors in the non-mask area, and obtains matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, and connects the matching feature point pairs to obtain connecting lines; A connection line screening unit, used to screen out connection lines that simultaneously meet a connection line length threshold interval and a connection line angle threshold interval; A registration unit is used to generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the selected connecting lines. The feature points of the visible light image and the thermal infrared image are mapped to the fitting points on the visible light image through the homography matrix to generate residuals to screen the feature points of the visible light image close to the correct position, wherein the fitting points with larger residuals are most likely to be points close to the correct position. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting lines are the final matching results.

[0013] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The present invention can effectively realize the registration of visible light images and thermal infrared images of rice drones.

[0014] 2. The present invention sets threshold intervals for the length and angle of the connecting line, and selects connecting lines that simultaneously meet the connecting line length threshold interval and the connecting line angle threshold interval, thereby improving the ability to obtain high-quality connecting lines.

[0015] 3. The present invention utilizes a cross-correlated dual-stream network model to obtain feature points and descriptors of non-masked areas of the image, thereby improving the registration speed while maintaining good registration accuracy.

[0016] 4. The present invention generates residuals based on the feature points of the visible light image and the thermal infrared image mapped to the fitting points on the visible light image through the homography matrix to screen the feature points of the visible light image close to the correct position, thereby reducing the position deviation of the feature points.

[0017] In summary, the present invention has low computational complexity and is easy to implement. It can effectively avoid the problem of partial thermal infrared image blurring caused by insufficient illumination, which in turn causes mismatching. It is a competitive method and system for the registration of visible light and thermal infrared images of rice UAVs. It can make important contributions to the accurate monitoring of rice and the assessment of its health and growth conditions, and is worthy of promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flow chart of the method of the present invention.

[0019] Figure 2 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0021] Example 1 like Figure 1 As shown, this embodiment discloses a visible light image-thermal infrared image matching method for rice phenotype, comprising the following steps: S1. Obtain several visible light images of rice and their corresponding thermal infrared images, as follows: S101, on the UAV ground station, setting a designated rice field as an aerial photography area, setting a heading overlap rate and a lateral overlap rate of the UAV aerial photography, and the UAV ground station automatically generates a route according to the operation information; S102, the UAV autonomously flies according to the planned route, and automatically collects visible light images and thermal infrared images of rice according to a set overlap rate; S103, the drone uploads the collected visible light image and thermal infrared image remote sensing data of rice to the SD card storage chip of the gimbal in real time; wherein the visible light image and its corresponding thermal infrared image are uploaded in pairs.

[0022] S2, through binarization and threshold matching, the blurred area A1 of the visible light image and the blurred area A2 of the thermal infrared image are respectively screened out, and the two blurred areas A1 and A2 are blackened and added to obtain area A as the mask area of ​​the visible light image and the thermal infrared image, while the areas of the visible light image and the thermal infrared image that are not blackened are the non-mask areas, as follows: S201, preprocessing of visible light images and thermal infrared images, including the following steps: S2011, converting the formats of the visible light image and the thermal infrared image into grayscale, the vertical resolution of the visible light image is adjusted to be the same as that of the thermal infrared image, and the horizontal resolution is adjusted according to the original image ratio; S2012, using contrast enhancement to process the visible light image and the thermal infrared image after adjusting the proportions, so that the visible light image and the thermal infrared image have clearer contours and edges, thereby making it easier to detect and locate key points; wherein the contrast enhancement method is as follows: ; In the formula, Input represents the input visible light image or thermal infrared image, and Output represents the output enhanced visible light image or thermal infrared image. and Respectively represent the minimum and maximum pixel values ​​of the input image, minmaxscale is a scaling factor used to control the degree of contrast adjustment; S202, binarize and threshold match the preprocessed visible light image and thermal infrared image, set the pixel points greater than the set value H to 256, and set the pixel points less than or equal to the set value H to 0, count the pixel average value of each binarized area based on the sliding window, the pixel size of each area is 10 in length and width, and the moving step is 2 pixels, set the area contour threshold M, compare with the area contour average value, and take the area less than the area contour threshold M as the blurred area, set the blurred area A1 of the visible light image and the blurred area A1 of the thermal infrared image to black, and then add them to obtain area A, A is the mask area of ​​the visible light image and the thermal infrared image, and the area of ​​the visible light image and the thermal infrared image that is not set to black is the non-mask area; wherein, the mask area and the non-mask area of ​​the visible light image and the thermal infrared image are the same size.

[0023] S3. The cross-correlation two-stream network model is used to extract the feature points of the non-masked area and their corresponding descriptors, and the matching feature point pairs on the visible light image and the thermal infrared image are obtained based on the feature points and descriptors. The matching feature point pairs are connected to obtain connecting lines, as follows: The cross-correlated dual-stream network model includes two symmetrical branches, each of which contains 9 network layers, including 3 convolutional layers, 3 rectified linear unit ReLU functions, 2 maximum pooling layers and 1 average pooling layer; wherein the first network layer is a convolutional layer, using a convolution kernel with a length of 3, a width of 3 and a step size of 2. The input of this network layer is 3×w×h image data, and the output is c1×w / 2×h / 2 feature data, where c1 is the number of convolution kernel channels, w is the image width, and h is the image height; the second network layer is a rectified linear unit ReLU function; the third network layer is a step size of The fourth network layer is a convolutional layer, which uses a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is a feature map C1 of c1×w / 4×h / 4, and the output is a feature map C1 of c2×w / 8×h / 8, where c2 is the number of convolution kernel channels. The fifth network layer is a rectified linear unit ReLU function. The sixth network layer is a maximum pooling layer with a stride of 2. The input of this network layer is a feature map C1, and the output is a feature map C1 of c2×w / 8×h / 8. h / 8 feature map C2; the 7th network layer is a convolution layer, using a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C2 of c2×w / 8×h / 8, and the output is the feature data of c3×w / 16×h / 16, where c3 is the number of convolution kernel channels; the 8th network layer is a rectified linear unit ReLU function; the 9th network layer is an average pooling layer with a stride of 2. The input of this network layer is the feature map C2, and the output is the feature map C3 of c3×w / 16×h / 16; in order to obtain the relevant information between the features of the visible light image and the features of the thermal infrared image, The feature map C3 obtained by the visible light image and the thermal infrared image through their respective branches is compressed into one-dimensional data of 1×(2×c3×w / 16×h / 16), and weighted summation is performed based on the cross-correlation of the one-dimensional data to further convert it into 2×(w / 16×h / 16) feature points and c3×(w / 16×h / 16) descriptors; wherein, the inputs at both ends of the cross-correlation two-stream network model are visible light images and thermal infrared images of the same size and range, respectively. After the visible light image and the thermal infrared image pass through the cross-correlation two-stream network model, the feature points of their respective non-mask areas and their corresponding descriptors are obtained.

[0024] According to the obtained feature points and descriptors, the nearest neighbor search algorithm is used to search for matching feature point pairs on the visible light image and the thermal infrared image, where the feature points with similar descriptors are the matching feature point pairs. The descriptors describe the feature vectors of the feature points. The similarity of the descriptors is measured by cosine similarity, which is the angle between the feature vectors of two feature points. The closer the cosine similarity value is to 1, the more similar the descriptors of the two feature points are. When matching, first randomly select a feature point from all the feature points in the thermal infrared image as the reference feature point, and then find the target feature point that matches the reference feature point and the feature point closest to the target feature point in all the feature points in the visible light image as the target neighboring feature point. The matching method is measured by distance. The distance between the reference feature point and the target feature point is called the first distance, and the distance between the reference feature point and the target adjacent feature point is called the second distance. All matching feature point pairs on the visible light image and the thermal infrared image are traversed, and the first distance average avg1 and the second distance average avg2 of all matching feature point pairs are calculated to obtain the difference avg=avg1-avg2; then all matching feature point pairs are screened, and the screening condition for each pair of feature point pairs is: first distance < (second distance-avg). If the condition of first distance < (second distance-avg) is met, the corresponding feature point pair is retained. After the screening is completed, all the matching feature point pairs retained are connected to obtain connecting lines.

[0025] S4. Filter out the connecting lines that meet both the connecting line length threshold interval and the connecting line angle threshold interval, as follows: Calculate the length and angle of all connecting lines, set threshold intervals for the length and angle of the connecting lines, and screen out connecting lines that meet both the connecting line length threshold interval and the connecting line angle threshold interval; wherein, L is defined as the longest connecting line length minus the shortest connecting line length, the length threshold interval is the minimum connecting line length among all connecting lines plus 1 / 3L to the longest connecting line length among all connecting lines minus 1 / 3L, D is defined as the maximum absolute value of the acute angles formed by all connecting lines with the horizontal line minus the minimum absolute value of the acute angles formed by all connecting lines with the horizontal line, and the angle threshold interval is the minimum angle among all connecting lines plus 1 / 3D to the maximum angle among all connecting lines minus 1 / 3D.

[0026] S5, generating a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the connecting lines obtained in step S4, and generating residuals based on the feature points of the visible light image and the thermal infrared image mapped to the fitting points on the visible light image through the homography matrix to screen the feature points of the visible light image close to the correct position, wherein the fitting points with larger residuals are most likely to be points close to the correct position. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the connecting lines obtained are the final matching results, as follows: S501, generating a homography matrix according to the feature point coordinate matrix of the connecting line on the visible light image and the feature point coordinate matrix on the thermal infrared image obtained in step S4, wherein the homography matrix describes the mapping relationship between the feature points of the visible light image and the feature points of the thermal infrared image; S502, filtering feature points of the visible light image close to the correct position by generating residuals, the residuals are expressed as: ; In the formula, Represents the feature points of the visible light image, Represents the feature points of thermal infrared images After being mapped to the fitting points on the visible light image through the homography matrix, represents the residual, i represents the i-th correct feature point pair, is the number of correct feature point pairs; among them, the fitting point with the larger residual is most likely to be the point close to the correct position. The thermal infrared image is used as the reference image, and the feature points of the thermal infrared image do not need to be transformed; S503, the residual Sort in descending order, find the first n feature points corresponding to the first n residuals in the visible light image, and replace the first n feature points with the corresponding n fitting points on the visible light image obtained after the homography matrix transformation; S504, repeat steps S501 to S503 for updating until the sum of the residuals is equal to zero. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting line is the final matching result.

[0027] Example 2 This embodiment discloses a rice phenotype visible light image-thermal infrared image matching system, which is used to implement the rice phenotype visible light image-thermal infrared image matching method described in Example 1. Figure 2 As shown, it includes: An image acquisition unit, used for acquiring a plurality of visible light images of rice and their corresponding thermal infrared images; An image mask processing unit, used for preprocessing the visible light image and the thermal infrared image, generating mask areas and non-mask areas of the visible light image and the thermal infrared image; wherein the mask areas and non-mask areas of the visible light image and the thermal infrared image are the same in size; The image feature extraction unit uses a cross-correlation two-stream network model to extract feature points and their corresponding descriptors in the non-mask area, and obtains matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, and connects the matching feature point pairs to obtain connecting lines; A connection line screening unit, used to screen out connection lines that simultaneously meet a connection line length threshold interval and a connection line angle threshold interval; The registration unit is used to generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the selected connecting lines. The feature points of the visible light image and the thermal infrared image are mapped to the fitting points on the visible light image through the homography matrix to generate residuals to screen the feature points of the visible light image close to the correct position, wherein the fitting points with larger residuals are most likely to be points close to the correct position. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting lines are the final matching results.

[0028] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A visible light image-thermal infrared image matching method for rice phenotype, characterized in that: The following steps are involved: S1, obtaining several visible light images of rice and their corresponding thermal infrared images; S2, by binarization and threshold matching, the blurred area A1 of the visible light image and the blurred area A2 of the thermal infrared image are respectively screened out, the two blurred areas A1 and A2 are blackened and then added to obtain area A as the mask area of ​​the visible light image and the thermal infrared image, and the areas of the visible light image and the thermal infrared image that are not blackened are the non-mask areas; S3, using a cross-correlation two-stream network model to extract feature points of the non-mask area and their corresponding descriptors, and obtaining matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, connecting the matching feature point pairs to obtain connecting lines; S4, screening out connection lines that meet both the connection line length threshold interval and the connection line angle threshold interval; S5. Generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the connecting lines obtained in step S4. Generate residuals based on the feature points of the visible light image and the thermal infrared image mapped to the fitting points on the visible light image through the homography matrix to screen the feature points of the visible light image that are close to the correct position. The fitting points with larger residuals are most likely to be points close to the correct position. At this time, connect the feature points on the visible light image with the corresponding feature points on the thermal infrared image, and the connecting lines obtained are the final matching results.

2. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: The step S1 comprises the following steps: S101, on the UAV ground station, setting a designated rice field as an aerial photography area, setting a heading overlap rate and a lateral overlap rate of the UAV aerial photography, and the UAV ground station automatically generates a route according to the operation information; S102, the UAV autonomously flies according to the planned route, and automatically collects visible light images and thermal infrared images of rice according to a set overlap rate; S103, the drone uploads the collected visible light image and thermal infrared image remote sensing data of rice to the SD card storage chip of the gimbal in real time; wherein the visible light image and its corresponding thermal infrared image are uploaded in pairs.

3. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: The step S2 comprises the following steps: S201, preprocessing of visible light images and thermal infrared images, including the following steps: S2011, converting the formats of the visible light image and the thermal infrared image into grayscale, the vertical resolution of the visible light image is adjusted to be the same as that of the thermal infrared image, and the horizontal resolution is adjusted according to the original image ratio; S2012, using contrast enhancement to process the visible light image and the thermal infrared image after adjusting the proportions, so that the visible light image and the thermal infrared image have clearer contours and edges, thereby making it easier to detect and locate key points; wherein the contrast enhancement method is as follows: ； In the formula, Input represents the input visible light image or thermal infrared image, and Output represents the output enhanced visible light image or thermal infrared image. and Respectively represent the minimum and maximum pixel values ​​of the input image, minmaxscale is a scaling factor used to control the degree of contrast adjustment; S202, binarize and threshold match the preprocessed visible light image and thermal infrared image, set the pixel points greater than the set value H to 256, and set the pixel points less than or equal to the set value H to 0, count the pixel average value of each binarized area based on the sliding window, the pixel size of each area is 10 in length and width, and the moving step is 2 pixels, set the area contour threshold M, compare with the area contour average value, and take the area less than the area contour threshold M as the blurred area, set the blurred area A1 of the visible light image and the blurred area A1 of the thermal infrared image to black, and then add them to obtain area A, A is the mask area of ​​the visible light image and the thermal infrared image, and the area of ​​the visible light image and the thermal infrared image that is not set to black is the non-mask area; wherein, the mask area and the non-mask area of ​​the visible light image and the thermal infrared image are the same size.

4. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: In step S3, the cross-correlated dual-stream network model includes two symmetrical branches, each of which contains 9 network layers, including 3 convolutional layers, 3 rectified linear unit ReLU functions, 2 maximum pooling layers and 1 average pooling layer; wherein, the first network layer is a convolutional layer, using a convolution kernel with a length of 3, a width of 3, and a step size of 2. The input of the network layer is 3×w×h image data, and the output is c1×w / 2×h / 2 feature data, where c1 is the number of convolution kernel channels, w is the image width, and h is the image height; the second network layer is a rectified linear unit ReLU function; the third network layer is a convolutional layer with a length of 3, a width of 3, and a step size of 2. The first layer is a maximum pooling layer with a stride of 2. The input of this network layer is the feature data of c1×w / 2×h / 2, and the output is the feature map C1 of c1×w / 4×h / 4; the fourth network layer is a convolution layer, which uses a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C1 of c1×w / 4×h / 4, and the output is the feature data of c2×w / 8×h / 8, where c2 is the number of convolution kernel channels; the fifth network layer is a rectified linear unit ReLU function; the sixth network layer is a maximum pooling layer with a stride of 2. The input of this network layer is the feature map C1, and the output is c2×w / 8×h / 8 feature map C2; the 7th network layer is a convolution layer, using a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C2 of c2×w / 8×h / 8, and the output is the feature data of c3×w / 16×h / 16, where c3 is the number of convolution kernel channels; the 8th network layer is a rectified linear unit ReLU function; the 9th network layer is an average pooling layer with a stride of 2. The input of this network layer is the feature map C2, and the output is the feature map C3 of c3×w / 16×h / 16; in order to obtain the relevant information between the features of the visible light image and the features of the thermal infrared image , the feature maps C3 obtained by the visible light image and the thermal infrared image through their respective branches are compressed into one-dimensional data of 1×(2×c3×w / 16×h / 16), and weighted summation is performed based on the cross-correlation of the one-dimensional data, and then converted into 2×(w / 16×h / 16) feature points and c3×(w / 16×h / 16) descriptors; wherein, the inputs at both ends of the cross-correlation two-stream network model are visible light images and thermal infrared images of the same size and range, respectively. After the visible light image and the thermal infrared image pass through the cross-correlation two-stream network model, the feature points of their respective non-mask areas and their corresponding descriptors are obtained.

5. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: In step S3, a nearest neighbor search algorithm is used according to the obtained feature points and descriptors to search for matching feature point pairs on the visible light image and the thermal infrared image, wherein feature points with similar descriptors are matching feature point pairs, wherein the descriptors describe feature vectors of the feature points, and the similarity of the descriptors is measured by cosine similarity, which is the angle between the feature vectors of two feature points. The closer the cosine similarity value is to 1, the more similar the descriptors of the two feature points are. When matching, firstly, a feature point is randomly selected from all the feature points in the thermal infrared image as a reference feature point, and then a target feature point matching the reference feature point and a feature point closest to the target feature point are found from all the feature points in the visible light image as target neighboring feature points. Points, the matching method is measured by distance. The distance between the reference feature point and the target feature point is called the first distance, and the distance between the reference feature point and the target adjacent feature point is called the second distance. All matching feature point pairs on the visible light image and the thermal infrared image are traversed, and the first distance average avg1 and the second distance average avg2 of all matching feature point pairs are calculated to obtain the difference avg=avg1-avg2; then all matching feature point pairs are screened, and the screening condition for each pair of feature point pairs is: first distance < (second distance-avg). If the condition of first distance < (second distance-avg) is met, the corresponding feature point pair is retained. After the screening is completed, all the matching feature point pairs retained are connected to obtain connecting lines.

6. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: In step S4, the lengths and angles of all connecting lines are calculated, threshold intervals are set for the lengths and angles of the connecting lines, and connecting lines that meet both the connecting line length threshold interval and the connecting line angle threshold interval are screened out; wherein, L is defined as the longest connecting line length minus the shortest connecting line length, the length threshold interval is the minimum connecting line length among all connecting lines plus 1 / 3L to the longest connecting line length among all connecting lines minus 1 / 3L, D is defined as the maximum absolute value of the acute angles formed by all connecting lines with the horizontal line minus the minimum absolute value of the acute angles formed by all connecting lines with the horizontal line, and the angle threshold interval is the minimum angle among all connecting lines plus 1 / 3D to the maximum angle among all connecting lines minus 1 / 3D.

7. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: The step S5 comprises the following steps: S501, generating a homography matrix according to the feature point coordinate matrix of the connecting line on the visible light image and the feature point coordinate matrix on the thermal infrared image obtained in step S4, wherein the homography matrix describes the mapping relationship between the feature points of the visible light image and the feature points of the thermal infrared image; S502, filtering feature points of the visible light image close to the correct position by generating residuals, the residuals are expressed as: ； In the formula, Represents the feature points of the visible light image, Represents the feature points of thermal infrared images After being mapped to the fitting points on the visible light image through the homography matrix, represents the residual, i represents the i-th correct feature point pair, is the number of correct feature point pairs; among them, the fitting point with the larger residual is most likely to be the point close to the correct position. The thermal infrared image is used as the reference image, and the feature points of the thermal infrared image do not need to be transformed; S503, the residual Sort in descending order, find the first n feature points corresponding to the first n residuals in the visible light image, and replace the first n feature points with the corresponding n fitting points on the visible light image obtained after the homography matrix transformation; S504, repeat steps S501 to S503 for updating until the sum of the residuals is equal to zero. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting line is the final matching result.

8. A rice phenotype visible light image-thermal infrared image matching system, characterized in that: A visible light image-thermal infrared image matching method for realizing the rice phenotype according to any one of claims 1 to 7, comprising: An image acquisition unit, used for acquiring a plurality of visible light images of rice and their corresponding thermal infrared images; An image mask processing unit, used for preprocessing the visible light image and the thermal infrared image, generating mask areas and non-mask areas of the visible light image and the thermal infrared image; wherein the mask areas and non-mask areas of the visible light image and the thermal infrared image are the same in size; The image feature extraction unit uses a cross-correlation two-stream network model to extract feature points and their corresponding descriptors in the non-mask area, and obtains matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, and connects the matching feature point pairs to obtain connecting lines; A connection line screening unit, used to screen out connection lines that simultaneously meet a connection line length threshold interval and a connection line angle threshold interval; A registration unit is used to generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the selected connecting lines. The feature points of the visible light image and the thermal infrared image are mapped to the fitting points on the visible light image through the homography matrix to generate residuals to screen the feature points of the visible light image close to the correct position, wherein the fitting points with larger residuals are most likely to be points close to the correct position. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting lines are the final matching results.

Images