A Visible Light Image-Thermal Infrared Image Matching Method and System for Rice Phenotypes

Abstract

The present invention discloses a method and system for matching visible light images and thermal infrared images of rice phenotypes, including: obtaining a plurality of visible light images and thermal infrared images of rice; determining the blurred regions and non-masked regions of the visible light images and thermal infrared images; extracting feature points and descriptors of the non-masked regions, and obtaining pairs of matching feature points on the visible light images and thermal infrared images, and connecting the pairs of matching feature points to obtain connecting lines; screening out the connecting lines that simultaneously satisfy the threshold intervals of the lengths and angles of the connecting lines; screening the feature points on the visible light images that are close to the correct positions by generating residuals from the fitting points mapped onto the visible light images through a homography matrix according to the feature points of the visible light images and thermal infrared images, and finally connecting the feature points on the visible light images with the corresponding feature points on the thermal infrared images to obtain the connecting lines, which are the final matching results. The present invention can avoid the problem that insufficient illumination causes some thermal infrared images to be blurred, thereby resulting in inaccurate matching.

Description

Technical Field

[0001] The present invention relates to the technical field of rice phenotype image registration, and in particular to a method and system for matching visible light images and thermal infrared images of rice phenotypes. Background Art

[0002] Accurately obtaining rice phenotype images can not only improve agricultural production efficiency, but also effectively ensure food security and quality. In these tasks, unmanned aerial vehicles (UAVs) have become one of the main methods for obtaining rice phenotype data due to their high efficiency, accuracy, and wide coverage. In UAV remote sensing monitoring, usually, the visible light images and thermal infrared images of the UAV for the same field are registered first, and then image analysis is performed to obtain agricultural information. Image registration can improve the accuracy of disease identification and rice constant inversion, and at the same time enhance image information, facilitating tracking and monitoring. It has broad application prospects and practical value. However, due to differences in resolution, spectrum, and viewing angle, as well as the complexity of field rice phenotype images, current registration methods often have difficulty successfully registering visible light images and thermal infrared images of paddy fields, and there is an urgent need to solve this problem. Summary of the Invention

[0003] The first object of the present invention is to overcome the deficiencies of the prior art and provide a method for matching visible light images and thermal infrared images of rice phenotypes, which can efficiently complete the precise alignment of visible light images and thermal infrared images of rice, enabling the rice images to effectively complete subsequent image stitching operations.

[0004] The second object of the present invention is to provide a system for matching visible light images and thermal infrared images of rice phenotypes.

[0005] The first object of the present invention is achieved by the following technical solutions: A method for matching visible light images and thermal infrared images of rice phenotypes, comprising the following steps:

[0006] S1. Obtain a plurality of visible light images of rice and their corresponding thermal infrared images;

[0007] S2. Respectively screen out the blurred region A1 of the visible light image and the blurred region A2 of the thermal infrared image through binarization and threshold matching, add the two blurred regions A1 and A2 after turning them black to obtain region A as the mask region of the visible light image and the thermal infrared image, and the non-black regions of the visible light image and the thermal infrared image are non-mask regions;

[0008] S3. Use a cross-correlation dual-stream network model to extract feature points and their corresponding descriptors in the non-mask region, obtain pairs of matching feature points on the visible light image and the thermal infrared image according to the feature points and descriptors, and connect the pairs of matching feature points to obtain connection lines;

[0009] S4. Screen out the connection lines that simultaneously meet the connection line length threshold range and the connection line angle threshold range;

[0010] S5. Generate a homography matrix based on the positions of all feature points of the connection lines obtained in step S4 on the visible light image and the positions of all feature points on the thermal infrared image. Screen the feature points on the visible light image that are close to the correct position by generating residuals from the fitting points mapped to the visible light image through the homography matrix according to the feature points of the visible light image and the thermal infrared image. Among them, the fitting points with larger residuals are most likely to be the points close to the correct position. At this time, connect the feature points on the visible light image with the corresponding feature points on the thermal infrared image, and the obtained connection line is the final matching result.

[0011] Further, step S1 includes the following steps:

[0012] S101. On the UAV ground station, set a designated paddy field as the aerial photography area, set the heading overlap rate and side overlap rate of the UAV aerial photography, and the UAV ground station automatically generates a flight path according to the operation information;

[0013] S102. The UAV autonomously flies according to the planned flight path and automatically collects visible light images and thermal infrared images of the rice according to the set overlap rate;

[0014] S103. The UAV uploads the remotely sensed data of the visible light images and thermal infrared images of the collected rice to the cloud platform SD card storage chip in real time; among them, the visible light images and their corresponding thermal infrared images are uploaded in pairs.

[0015] Further, step S2 includes the following steps:

[0016] S201. Preprocessing of the visible light image and the thermal infrared image, including the following steps:

[0017] S2011. Convert the formats of the visible light image and the thermal infrared image to grayscale, adjust the vertical resolution of the visible light image to be the same as that of the thermal infrared image, and adjust the horizontal resolution according to the original image ratio;

[0018] S2012. Process the adjusted visible light image and thermal infrared image by contrast enhancement to make the visible light image and the thermal infrared image have clearer contours and edges, so as to facilitate the detection and positioning of key points; among them, the contrast enhancement method is as follows:

[0019] ;

[0020] In the formula, Input represents the input visible light image or thermal infrared image, and Output represents the enhanced visible light image or thermal infrared image output. and respectively represent the minimum and maximum pixel values of the input image, and minmaxscale is a scaling factor used to control the degree of contrast adjustment;

[0021] S202. Binarize and perform threshold matching on the preprocessed visible light image and thermal infrared image. For binarization, pixels greater than the set value H are set to 256, and pixels less than or equal to the set value H are set to 0. Based on a sliding window, statistically calculate the average pixel value of each binarized region. The pixel size of each region, with both length and width being 10, and the moving step size being 2 pixels. Set the region contour threshold M, compare it with the average region contour value, and regard the region smaller than the region contour threshold M as a blurred region. Add the blurred regions A1 of the visible light image and the blurred regions A1 of the thermal infrared image after blackening them, to obtain region A, which is the mask region of the visible light image and the thermal infrared image. The non-blackened regions of the visible light image and the thermal infrared image are non-mask regions; among them, the mask regions and non-mask regions of the visible light image and the thermal infrared image are of the same size.

[0022] Further, in step S3, the cross-correlation two-stream network model includes two symmetric branches, each branch containing 9 network layers, which are respectively 3 convolutional layers, 3 rectified linear unit (ReLU) functions, 2 max pooling layers, and 1 average pooling layer. Among them, the first network layer is a convolutional layer, using a convolutional kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is image data of 3×w×h, and the output is feature data of c1×w / 2×h / 2, where c1 is the number of channels of the convolutional kernel, w is the width of the image, and h is the height of the image. The second network layer is a rectified linear unit (ReLU) function. The third network layer is a max pooling layer with a stride of 2. The input of this network layer is feature data of c1×w / 2×h / 2, and the output is a feature map C1 of c1×w / 4×h / 4. The fourth network layer is a convolutional layer, using a convolutional kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C1 of c1×w / 4×h / 4, and the output is feature data of c2×w / 8×h / 8, where c2 is the number of channels of the convolutional kernel. The fifth network layer is a rectified linear unit (ReLU) function. The sixth network layer is a max pooling layer with a stride of 2. The input of this network layer is the feature map C1, and the output is a feature map C2 of c2×w / 8×h / 8. The seventh network layer is a convolutional layer, using a convolutional kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C2 of c2×w / 8×h / 8, and the output is feature data of c3×w / 16×h / 16, where c3 is the number of channels of the convolutional kernel. The eighth network layer is a rectified linear unit (ReLU) function. The ninth network layer is an average pooling layer with a stride of 2. The input of this network layer is the feature map C2, and the output is a feature map C3 of c3×w / 16×h / 16. To obtain the correlation information between the features of the visible light image and the thermal infrared image, the feature maps C3 obtained from the visible light image and the thermal infrared image through their respective branches are compressed into one-dimensional data of 1×(2×c3×w / 16×h / 16). Weighted summation is performed based on the cross-correlation of the one-dimensional data, and then it is transformed into feature points of 2×(w / 16×h / 16) and descriptors of c3×(w / 16×h / 16). Among them, the two ends of the cross-correlation two-stream network model are respectively input with visible light images and thermal infrared images of the same size and range. After passing through the cross-correlation two-stream network model, the visible light image and the thermal infrared image obtain the feature points of their respective non-masked regions and their corresponding descriptors.

[0023] Further, in step S3, according to the obtained feature points and descriptors, the nearest neighbor search algorithm is used to search for the pairs of matching feature points on the visible light image and the thermal infrared image. The feature points with similar descriptors are the pairs of matching feature points. The descriptor describes the feature vector of the feature point, and the similarity of the descriptors is measured by cosine similarity. The closer the cosine similarity value is to 1 for the included angle between the feature vectors of two feature points, the more similar the descriptors of the two feature points are. When matching, first, a feature point is randomly selected from all the feature points in the thermal infrared image as the reference feature point, and then the target feature point that matches the reference feature point and the feature point closest to the target feature point are found among all the feature points in the visible light image as the target neighboring feature point. The matching method is measured by distance. The distance between the reference feature point and the target feature point is called the first distance, and the distance between the reference feature point and the target neighboring feature point is called the second distance. All the pairs of matching feature points on the visible light image and the thermal infrared image are traversed, and the average value avg1 of the first distances and the average value avg2 of the second distances of all the pairs of matching feature points are calculated to obtain the difference avg = avg1 - avg2. Then, all the pairs of matching feature points are screened. The screening condition for each pair of feature points is: the first distance < (the second distance - avg). If the condition of the first distance < (the second distance - avg) is satisfied, the corresponding pair of feature points is retained. After screening, all the retained pairs of matching feature points are connected to obtain the connecting lines.

[0024] Further, in step S4, the lengths and angles of all the connecting lines are calculated, and a threshold interval is set for the lengths and angles of the connecting lines. The connecting lines that simultaneously satisfy the length threshold interval and the angle threshold interval of the connecting lines are screened out. Among them, L is defined as the difference between the longest connecting line length and the shortest connecting line length, and the length threshold interval is from the minimum connecting line length among all the connecting lines plus 1 / 3L to the longest connecting line length among all the connecting lines minus 1 / 3L. D is defined as the difference between the maximum absolute value of the acute angle formed by all the connecting lines and the horizontal line and the minimum absolute value of the acute angle formed by all the connecting lines and the horizontal line, and the angle threshold interval is from the minimum angle among all the connecting lines plus 1 / 3D to the maximum angle among all the connecting lines minus 1 / 3D.

[0025] Further, step S5 includes the following steps:

[0026] S501. According to the coordinate matrices of the feature points on the visible light image and the coordinate matrices of the feature points on the thermal infrared image obtained in step S4, a homography matrix is generated. Among them, the homography matrix describes the mapping relationship between the feature points on the visible light image and the feature points on the thermal infrared image.

[0027] S502. Residuals are generated to screen the feature points on the visible light image that are close to the correct positions. The residual is expressed as:

[0028] ;

[0029] In the formula, represents the feature points of the visible light image, represents the feature points of the thermal infrared image is the fitting point mapped onto the visible light image through the homography matrix, represents the residual, i represents the i-th correct feature point pair, is the number of correct feature point pairs; among them, the fitting point with a larger residual is most likely to be the point close to the correct position. Taking the thermal infrared image as the reference image, the feature points of the thermal infrared image do not need to be transformed;

[0030] S503. Sort the obtained residuals in descending order, find the top n feature points corresponding to the top n residuals in the visible light image, and replace the top n feature points with the corresponding n fitting points on the visible light image obtained through the homography matrix transformation;

[0031] S504. Repeat steps S501 - S503 for updating until the sum of the residuals is equal to zero. At this time, connect the feature points on the visible light image and the corresponding feature points on the thermal infrared image, and the obtained connection line is the final matching result.

[0032] The second object of the present invention is achieved by the following technical solution: A visible light image - thermal infrared image matching system for rice phenotypes, used to implement the above - mentioned visible light image - thermal infrared image matching method for rice phenotypes, which includes:

[0033] An image acquisition unit, used to acquire a plurality of visible light images of rice and their corresponding thermal infrared images;

[0034] An image masking processing unit, used for pre - processing the visible light image and the thermal infrared image, generating the masked area and non - masked area of the visible light image and the thermal infrared image; wherein, the sizes of the masked area and non - masked area of the visible light image and the thermal infrared image are the same;

[0035] An image feature extraction unit, which uses a cross - correlation two - stream network model to extract the feature points and their corresponding descriptors of the non - masked area, and obtains the matching feature point pairs on the visible light image and the thermal infrared image according to the feature points and descriptors, and connects the matching feature point pairs to obtain a connection line;

[0036] A connection line screening unit, used to screen out the connection lines that simultaneously meet the connection line length threshold range and the connection line angle threshold range;

[0037] A registration unit is used to generate a homography matrix based on the positions of all feature points of the selected connection lines on the visible light image and the positions of all feature points on the thermal infrared image. Residuals are generated based on the feature points of the visible light image and the fitted points mapped onto the visible light image through the homography matrix from the feature points of the thermal infrared image to screen the feature points on the visible light image that are close to the correct position. Among them, the fitted points with larger residuals are most likely to be the points close to the correct position. At this time, the feature points on the visible light image are connected to the corresponding feature points on the thermal infrared image, and the obtained connection lines are the final matching results.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] 1. The present invention can effectively achieve the registration of visible light images and thermal infrared images of rice drones.

[0040] 2. The present invention sets a threshold range for the length and angle of the connection lines, and screens out the connection lines that simultaneously meet the threshold range of the connection line length and the threshold range of the connection line angle, improving the ability to obtain high-quality connection lines.

[0041] 3. The present invention uses a cross-correlation two-stream network model to obtain feature points and descriptors in the non-masked area of the image, which improves the registration speed while maintaining good registration accuracy.

[0042] 4. The present invention screens the feature points on the visible light image that are close to the correct position by generating residuals based on the feature points of the visible light image and the fitted points mapped onto the visible light image through the homography matrix from the feature points of the thermal infrared image, reducing the position deviation of the feature points.

[0043] In summary, the present invention has a low computational complexity, is easy to implement, can effectively avoid the problem that part of the thermal infrared image is blurred due to insufficient light, resulting in inaccurate matching, and is a competitive method and system for the registration of visible light and thermal infrared images of rice drones. It can make important contributions to the accurate monitoring of rice and the evaluation of its health and growth status, and is worthy of promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of the method of the present invention.

[0045] Figure 2 is an architecture diagram of the system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following further describes the present invention in detail in conjunction with the embodiments and the drawings, but the embodiments of the present invention are not limited thereto.

[0047] Embodiment 1

[0048] As Figure 1As shown in the figure, this embodiment discloses a method for matching visible light images and thermal infrared images of rice phenotypes, which includes the following steps:

[0049] S1. Obtain a number of visible light images of rice and their corresponding thermal infrared images, specifically as follows:

[0050] S101. On the UAV ground station, set a designated rice field as the aerial photography area, and set the heading overlap rate and side overlap rate of UAV aerial photography. The UAV ground station automatically generates a flight path according to the operation information;

[0051] S102. The UAV autonomously flies according to the planned flight path and automatically collects visible light images and thermal infrared images of rice according to the set overlap rate;

[0052] S103. The UAV uploads the remotely sensed data of the collected visible light images and thermal infrared images of rice to the SD card storage chip of the cloud platform in real time; among them, the visible light images and their corresponding thermal infrared images are uploaded in pairs.

[0053] S2. Respectively screen out the blurred area A1 of the visible light image and the blurred area A2 of the thermal infrared image through binarization and threshold matching, add the two blurred areas A1 and A2 after turning them black to obtain area A as the mask area of the visible light image and the thermal infrared image, and the non-blackened areas of the visible light image and the thermal infrared image are non-mask areas, specifically as follows:

[0054] S201. Preprocessing of visible light images and thermal infrared images, including the following steps:

[0055] S2011. Convert the formats of the visible light image and the thermal infrared image to grayscale, adjust the vertical resolution of the visible light image to be the same as that of the thermal infrared image, and adjust the horizontal resolution according to the proportion of the original image;

[0056] S2012. Use contrast enhancement to process the adjusted visible light image and thermal infrared image, so that the visible light image and the thermal infrared image have clearer contours and edges, making it more convenient to detect and locate key points; among them, the contrast enhancement method is as follows:

[0057] ;

[0058] In the formula, Input represents the input visible light image or thermal infrared image, Output represents the enhanced visible light image or thermal infrared image after output, and respectively represent the minimum and maximum pixel values of the input image, and minmaxscale is a scaling factor used to control the degree of contrast adjustment;

[0059] S202. Binarize and perform threshold matching on the preprocessed visible light image and thermal infrared image. For binarization, set the pixel points greater than the set value H to 256, and set the pixel points less than or equal to the set value H to 0. Based on a sliding window, statistically calculate the pixel average value of each binarized region. The pixel size of each region, both the length and width, is 10, and the moving step size is 2 pixels. Set the region contour threshold M and compare it with the average value of the region contour. Consider the regions with values less than the region contour threshold M as blurred regions. Add the blurred regions A1 of the visible light image and the blurred regions A1 of the thermal infrared image after blackening them to obtain region A. Region A is the mask region of the visible light image and the thermal infrared image, and the non-blackened regions of the visible light image and the thermal infrared image are non-mask regions; among them, the mask regions and non-mask regions of the visible light image and the thermal infrared image are of the same size.

[0060] S3. Use the cross-correlation two-stream network model to extract the feature points and their corresponding descriptors of the non-mask regions, and obtain the pairs of matching feature points on the visible light image and the thermal infrared image according to the feature points and descriptors. Connect the pairs of matching feature points to obtain connection lines, specifically as follows:

[0061] The cross-correlation two-stream network model includes two symmetric branches, each branch containing 9 network layers, which are respectively 3 convolutional layers, 3 rectified linear unit (ReLU) functions, 2 max pooling layers, and 1 average pooling layer; among them, the first network layer is a convolutional layer, using a convolutional kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is image data of 3×w×h, and the output is feature data of c1×w / 2×h / 2, where c1 is the number of channels of the convolutional kernel, w is the width of the image, and h is the height of the image; the second network layer is a rectified linear unit (ReLU) function; the third network layer is a max pooling layer with a stride of 2. The input of this network layer is feature data of c1×w / 2×h / 2, and the output is a feature map C1 of c1×w / 4×h / 4; the fourth network layer is a convolutional layer, using a convolutional kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C1 of c1×w / 4×h / 4, and the output is feature data of c2×w / 8×h / 8, where c2 is the number of channels of the convolutional kernel; the fifth network layer is a rectified linear unit (ReLU) function; the sixth network layer is a max pooling layer with a stride of 2. The input of this network layer is the feature map C1, and the output is a feature map C2 of c2×w / 8×h / 8; the seventh network layer is a convolutional layer, using a convolutional kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C2 of c2×w / 8×h / 8, and the output is feature data of c3×w / 16×h / 16, where c3 is the number of channels of the convolutional kernel; the eighth network layer is a rectified linear unit (ReLU) function; the ninth network layer is an average pooling layer with a stride of 2. The input of this network layer is the feature map C2, and the output is a feature map C3 of c3×w / 16×h / 16; to obtain the correlation information between the features of the visible light image and the features of the thermal infrared image, the feature maps C3 obtained by the visible light image and the thermal infrared image through their respective branches are compressed into one-dimensional data of 1×(2×c3×w / 16×h / 16), and weighted summation is performed based on the cross-correlation of the one-dimensional data, and then transformed into feature points of 2×(w / 16×h / 16) and descriptors of c3×(w / 16×h / 16); among them, the two ends of the cross-correlation two-stream network model are respectively input with visible light images and thermal infrared images of the same size and within the same range. After passing through the cross-correlation two-stream network model, the visible light image and the thermal infrared image obtain the feature points of their respective non-masked regions and their corresponding descriptors.

[0062] Using the nearest neighbor search algorithm based on the obtained feature points and descriptors, find the pairs of matching feature points on the visible light image and the thermal infrared image. Among them, the feature points with similar descriptors are the pairs of matching feature points. The descriptor describes the feature vector of the feature point, and the similarity of the descriptors is measured by cosine similarity. The smaller the angle between the feature vectors of two feature points and the closer the cosine similarity value is to 1, the more similar the descriptors of the two feature points are. When matching, first randomly select a feature point in all the feature points of the thermal infrared image as the reference feature point, and then find the target feature point that matches the reference feature point and the feature point closest to the target feature point in all the feature points of the visible light image as the target neighboring feature point. The matching method is measured by distance. The distance between the reference feature point and the target feature point is called the first distance, and the distance between the reference feature point and the target neighboring feature point is called the second distance. Traverse all the pairs of matching feature points on the visible light image and the thermal infrared image, calculate the average value avg1 of the first distances and the average value avg2 of the second distances of all the pairs of matching feature points, and obtain the difference avg = avg1 - avg2. Then, screen all the pairs of matching feature points. The screening condition for each pair of feature points is: the first distance < (the second distance - avg). If the condition of the first distance < (the second distance - avg) is satisfied, then retain the corresponding pair of feature points. After screening, connect all the retained pairs of matching feature points to obtain the connection lines.

[0063] S4. Screen out the connection lines that simultaneously satisfy the connection line length threshold interval and the connection line angle threshold interval, specifically as follows:

[0064] Calculate the lengths and angles of all the connection lines, set threshold intervals for the lengths and angles of the connection lines, and screen out the connection lines that simultaneously satisfy the connection line length threshold interval and the connection line angle threshold interval. Among them, define L as the difference between the longest connection line length and the shortest connection line length. The length threshold interval is from the minimum connection line length among all the connection lines plus 1 / 3L to the longest connection line length among all the connection lines minus 1 / 3L. Define D as the difference between the maximum absolute value of the acute angle formed by all the connection lines and the horizontal line and the minimum absolute value of the acute angle formed by all the connection lines and the horizontal line. The angle threshold interval is from the minimum angle among all the connection lines plus 1 / 3D to the maximum angle among all the connection lines minus 1 / 3D.

[0065] S5. Generate a homography matrix based on the positions of all the feature points of the connection lines obtained in step S4 on the visible light image and the positions of all the feature points on the thermal infrared image. Screen the feature points on the visible light image that are close to the correct position by generating residuals from the fitting points mapped onto the visible light image through the homography matrix based on the feature points of the visible light image and the thermal infrared image. Among them, the fitting points with larger residuals are most likely to be the points close to the correct position. At this time, connect the feature points on the visible light image with the corresponding feature points on the thermal infrared image, and the obtained connection line is the final matching result. Specifically as follows:

[0066] S501. Generate a homography matrix based on the feature point coordinate matrices of the connection lines obtained in step S4 on the visible light image and the thermal infrared image respectively. Among them, the homography matrix describes the mapping relationship between the feature points of the visible light image and the feature points of the thermal infrared image;

[0067] S502. Screen the feature points on the visible light image that are close to the correct position by generating residuals. The residual is expressed as:

[0068] ;

[0069] In the formula, represents the feature points of the visible light image, represents the feature points of the thermal infrared image The fitting points mapped onto the visible light image through the homography matrix, represents the residual, i represents the i-th pair of correct feature points, is the number of pairs of correct feature points; among them, the fitting points with larger residuals are most likely to be the points close to the correct position. The thermal infrared image is used as the reference image, and the feature points of the thermal infrared image do not need to be transformed;

[0070] S503. Sort the obtained residuals in descending order, find the top n feature points corresponding to the top n residuals in the visible light image, and replace the top n feature points with the corresponding n fitting points on the visible light image obtained after the homography matrix transformation;

[0071] S504. Repeat steps S501 - S503 for updating until the sum of the residuals is equal to zero. At this time, connect the feature points on the visible light image with the corresponding feature points on the thermal infrared image, and the obtained connection line is the final matching result.

[0072] Example 2

[0073] This example discloses a visible light image - thermal infrared image matching system for rice phenotypes, which is used to implement the visible light image - thermal infrared image matching method for rice phenotypes described in Example 1, as Figure 2As shown in the figure, it includes:

[0074] An image acquisition unit for acquiring a plurality of visible light images of rice and their corresponding thermal infrared images;

[0075] An image masking processing unit for preprocessing the visible light image and the thermal infrared image to generate a masked area and an unmasked area of the visible light image and the thermal infrared image; wherein, the sizes of the masked area and the unmasked area of the visible light image and the thermal infrared image are the same;

[0076] An image feature extraction unit that uses a cross-correlation two-stream network model to extract feature points and their corresponding descriptors in the unmasked area, and obtains pairs of matching feature points on the visible light image and the thermal infrared image according to the feature points and descriptors, connects the pairs of matching feature points, and obtains connection lines;

[0077] A connection line screening unit for screening out connection lines that simultaneously satisfy the connection line length threshold range and the connection line angle threshold range;

[0078] A registration unit for generating a homography matrix according to the positions of all feature points on the visible light image in the image and the positions of all feature points on the thermal infrared image in the image based on the screened connection lines, and screening out feature points on the visible light image that are close to the correct position by generating a residual according to the fitting points obtained by mapping the feature points of the visible light image and the thermal infrared image to the visible light image through the homography matrix. Among them, the fitting point with the larger residual is most likely to be the point close to the correct position. At this time, the feature points on the visible light image are connected to the corresponding feature points on the thermal infrared image, and the obtained connection line is the final matching result.

[0079] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A visible light image-thermal infrared image matching method for rice phenotype, characterized in that: The following steps are involved: S1, obtaining several visible light images of rice and their corresponding thermal infrared images; S2, by binarization and threshold matching, the blurred area A1 of the visible light image and the blurred area A2 of the thermal infrared image are respectively screened out, the two blurred areas A1 and A2 are blackened and then added to obtain area A as the mask area of ​​the visible light image and the thermal infrared image, and the areas of the visible light image and the thermal infrared image that are not blackened are the non-mask areas; S3, using a cross-correlation two-stream network model to extract feature points of the non-mask area and their corresponding descriptors, and obtaining matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, connecting the matching feature point pairs to obtain connecting lines; S4, screening out connection lines that meet both the connection line length threshold interval and the connection line angle threshold interval; S5. Generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the connecting lines obtained in step S4. Generate residuals based on the feature points of the visible light image and the thermal infrared image mapped to the fitting points on the visible light image through the homography matrix to screen the feature points of the visible light image that are close to the correct position. The fitting points with larger residuals are most likely to be points close to the correct position. At this time, connect the feature points on the visible light image with the corresponding feature points on the thermal infrared image, and the connecting lines obtained are the final matching results.

2. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: The step S1 comprises the following steps: S101, on the UAV ground station, setting a designated rice field as an aerial photography area, setting a heading overlap rate and a lateral overlap rate of the UAV aerial photography, and the UAV ground station automatically generates a route according to the operation information; S102, the UAV autonomously flies according to the planned route, and automatically collects visible light images and thermal infrared images of rice according to a set overlap rate; S103, the drone uploads the collected visible light image and thermal infrared image remote sensing data of rice to the SD card storage chip of the gimbal in real time; wherein the visible light image and its corresponding thermal infrared image are uploaded in pairs.

3. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: The step S2 comprises the following steps: S201, preprocessing of visible light images and thermal infrared images, including the following steps: S2011, converting the formats of the visible light image and the thermal infrared image into grayscale, the vertical resolution of the visible light image is adjusted to be the same as that of the thermal infrared image, and the horizontal resolution is adjusted according to the original image ratio; S2012, using contrast enhancement to process the visible light image and the thermal infrared image after adjusting the proportions, so that the visible light image and the thermal infrared image have clearer contours and edges, thereby making it easier to detect and locate key points; wherein the contrast enhancement method is as follows: ； In the formula, Input represents the input visible light image or thermal infrared image, and Output represents the output enhanced visible light image or thermal infrared image. and Respectively represent the minimum and maximum pixel values ​​of the input image, minmaxscale is a scaling factor used to control the degree of contrast adjustment; S202, binarize and threshold match the preprocessed visible light image and thermal infrared image, set the pixel points greater than the set value H to 256, and set the pixel points less than or equal to the set value H to 0, count the pixel average value of each binarized area based on the sliding window, the pixel size of each area is 10 in length and width, and the moving step is 2 pixels, set the area contour threshold M, compare with the area contour average value, and take the area less than the area contour threshold M as the blurred area, set the blurred area A1 of the visible light image and the blurred area A1 of the thermal infrared image to black, and then add them to obtain area A, A is the mask area of ​​the visible light image and the thermal infrared image, and the area of ​​the visible light image and the thermal infrared image that is not set to black is the non-mask area; wherein, the mask area and the non-mask area of ​​the visible light image and the thermal infrared image are the same size.

4. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: In step S3, the cross-correlated dual-stream network model includes two symmetrical branches, each of which contains 9 network layers, including 3 convolutional layers, 3 rectified linear unit ReLU functions, 2 maximum pooling layers and 1 average pooling layer; wherein, the first network layer is a convolutional layer, using a convolution kernel with a length of 3, a width of 3, and a step size of 2. The input of the network layer is 3×w×h image data, and the output is c1×w / 2×h / 2 feature data, where c1 is the number of convolution kernel channels, w is the image width, and h is the image height; the second network layer is a rectified linear unit ReLU function; the third network layer is a convolutional layer with a length of 3, a width of 3, and a step size of 2. The first layer is a maximum pooling layer with a stride of 2. The input of this network layer is the feature data of c1×w / 2×h / 2, and the output is the feature map C1 of c1×w / 4×h / 4; the fourth network layer is a convolution layer, which uses a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C1 of c1×w / 4×h / 4, and the output is the feature data of c2×w / 8×h / 8, where c2 is the number of convolution kernel channels; the fifth network layer is a rectified linear unit ReLU function; the sixth network layer is a maximum pooling layer with a stride of 2. The input of this network layer is the feature map C1, and the output is c2×w / 8×h / 8 feature map C2; the 7th network layer is a convolution layer, using a convolution kernel with a length of 3, a width of 3, and a stride of 2. The input of this network layer is the feature map C2 of c2×w / 8×h / 8, and the output is the feature data of c3×w / 16×h / 16, where c3 is the number of convolution kernel channels; the 8th network layer is a rectified linear unit ReLU function; the 9th network layer is an average pooling layer with a stride of 2. The input of this network layer is the feature map C2, and the output is the feature map C3 of c3×w / 16×h / 16; in order to obtain the relevant information between the features of the visible light image and the features of the thermal infrared image , the feature maps C3 obtained by the visible light image and the thermal infrared image through their respective branches are compressed into one-dimensional data of 1×(2×c3×w / 16×h / 16), and weighted summation is performed based on the cross-correlation of the one-dimensional data, and then converted into 2×(w / 16×h / 16) feature points and c3×(w / 16×h / 16) descriptors; wherein, the inputs at both ends of the cross-correlation two-stream network model are visible light images and thermal infrared images of the same size and range, respectively. After the visible light image and the thermal infrared image pass through the cross-correlation two-stream network model, the feature points of their respective non-mask areas and their corresponding descriptors are obtained.

5. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: In step S3, a nearest neighbor search algorithm is used according to the obtained feature points and descriptors to search for matching feature point pairs on the visible light image and the thermal infrared image, wherein feature points with similar descriptors are matching feature point pairs, wherein the descriptors describe feature vectors of the feature points, and the similarity of the descriptors is measured by cosine similarity, which is the angle between the feature vectors of two feature points. The closer the cosine similarity value is to 1, the more similar the descriptors of the two feature points are. When matching, firstly, a feature point is randomly selected from all the feature points in the thermal infrared image as a reference feature point, and then a target feature point matching the reference feature point and a feature point closest to the target feature point are found from all the feature points in the visible light image as target neighboring feature points. Points, the matching method is measured by distance. The distance between the reference feature point and the target feature point is called the first distance, and the distance between the reference feature point and the target adjacent feature point is called the second distance. All matching feature point pairs on the visible light image and the thermal infrared image are traversed, and the first distance average avg1 and the second distance average avg2 of all matching feature point pairs are calculated to obtain the difference avg=avg1-avg2; then all matching feature point pairs are screened, and the screening condition for each pair of feature point pairs is: first distance < (second distance-avg). If the condition of first distance < (second distance-avg) is met, the corresponding feature point pair is retained. After the screening is completed, all the matching feature point pairs retained are connected to obtain connecting lines.

6. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: In step S4, the lengths and angles of all connecting lines are calculated, threshold intervals are set for the lengths and angles of the connecting lines, and connecting lines that meet both the connecting line length threshold interval and the connecting line angle threshold interval are screened out; wherein, L is defined as the longest connecting line length minus the shortest connecting line length, the length threshold interval is the minimum connecting line length among all connecting lines plus 1 / 3L to the longest connecting line length among all connecting lines minus 1 / 3L, D is defined as the maximum absolute value of the acute angles formed by all connecting lines with the horizontal line minus the minimum absolute value of the acute angles formed by all connecting lines with the horizontal line, and the angle threshold interval is the minimum angle among all connecting lines plus 1 / 3D to the maximum angle among all connecting lines minus 1 / 3D.

7. The visible light image-thermal infrared image matching method of rice phenotype according to claim 1, characterized in that: The step S5 comprises the following steps: S501, generating a homography matrix according to the feature point coordinate matrix of the connecting line on the visible light image and the feature point coordinate matrix on the thermal infrared image obtained in step S4, wherein the homography matrix describes the mapping relationship between the feature points of the visible light image and the feature points of the thermal infrared image; S502, filtering feature points of the visible light image close to the correct position by generating residuals, the residuals are expressed as: ； In the formula, Represents the feature points of the visible light image, Represents the feature points of thermal infrared images After being mapped to the fitting points on the visible light image through the homography matrix, represents the residual, i represents the i-th correct feature point pair, is the number of correct feature point pairs; among them, the fitting point with the larger residual is most likely to be the point close to the correct position. The thermal infrared image is used as the reference image, and the feature points of the thermal infrared image do not need to be transformed; S503, the residual Sort in descending order, find the first n feature points corresponding to the first n residuals in the visible light image, and replace the first n feature points with the corresponding n fitting points on the visible light image obtained after the homography matrix transformation; S504, repeat steps S501 to S503 for updating until the sum of the residuals is equal to zero. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting line is the final matching result.

8. A rice phenotype visible light image-thermal infrared image matching system, characterized in that: A visible light image-thermal infrared image matching method for realizing the rice phenotype according to any one of claims 1 to 7, comprising: An image acquisition unit, used to acquire a plurality of visible light images of rice and their corresponding thermal infrared images; An image mask processing unit, used for preprocessing the visible light image and the thermal infrared image, generating mask areas and non-mask areas of the visible light image and the thermal infrared image; wherein the mask areas and non-mask areas of the visible light image and the thermal infrared image are the same in size; The image feature extraction unit uses a cross-correlation two-stream network model to extract feature points and their corresponding descriptors in the non-mask area, and obtains matching feature point pairs on the visible light image and the thermal infrared image based on the feature points and descriptors, and connects the matching feature point pairs to obtain connecting lines; A connection line screening unit, used to screen out connection lines that simultaneously meet a connection line length threshold interval and a connection line angle threshold interval; A registration unit is used to generate a homography matrix based on the positions of all feature points on the visible light image and the positions of all feature points on the thermal infrared image according to the selected connecting lines. The feature points of the visible light image and the thermal infrared image are mapped to the fitting points on the visible light image through the homography matrix to generate residuals to screen the feature points of the visible light image close to the correct position, wherein the fitting points with larger residuals are most likely to be points close to the correct position. At this time, the feature points on the visible light image are connected with the corresponding feature points on the thermal infrared image, and the obtained connecting lines are the final matching results.

Images