An image matching method, device, equipment and storage medium

Abstract

The present invention discloses an image matching method, device, equipment and storage medium, which are applied to the field of plant pest and disease analysis. An average segmentation image is obtained by performing image segmentation on a visible light image; the plant temperature is hierarchically divided based on an infrared image, and a temperature hierarchy list is constructed; a temperature hierarchy operator is constructed, and the upper and lower limits of the temperature thresholds of each temperature hierarchy in the temperature hierarchy list are sequentially input into the temperature hierarchy operator to obtain a homography matrix of the matching points between the infrared image and the average segmentation image at each temperature hierarchy; the infrared image is perspectively transformed based on the homography matrix in the order of the temperature hierarchy to obtain a transformed image; the transformed images at all temperature hierarchies are sequentially updated into the infrared image to obtain an image matching result. Using this operator, the infrared image is divided into multiple hierarchies from the temperature perspective, and the infrared image is gradually stretched and adjusted for matching, so that accurate matching of the image can be achieved without training samples, and the matching efficiency and accuracy of the image are improved.

Description

Technical Field

[0001] The present invention relates to the field of plant pest and disease analysis, and particularly to an image matching method, an image matching device, an electronic device, and a computer-readable storage medium. Background Art

[0002] The temperature of a specific area on a plant visible light image can be obtained through an infrared image. These temperature data can better reflect the growth status of plants, pest and diseases, and the growth trend of key tissues, which is very important for controlling the growth status of rice grains, accurately monitoring pest and diseases, and predicting yields. Due to certain differences in the resolution, angle, and distortion of infrared sensors and visible light sensors, there are large misalignments in position, direction, and size after shooting, and they need to be matched correctly before effective processing can be carried out. Therefore, it is very necessary to carry out infrared and visible light image matching work to improve the modern monitoring and management level of planting industry.

[0003] Currently, in the infrared and visible light image matching work, deep learning methods are generally used, and a neural network architecture is introduced to match images. Deep learning methods require a large number of samples for support. The texture of rice is complex, and it is difficult to correctly detect the significant feature points of plants. In the case of complex sample feature points, the deep learning algorithm cannot play a role; and the deep learning model may perform well on a specific data set, but its generalization ability may be limited when facing rice vegetation in new or different seasons, varieties, and lighting conditions, resulting in a significant decline in matching performance. Summary of the Invention

[0004] The purpose of the present invention is to provide an image matching method, device, equipment, and storage medium, which are applied to the field of plant pest and disease analysis. This method establishes a temperature level operator, uses this operator to divide the infrared image into multiple levels from the temperature perspective, and gradually stretches and adjusts the infrared image for matching, realizing accurate matching of images without training samples, and improving the matching efficiency and accuracy of images.

[0005] To solve the above technical problems, the present invention provides an image matching method, including:

[0006] Obtain a visible light image and an infrared image of a target plant planting area, and perform image segmentation on the visible light image based on a clustering algorithm to obtain a mean segmentation image;

[0007] Based on the infrared image, perform level division on the plant temperature to construct a temperature level list;

[0008] Construct a temperature level operator, and sequentially input the upper and lower limits of the temperature thresholds of each temperature level in the temperature level list into the temperature level operator to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at each temperature level;

[0009] Perform a perspective transformation on the infrared image based on the homography matrix in the order of the temperature levels to obtain a transformed image;

[0010] Update the transformed images at all the temperature levels to the infrared image in sequence to obtain an image matching result.

[0011] Optionally, the construction of the temperature level operator includes inputting the upper and lower threshold values of each level in the temperature level list into the temperature level operator in sequence to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at the current temperature level, including:

[0012] Construct the temperature level operator; the input of the temperature level operator is the lower temperature threshold and the upper temperature threshold, and the output is the homography matrix;

[0013] Input the lower temperature threshold and the upper temperature threshold of each level in the temperature level list into the temperature level operator in sequence;

[0014] Based on the temperature level operator, set the values of the pixels in the infrared image that are less than the lower temperature threshold and greater than the upper temperature threshold to the minimum infrared pixel value to obtain a to-be-calculated infrared image;

[0015] Based on the temperature level operator, call the SIFT algorithm to match the mean segmentation image and the to-be-calculated infrared image to obtain a first key point list of the visible light image and a second key point list of the to-be-calculated infrared image;

[0016] Based on the temperature level operator, call the brute-force matching algorithm to determine the matching point pairs in the first key point list and the second key point list, and store the matching point pairs in the temperature level operator matching point list;

[0017] Based on the temperature level operator, call the RANSAC algorithm to determine the homography matrix between the matching point pairs in the temperature level operator matching point list.

[0018] Optionally, the hierarchical division of the plant temperature based on the infrared image to construct a temperature level list includes:

[0019] Construct a temperature level list; each element of the temperature level list contains two fields, namely the upper temperature threshold and the lower temperature threshold;

[0020] Assign the current to-be-processed high temperature value to the maximum infrared pixel value of the infrared image;

[0021] Determine the standard deviation of all pixel values in the infrared image, and use the result of dividing the standard deviation by 4 as the temperature span value;

[0022] Use the result of subtracting the temperature span value from the current temperature high value to be processed as the temporary temperature lower limit value, and use the current temperature high value to be processed as the temporary temperature upper limit value;

[0023] Obtain all pixels in the infrared image whose pixel values are less than or equal to the temporary temperature lower limit value and greater than or equal to the temporary temperature upper limit value as the filtered pixels;

[0024] If the number of the filtered pixels is greater than the result of dividing the number of pixels in the infrared image by 4, then use the result of subtracting half of the temperature span value from the current temperature high value to be processed as the temporary temperature lower limit value;

[0025] Store the temporary temperature upper limit value and the temporary temperature lower limit value as an element in the temperature level list;

[0026] Assign the current temperature high value to be processed to the temporary temperature lower limit value;

[0027] If the current temperature high value to be processed is greater than the minimum infrared pixel value, then start executing again from the step of using the result of subtracting the temperature span value from the current temperature high value to be processed as the temporary temperature lower limit value.

[0028] Optionally, the image segmentation of the visible light image based on the clustering algorithm to obtain the mean segmentation image includes:

[0029] Obtain the maximum infrared pixel value, the minimum infrared pixel value, and the standard deviation of all pixel values in the infrared image;

[0030] Subtract the minimum infrared pixel value from the maximum infrared pixel value to obtain the infrared pixel value difference;

[0031] Perform a rounding operation on the result of dividing the infrared pixel value difference by the standard deviation and multiplying by 4 to obtain the number of segments;

[0032] Perform K-means clustering algorithm segmentation on the visible light image based on the number of segments to obtain the mean segmentation image.

[0033] To solve the above technical problems, the present invention provides an image matching device, including:

[0034] A first module, configured to obtain a visible light image and an infrared image of a target plant planting area, and perform image segmentation on the visible light image based on a clustering algorithm to obtain a mean segmentation image;

[0035] A second module, configured to hierarchically classify the plant temperature based on the infrared image to construct a temperature hierarchy list;

[0036] A third module, configured to construct a temperature hierarchy operator, and sequentially input the upper and lower limits of the temperature thresholds of each level in the temperature hierarchy list into the temperature hierarchy operator to obtain a homography matrix of the matching points between the infrared image and the mean segmentation image at the current temperature level;

[0037] A fourth module, configured to perform perspective transformation on the infrared image based on the homography matrix in the order of the temperature levels to obtain a transformed image;

[0038] A fifth module, configured to sequentially update the transformed images at all the temperature levels into the infrared image to obtain an image matching result.

[0039] Optionally, the third module includes:

[0040] A first unit, configured to construct the temperature hierarchy operator; the input of the temperature hierarchy operator is the lower limit of the temperature threshold and the upper limit of the temperature threshold, and the output is the homography matrix;

[0041] A second unit, configured to sequentially input the lower limit of the temperature threshold and the upper limit of the temperature threshold of each temperature level in the temperature hierarchy list into the temperature hierarchy operator;

[0042] A third unit, configured to set the values of the pixels in the infrared image that are less than the lower limit of the temperature threshold and greater than the upper limit of the temperature threshold to the minimum infrared pixel value based on the temperature hierarchy operator to obtain a to-be-calculated infrared image;

[0043] A fourth unit, configured to call the SIFT algorithm based on the temperature hierarchy operator to match the mean segmentation image and the to-be-calculated infrared image to obtain a first key point list of the visible light image and a second key point list of the to-be-calculated infrared image;

[0044] A fifth unit, configured to call a brute-force matching algorithm based on the temperature hierarchy operator to determine the matching point pairs in the first key point list and the second key point list, and store the matching point pairs into a temperature hierarchy operator matching point list;

[0045] A sixth unit, configured to call the RANSAC algorithm based on the temperature hierarchy operator to determine the homography matrix between the matching point pairs in the temperature hierarchy operator matching point list.

[0046] Optionally, the second module includes:

[0047] The seventh unit is used to construct a temperature level list; each element of the temperature level list contains two fields, namely the upper temperature threshold and the lower temperature threshold;

[0048] The eighth unit is used to assign the high value of the current temperature to be processed to the maximum infrared pixel value of the infrared image;

[0049] The ninth unit is used to determine the standard deviation of all pixel values in the infrared image, and use the result of dividing the standard deviation by 4 as the temperature span value;

[0050] The tenth unit is used to use the result of subtracting the temperature span value from the high value of the current temperature to be processed as the temporary lower temperature limit value, and use the high value of the current temperature to be processed as the temporary upper temperature limit value;

[0051] The eleventh unit is used to obtain all pixels in the infrared image whose pixel values are less than or equal to the temporary lower temperature limit value and greater than or equal to the temporary upper temperature limit value as the filtered pixels;

[0052] The twelfth unit is used to, if the number of the filtered pixels is greater than the result of dividing the number of pixels of the infrared image by 4, use the result of subtracting half of the temperature span value from the high value of the current temperature to be processed as the temporary lower temperature limit value;

[0053] The thirteenth unit is used to store the temporary upper temperature limit value and the temporary lower temperature limit value as an element in the temperature level list;

[0054] The fourteenth unit is used to assign the high value of the current temperature to be processed to the temporary lower temperature limit value;

[0055] The fifteenth unit is used to, if the high value of the current temperature to be processed is greater than the minimum infrared pixel value, start executing from the tenth unit again.

[0056] Optionally, the first module includes:

[0057] The sixteenth unit is used to obtain the maximum infrared pixel value, the minimum infrared pixel value, and the standard deviation of all pixel values in the infrared image;

[0058] The seventeenth unit is used to subtract the minimum infrared pixel value from the maximum infrared pixel value to obtain an infrared pixel value difference;

[0059] The eighteenth unit is used to perform a rounding operation on the result of dividing the infrared pixel value difference by the standard deviation and multiplying by 4 to obtain the number of segments;

[0060] The nineteenth unit is used to perform K-means clustering algorithm segmentation on the visible light image based on the number of segments to obtain the mean segmented image.

[0061] To solve the above technical problems, the present invention provides an electronic device, including:

[0062] A memory for storing a computer program;

[0063] A processor for implementing the above-mentioned image matching method when executing the computer program.

[0064] To solve the above technical problems, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the above-mentioned image matching method is implemented.

[0065] It can be seen that the method of the present invention obtains visible light images and infrared images of the target plant planting area, performs image segmentation on the visible light images based on a clustering algorithm to obtain a mean segmentation image; hierarchically divides the plant temperature based on the infrared images to construct a temperature hierarchy list; constructs a temperature hierarchy operator, and sequentially inputs the upper and lower limits of the temperature thresholds of each level in the temperature hierarchy list into the temperature hierarchy operator to obtain a homography matrix of the matching points between the infrared image and the mean segmentation image under the current temperature level; performs perspective transformation on the infrared image based on the homography matrix in the order of the temperature levels to obtain a transformed image; sequentially updates the transformed images of all temperature levels to the infrared image to obtain an image matching result.

[0066] The present invention provides an image matching method. By establishing a temperature hierarchy operator, the infrared image is divided into multiple levels from the temperature perspective using this operator, and the infrared image is gradually stretched and adjusted for matching, so that precise matching of the image can be achieved without training samples, improving the matching efficiency and accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0068] Figure 1 It is a flowchart of an image matching method provided by an embodiment of the present invention;

[0069] Figure 2 It is a structural block diagram of an image matching device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0071] The temperature of a specific area on the visible light image of a plant can be obtained through an infrared image, and these temperature data can better reflect the growth status of vegetation, pests and diseases, and the growth trend of key tissues, which is very important for controlling the growth status of grains, accurately monitoring crop pests and diseases, and predicting crop yields.

[0072] There are certain differences in the resolution, angle, and distortion between the infrared sensor and the visible light sensor. After shooting, there are large misalignments in position, direction, and size, and they need to be matched before effective processing can be carried out. Therefore, it is very necessary to carry out the matching work of infrared and visible light images to improve the level of agricultural modern monitoring and management.

[0073] The image matching method provided by the present invention can be the matching of the visible light image and the infrared image of a plant, where the plant can specifically refer to crops such as rice.

[0074] There are currently two main methods for automated infrared and visible light image matching: The first is the traditional feature-based matching method. First, specific algorithms (such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.) are used to detect significant feature points in the source image and the target image and describe the feature points. Then, similarity metrics (such as Euclidean distance, Hamming distance, etc.) are used to find the best match between the feature vectors of the source image and the target image to achieve image alignment. Such methods are easy to implement. However, a key problem with such methods is that the matching effect depends to a large extent on the "matching points" of the scene being matched. Since there are many very similar "matching points" on rice leaves and panicles (such as the tips of leaves), there may be serious problems such as uneven matching density and matching errors for these matching points. Therefore, it is difficult for such methods to handle the infrared and visible light image matching of rice well. The second is the deep learning method, which introduces the convolutional neural network (CNN, Convolutional Neural Networks) architecture to match images. Such methods utilize the powerful capabilities of CNN in image recognition and feature extraction and can obtain good matching results. However, on the one hand, all deep learning series methods require a large number of samples for support. The texture of plants is complex, and it is difficult to correctly detect the significant feature points of plants. In the case of complex sample feature points, the deep learning algorithm cannot play its role. On the other hand, the CNN model may perform well on specific datasets, but its generalization ability may be limited when facing plants in new or different seasons, varieties, and lighting conditions, resulting in a significant decline in matching performance.

[0075] Facing the problem of infrared and visible light image matching of plants with complex textures and key points, even experienced experts cannot find the correct matching points at once. Finding the correct matching points is a gradual and progressive process. Therefore, a method is needed that can gradually improve the matching accuracy from easily found matching points to finely searched matching points at each level.

[0076] In view of the problems existing in the prior art, the present invention provides an image matching method. This method constructs a temperature level operator, which can divide the plant infrared image into multiple levels from the temperature perspective and gradually stretch and adjust the infrared image for matching, forming a dedicated method for plant infrared and visible light image matching.

[0077] Using the method of the present invention, the matching of complex plant image contents of two different sensors can be transformed into a hierarchical step-by-step matching process. The method can first match the part of the rice ear with higher temperature and obvious directivity in the plant species, so that the two images are initially registered. Then, it gradually transitions to the relatively complex and relatively low-temperature leaves, and finally achieves accurate matching. Using the patent of the present invention helps to achieve accurate matching of plant infrared and visible light images, enables the farm monitoring program to more accurately find the relative temperatures of different components in the plant visible light image, and further obtains the growth status of plant grains and the pest and disease status, improving the efficiency of agricultural production and the scientific nature of decision-making.

[0078] The following combines Figure 1 , Figure 1 which is a flowchart of an image matching method provided by an embodiment of the present invention. The method may include:

[0079] S101: Obtain a visible light image and an infrared image of a target plant planting area, and perform image segmentation on the visible light image based on a clustering algorithm to obtain an average segmentation image.

[0080] In this embodiment, a visible light image and an infrared image of a target plant planting area can be obtained. The specific type of the plant is not limited in this embodiment, and it can generally be rice or the like.

[0081] Furthermore, in this embodiment, an average segmentation image can be obtained by performing image segmentation on the visible light image based on a clustering algorithm. The specific method for obtaining the average segmentation image is not limited in this embodiment. Generally, the maximum infrared pixel value, the minimum infrared pixel value, and the standard deviation of all pixel values in the infrared image can be obtained; the difference between the maximum infrared pixel value and the minimum infrared pixel value is obtained to get the infrared pixel value difference; the result of dividing the infrared pixel value difference by the standard deviation and multiplying by 4 is rounded to an integer operation to obtain the number of segments; the K-means clustering algorithm is used to segment the visible light image based on the number of segments to obtain an average segmentation image.

[0082] Specifically, in this embodiment, a visible light image KJImage and an infrared image HWImage of a target plant planting area (such as a rice paddy area) can be obtained; KJImage comes from a visible light sensor and is a color image composed of three bands of red, green, and blue; HWImage comes from an infrared sensor and is a black-and-white image composed of one band. Each pixel in the infrared image represents a relative temperature value, and the higher the value, the higher the temperature.

[0083] Furthermore, the maximum pixel value in the infrared image, that is, the maximum infrared pixel value HWImageMax, can be obtained;

[0084] The minimum pixel value in the infrared image, that is, the minimum infrared pixel value HWImageMin, can be obtained;

[0085] Obtain the number of pixels HWNumber of the infrared image;

[0086] Obtain the standard deviation HWStd of all pixel values in the infrared image;

[0087] Subtract the minimum infrared pixel value from the maximum infrared pixel value to obtain the infrared pixel value difference; Round the result of dividing the infrared pixel value difference by the standard deviation and multiplying by 4 to obtain the number of segments KCluster. That is, KCluster = Round((HWImageMax - HWImageMin) / HWStd × 4), where Round is a function for rounding the decimal part to the nearest integer.

[0088] Perform segmentation on the KJImage using the K-Means clustering algorithm. K-Means is a classic unsupervised learning algorithm widely used in fields such as data clustering, image segmentation, and pattern recognition. The number of clusters is KCluster, and the mean-segmented image MeansImage is obtained.

[0089] S102: Hierarchically divide the plant temperature based on the infrared image to construct a temperature level list.

[0090] In this embodiment, the plant temperature can be hierarchically divided based on the infrared image, and then a temperature level list can be constructed.

[0091] This embodiment does not limit the specific method of constructing the temperature level list. Generally, a temperature level list can be constructed; each element of the temperature level list contains two fields, namely the upper temperature threshold and the lower temperature threshold;

[0092] Assign the current high temperature value to be processed to the maximum infrared pixel value of the infrared image;

[0093] Determine the standard deviation of all pixel values in the infrared image, and use the result of dividing the standard deviation by 4 as the temperature span value;

[0094] Use the result of subtracting the temperature span value from the current high temperature value to be processed as the temporary lower temperature limit value, and use the current high temperature value to be processed as the temporary upper temperature limit value;

[0095] Obtain all pixels in the infrared image whose pixel values are less than or equal to the temporary lower temperature limit value and greater than or equal to the temporary upper temperature limit value as the filtered pixels;

[0096] If the number of filtered pixels is greater than the result of dividing the number of pixels of the infrared image by 4, then use the result of subtracting half of the temperature span value from the current high temperature value to be processed as the temporary lower temperature limit value;

[0097] Store the temporary temperature upper limit value and the temporary temperature lower limit value as an element in the temperature level list;

[0098] Assign the current high value of the temperature to be processed to the temporary temperature lower limit value;

[0099] If the current high value of the temperature to be processed is greater than the minimum infrared pixel value, then start over from the step of taking the result of subtracting the temperature span value from the current high value of the temperature to be processed as the temporary temperature lower limit value.

[0100] Specifically, in this embodiment, an empty temperature level list CJPPList can be established first. Each element of CJPPList contains two fields, namely the lower temperature threshold and the upper temperature threshold;

[0101] Set the current high value of the temperature to be processed WDGW, and assign the current high value of the temperature to be processed to the maximum infrared pixel value, that is, WDGW = HWImageMax;

[0102] Set the temperature span value WDKD, and assign the temperature span value to the result of dividing the standard deviation of all pixel values in the infrared image by 4, that is, WDKD = HWStd / 4;

[0103] Set the temporary temperature lower limit value TempXX and the temporary temperature upper limit value TempSX; take the result of subtracting the temperature span value from the current high value of the temperature to be processed as the temporary temperature lower limit value, that is, TempXX = WDGW - WDKD; take the current high value of the temperature to be processed as the temporary temperature upper limit value, that is, TempSX = WDGW.

[0104] Furthermore, obtain all pixels in the infrared image whose pixel values are less than or equal to the temporary temperature lower limit value and greater than or equal to the temporary temperature upper limit value as the filtered pixels;

[0105] Determine the number of filtered pixels. If the number of filtered pixels is greater than the result of dividing the number of pixels in the infrared image by 4, that is, greater than HWNumber / 4, then take the result of subtracting half of the temperature span value from the current high value of the temperature to be processed as the temporary temperature lower limit value, that is, TempXX = WDGW - WDKD / 2; further store the temporary temperature upper limit value and the temporary temperature lower limit value as an element in the temperature level list. The upper temperature threshold of this element is equal to TempXX, and the lower temperature threshold is equal to TempSX;

[0106] If the number of filtered pixels is less than or equal to the result of dividing the number of pixels in the infrared image by 4, that is, less than or equal to HWNumber / 4, then the temporary temperature upper limit value and the temporary temperature lower limit value can be directly stored as an element in the temperature level list;

[0107] After storage, assign the current high value of the temperature to be processed to the lower limit value of the temporarily stored temperature, i.e., WDGW = TempSX;

[0108] Further, determine whether WDGW is greater than HWImageMin;

[0109] If so, it can be determined that the temperature level of the plant has not been completely divided. At this time, it is necessary to jump to the step of taking the result of subtracting the temperature span value from the current high value of the temperature to be processed as the lower limit value of the temporarily stored temperature and start execution, and loop execution in the above process order;

[0110] If not, it can be determined that the temperature level of the plant has been completely divided, and a complete temperature level list can be obtained.

[0111] S103: Construct a temperature level operator, and input the upper and lower limits of the temperature thresholds of each temperature level in the temperature level list into the temperature level operator in turn to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at each temperature level.

[0112] In this embodiment, a temperature level operator can be constructed, and the upper and lower limits of the temperature thresholds of each temperature level in the temperature level list are input into the temperature level operator in turn to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at each temperature level.

[0113] This embodiment does not limit the specific method for obtaining the homography matrix. Generally, a temperature level operator can be constructed; the input of the temperature level operator is the lower limit and the upper limit of the temperature threshold, and the output is the homography matrix;

[0114] Input the lower limit and the upper limit of the temperature thresholds of each level in the temperature level list into the temperature level operator in turn;

[0115] Based on the temperature level operator, set the values of the pixels in the infrared image that are less than the lower limit of the temperature threshold and greater than the upper limit of the temperature threshold to the minimum infrared pixel value to obtain the infrared image to be calculated;

[0116] Based on the temperature level operator, call the SIFT algorithm to match the mean segmentation image and the infrared image to be calculated to obtain the first key point list of the visible light image and the second key point list of the infrared image to be calculated;

[0117] Based on the temperature level operator, call the brute-force matching algorithm to determine the matching point pairs in the first key point list and the second key point list, and store the matching point pairs in the matching point list of the temperature level operator;

[0118] Based on the temperature level operator, call the RANSAC (Random Sample Consensus) algorithm to determine the homography matrix between the matching point pairs in the matching point list of the temperature level operator.

[0119] Specifically, a temperature level operator CJOpt is established. The inputs of CJOpt are the lower temperature threshold TempXX and the upper temperature threshold TempSX.

[0120] The built-in operation process of the temperature level operator for TempXX and TempSX is as follows:

[0121] Process all pixels in HWImage. Keep the pixel values of the pixels greater than or equal to TempXX and less than or equal to TempSX unchanged, and set the pixel values of other pixels to HWImageMin. Store the processed result in the infrared image to be calculated CJOptTempImage.

[0122] Use the SIFT algorithm to match MeansImage and CJOptTempImage to obtain the first key point list srcpts on KJImage and the second key point list dstpts on CJOptTempImage;

[0123] Use the brute-force matching algorithm to obtain the most approximate key point pairs in srcpts and dstpts as the matching point pairs, and store the matching point pairs in the temperature level operator matching point list CJOptMatches;

[0124] Use the RANSAC algorithm to calculate the homography matrix CJOptMatrix between the corresponding matching point pairs in CJOptMatches;

[0125] Output CJOptMatrix as the result of CJOpt.

[0126] In this embodiment, the upper temperature threshold and the lower temperature threshold of each level in each temperature level list can be input into the temperature level operator in sequence to obtain the corresponding homography matrix for each level.

[0127] Specifically, in this embodiment, the matching counter PPCounter = 1 can be set;

[0128] Input the lower temperature threshold and the upper temperature threshold of the PPCounter-th element in the temperature level list into CJOpt to obtain the homography matrix CJOptMatrix of the PPCounter-th temperature level as the output;

[0129] Set PPCounter = PPCounter + 1;

[0130] Judge whether PPCounter is less than or equal to the number of elements in CJPPList;

[0131] If so, jump to the step of inputting the lower temperature threshold and the upper temperature threshold of the PPCounter-th element in the temperature level list into CJOpt and start execution;

[0132] If not, determine whether the threshold upper and lower limits of all temperature levels have been input into CJOpt, and complete the calculation of obtaining the homography matrix corresponding to all levels.

[0133] In this embodiment, when constructing the temperature level list, the elements are stored in the order of decreasing temperature. Therefore, in this embodiment, the upper temperature threshold and the lower temperature threshold of each temperature level in each temperature level list can be input into the temperature level operator in the order of decreasing temperature.

[0134] S104: Perform perspective transformation on the infrared image based on the homography matrix in the order of temperature levels to obtain a transformed image.

[0135] S105: Update the transformed images under all temperature levels to the infrared image in sequence to obtain an image matching result.

[0136] In this embodiment, the infrared image can be perspectively transformed based on the homography matrix in the order of temperature levels to obtain a transformed image, and the transformed images under all temperature levels are updated to the infrared image in sequence to obtain an image matching result.

[0137] Specifically, in this embodiment, when the homography matrix under each temperature level is calculated, all pixels of HWImage can be perspectively transformed through the homography matrix to obtain a transformed image after transformation; and the transformed images under all temperature levels are updated to the infrared image in sequence to obtain an image matching result.

[0138] The overall process example can be:

[0139] Set the matching counter PPCounter = 1;

[0140] Input the lower temperature threshold and the upper temperature threshold of the PPCounter-th element in the temperature level list into CJOpt to obtain the homography matrix CJOptMatrix of the PPCounter-th temperature level as the output;

[0141] Set the mapping matrix PPMatrix, and assign the mapping matrix to the homography matrix CJOptMatrix of the PPCounter-th temperature level, that is, PPMatrix = CJOptMatrix;

[0142] Perform perspective transformation on all pixels of HWImage through PPMatrix to obtain a transformed image BHResult after transformation;

[0143] Update the transformed image into the infrared image, i.e., HWImage = BHResult;

[0144] Set PPCounter = PPCounter + 1;

[0145] Determine whether PPCounter is less than or equal to the number of elements in CJPPList;

[0146] If so, jump to the step of inputting the lower temperature threshold and the upper temperature threshold of the PPCounter-th element in the temperature level list into CJOpt and start execution;

[0147] If not, determine whether the threshold upper and lower limits of all temperature levels have been input into CJOpt, complete the calculation of the homography matrix corresponding to all levels, and obtain the finally updated HWImage, which is the matching result of the infrared image and the visible light image.

[0148] This embodiment does not limit the specific application of the image matching result. In this embodiment, it can be applied to the growth status and pest and disease status of rice grains, scientific farming, increasing yield, and reducing economic losses.

[0149] Based on the above embodiments, the method of the present invention provides an image matching method. By establishing a temperature level operator, the infrared image is divided into multiple levels from the temperature perspective using this operator, and the infrared image is gradually stretched and adjusted for matching, enabling precise matching of images without training samples, and improving the matching efficiency and accuracy of the images.

[0150] The following combines Figure 2 , Figure 2 is a structural block diagram of an image matching device provided by an embodiment of the present invention. The device may include:

[0151] The first module 100 is used to obtain the visible light image and the infrared image of the target plant planting area, and perform image segmentation on the visible light image based on the clustering algorithm to obtain the mean segmentation image;

[0152] The second module 200 is used to perform level division on the plant temperature based on the infrared image to construct a temperature level list;

[0153] The third module 300 is used to construct a temperature level operator, and sequentially input the temperature threshold upper and lower limits of each temperature level in the temperature level list into the temperature level operator to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at each temperature level;

[0154] The fourth module 400 is configured to perform perspective transformation on the infrared image based on the homography matrix in the order of the temperature levels to obtain a transformed image;

[0155] The fifth module 500 is configured to sequentially update all the transformed images at the temperature levels to the infrared image to obtain an image matching result.

[0156] Based on the above embodiments, the present invention provides an image matching method. By establishing a temperature level operator, the infrared image is divided into multiple levels from the temperature perspective using this operator, and the infrared image is gradually stretched and adjusted for matching. Precise matching of the image can be achieved without training samples, improving the matching efficiency and accuracy of the image.

[0157] Based on the above embodiments, the third module 300 may include:

[0158] The first unit is configured to construct the temperature level operator; the input of the temperature level operator is the lower temperature threshold and the upper temperature threshold, and the output is the homography matrix;

[0159] The second unit is configured to sequentially input the lower temperature threshold and the upper temperature threshold of each temperature level in the temperature level list into the temperature level operator;

[0160] The third unit is configured to set the values of the pixels in the infrared image that are less than the lower temperature threshold and greater than the upper temperature threshold to the minimum infrared pixel value based on the temperature level operator to obtain a to-be-calculated infrared image;

[0161] The fourth unit is configured to call the SIFT algorithm based on the temperature level operator to match the mean-segmented image and the to-be-calculated infrared image to obtain a first key point list of the mean-segmented image and a second key point list of the to-be-calculated infrared image;

[0162] The fifth unit is configured to call the brute-force matching algorithm based on the temperature level operator to determine the matching point pairs in the first key point list and the second key point list, and store the matching point pairs in the temperature level operator matching point list;

[0163] The sixth unit is configured to call the RANSAC algorithm based on the temperature level operator to determine the homography matrix between the matching point pairs in the temperature level operator matching point list.

[0164] Based on the above embodiments, the second module 200 may include:

[0165] The seventh unit is configured to construct a temperature level list; each element of the temperature level list contains two fields, namely the upper temperature threshold and the lower temperature threshold;

[0166] The eighth unit is used to assign the high value of the currently to-be-processed temperature to the maximum infrared pixel value of the infrared image;

[0167] The ninth unit is used to determine the standard deviation of all pixel values in the infrared image, and use the result of dividing the standard deviation by 4 as the temperature span value;

[0168] The tenth unit is used to use the result of subtracting the temperature span value from the high value of the currently to-be-processed temperature as the temporary temperature lower limit value, and use the high value of the currently to-be-processed temperature as the temporary temperature upper limit value;

[0169] The eleventh unit is used to obtain all pixels in the infrared image whose pixel values are less than or equal to the temporary temperature lower limit value and greater than or equal to the temporary temperature upper limit value as the screened pixels;

[0170] The twelfth unit is used to, if the number of the screened pixels is greater than the result of dividing the number of pixels of the infrared image by 4, use the result of subtracting half of the temperature span value from the high value of the currently to-be-processed temperature as the temporary temperature lower limit value;

[0171] The thirteenth unit is used to store the temporary temperature upper limit value and the temporary temperature lower limit value as an element in the temperature level list;

[0172] The fourteenth unit is used to assign the high value of the currently to-be-processed temperature to the temporary temperature lower limit value;

[0173] The fifteenth unit is used to, if the high value of the currently to-be-processed temperature is greater than the minimum infrared pixel value, start executing from the tenth unit again.

[0174] Based on the above embodiments, the first module 100 may include:

[0175] The sixteenth unit is used to obtain the maximum infrared pixel value, the minimum infrared pixel value, and the standard deviation of all pixel values in the infrared image;

[0176] The seventeenth unit is used to subtract the minimum infrared pixel value from the maximum infrared pixel value to obtain an infrared pixel value difference;

[0177] The eighteenth unit is used to perform a rounding operation on the result of dividing the infrared pixel value difference by the standard deviation and multiplying by 4 to obtain the number of segments;

[0178] The nineteenth unit is used to perform K-means clustering algorithm segmentation on the visible light image based on the number of segments to obtain the mean segmentation image.

[0179] Based on the above embodiments, the present invention further provides an electronic device, which may include a memory and a processor. Among them, a computer program is stored in the memory. When the processor calls the computer program in the memory, the steps provided by the above embodiments can be implemented. Of course, the device may further include various necessary network interfaces, power supplies, and other components, etc.

[0180] The present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by an execution terminal or a processor, the method provided by the embodiments of the present invention can be implemented; the storage medium may include: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0181] In this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

Claims

1. An image matching method, characterized in that: include: Obtaining a visible light image and an infrared image of a target plant planting area, and performing image segmentation on the visible light image based on a clustering algorithm to obtain a mean segmentation image; Based on the infrared image, the plant temperature is divided into levels to construct a temperature level list; Constructing a temperature level operator, inputting the upper and lower limits of the temperature thresholds of each temperature level in the temperature level list into the temperature level operator in turn, and obtaining a homography matrix of matching points between the infrared image and the mean segmentation image at each temperature level; Performing perspective transformation on the infrared image based on the homography matrix in the order of the temperature levels to obtain a transformed image; Sequentially updating the transformed images at all the temperature levels into the infrared image to obtain an image matching result; The temperature level operator is constructed, and the upper and lower limits of the temperature thresholds of each temperature level in the temperature level list are input into the temperature level operator in sequence to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at each temperature level, including: Constructing the temperature level operator; the input of the temperature level operator is the lower limit of the temperature threshold and the upper limit of the temperature threshold, and the output is the homography matrix; Inputting the lower temperature threshold and the upper temperature threshold of each temperature level in the temperature level list into the temperature level operator in sequence; Based on the temperature level operator, the values ​​of the pixels in the infrared image that are less than the lower limit of the temperature threshold and greater than the upper limit of the temperature threshold are set as the minimum infrared pixel value to obtain the infrared image to be calculated; Based on the temperature level operator, the SIFT algorithm is called to match the mean segmented image and the infrared image to be calculated, so as to obtain a first key point list of the visible light image and a second key point list of the infrared image to be calculated; Based on the temperature level operator, a brute force matching algorithm is called to determine matching point pairs in the first key point list and the second key point list, and the matching point pairs are stored in the temperature level operator matching point list; The RANSAC algorithm is called based on the temperature level operator to determine the homography matrix between the matching point pairs in the matching point list of the temperature level operator.

2. The image matching method according to claim 1, characterized in that: The step of dividing the plant temperature into different levels based on the infrared image and constructing a temperature level list includes: Constructing a temperature level list; each element of the temperature level list includes two fields, namely, an upper temperature threshold and a lower temperature threshold; Assigning the current high temperature value to be processed as the maximum infrared pixel value of the infrared image; Determine the standard deviation of all pixel values ​​in the infrared image, and divide the standard deviation by 4 as the temperature span value; The result of subtracting the temperature span value from the current high value of the temperature to be processed is used as the lower limit value of the temporary temperature, and the current high value of the temperature to be processed is used as the upper limit value of the temporary temperature; Acquire all pixels in the infrared image whose pixel values ​​are less than or equal to the lower limit value of the temporary storage temperature or greater than or equal to the upper limit value of the temporary storage temperature as filtered pixels; If the number of pixels after the screening is greater than the result of dividing the number of pixels in the infrared image by 4, the result of subtracting the temperature span value divided by 2 from the current high value of the temperature to be processed is used as the lower limit of the temporary temperature; wherein the lower limit of the temporary temperature = the current high value of the temperature to be processed - (the temperature span value / 2); storing the temporary temperature upper limit value and the temporary temperature lower limit value as an element in the temperature level list; Assigning the current high value of the temperature to be processed to the lower limit value of the temporary temperature; If the current high temperature value to be processed is greater than the minimum infrared pixel value, the step of subtracting the temperature span value from the current high temperature value to be processed as the lower limit value of the temporary temperature is started again.

3. The image matching method according to claim 1, characterized in that: The step of performing image segmentation on the visible light image based on a clustering algorithm to obtain a mean segmented image includes: Obtaining a maximum infrared pixel value, a minimum infrared pixel value, and a standard deviation of all pixel values ​​in the infrared image; Subtracting the minimum infrared pixel value from the maximum infrared pixel value to obtain an infrared pixel value difference; The infrared pixel value difference is divided by the standard deviation and multiplied by 4, and the result is rounded to an integer to obtain the number of segmentations; The visible light image is segmented using a K-means clustering algorithm based on the number of segmentations to obtain the mean segmented image.

4. An image matching device, characterized in that: include: The first module is used to obtain visible light images and infrared images of the target plant planting area, and perform image segmentation on the visible light images based on a clustering algorithm to obtain mean segmentation images; The second module is used to classify the plant temperature based on the infrared image and construct a temperature level list; The third module is used to construct a temperature level operator, and input the upper and lower limits of the temperature threshold of each temperature level in the temperature level list into the temperature level operator in turn to obtain the homography matrix of the matching points between the infrared image and the mean segmentation image at each temperature level; A fourth module is used to perform perspective transformation on the infrared image based on the homography matrix according to the order of the temperature levels to obtain a transformed image; A fifth module is used to sequentially update the transformed images at all the temperature levels into the infrared image to obtain an image matching result; The third module comprises: The first unit is used to construct the temperature level operator; the input of the temperature level operator is the lower temperature threshold and the upper temperature threshold, and the output is the homography matrix; A second unit is used to sequentially input the lower limit of the temperature threshold and the upper limit of the temperature threshold of each temperature level in the temperature level list into the temperature level operator; The third unit is used to set the values ​​of the pixels in the infrared image that are less than the lower limit of the temperature threshold and greater than the upper limit of the temperature threshold as the minimum infrared pixel value based on the temperature level operator to obtain the infrared image to be calculated; The fourth unit is used to call the SIFT algorithm based on the temperature level operator to match the mean segmentation image and the infrared image to be calculated, so as to obtain a first key point list of the visible light image and a second key point list of the infrared image to be calculated; A fifth unit is configured to determine matching point pairs in the first key point list and the second key point list by calling a brute force matching algorithm based on the temperature level operator, and store the matching point pairs in the temperature level operator matching point list; The sixth unit is used to call the RANSAC algorithm based on the temperature level operator to determine the homography matrix between the matching point pairs in the matching point list of the temperature level operator.

5. The image matching device according to claim 4, characterized in that: The second module comprises: The seventh unit is used to construct a temperature level list; each element of the temperature level list includes two fields, namely, an upper temperature threshold and a lower temperature threshold; An eighth unit is used to assign the current high-bit value of the temperature to be processed as the maximum infrared pixel value of the infrared image; A ninth unit is used to determine a standard deviation of all pixel values ​​in the infrared image, and divide the standard deviation by 4 to obtain a temperature span value; A tenth unit is used for subtracting the temperature span value from the current high value of the temperature to be processed as the lower limit value of the temporary temperature, and taking the current high value of the temperature to be processed as the upper limit value of the temporary temperature; The eleventh unit is used to obtain all pixels in the infrared image whose pixel values ​​are less than or equal to the lower limit value of the temporary storage temperature or greater than or equal to the upper limit value of the temporary storage temperature as the filtered pixels; The twelfth unit is used for, if the number of the screened pixels is greater than the number of pixels of the infrared image divided by 4, taking the result of subtracting the temperature span value divided by 2 from the current high value of the temperature to be processed as the lower limit value of the temporary temperature; wherein the lower limit value of the temporary temperature = the current high value of the temperature to be processed - (the temperature span value / 2); A thirteenth unit, used for storing the temporary storage temperature upper limit value and the temporary storage temperature lower limit value as an element in the temperature level list; A fourteenth unit is used to assign the current high value of the temperature to be processed to the lower limit value of the temporary temperature; The fifteenth unit is used to restart execution from the tenth unit if the current high temperature value to be processed is greater than the minimum infrared pixel value.

6. The image matching device according to claim 4, characterized in that: The first module comprises: A sixteenth unit is used to obtain a maximum infrared pixel value, a minimum infrared pixel value and a standard deviation of all pixel values ​​in the infrared image; A seventeenth unit is used to obtain an infrared pixel value difference by subtracting the minimum infrared pixel value from the maximum infrared pixel value; An eighteenth unit is used for rounding off the result of dividing the infrared pixel value difference by the standard deviation and multiplying by 4 to obtain the number of divisions; The nineteenth unit is used to perform K-means clustering algorithm segmentation on the visible light image based on the number of segmentations to obtain the mean segmented image.

7. An electronic device, characterized in that: include: Memory, for storing computer programs; A processor, configured to implement the image matching method as claimed in any one of claims 1 to 3 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image matching method according to any one of claims 1 to 3 is implemented.

Images