A fruit tree canopy information multiple recognition device and recognition method

Abstract

The invention discloses a complex identification device and method for fruit tree canopy information, and the invention relates to the technical field of crop phenotype acquisition. The invention comprises the following steps: calibrating binocular camera and infrared camera parameters, using a drone carrying the calibrated binocular camera and infrared camera to collect images around different fruit trees with known canopy information; preprocessing original color images and infrared images; establishing original color image and infrared image detection models, using the two preprocessed images as sample images, inputting them into two models respectively, and training the two models respectively with known canopy information as labels; collecting canopy images of fruit trees to be detected, and inputting them into the two trained models respectively after preprocessing; calculating the similarity values ​​of two groups of fruit tree canopy information according to model output data; setting a similarity threshold, comparing the similarity value with the threshold, and outputting corresponding data as accurate fruit tree canopy information based on the comparison situation.

Description

Technical Field

[0001] The invention relates to the technical field of crop phenotype acquisition, and in particular to a fruit tree canopy information multiple recognition device and recognition method. Background Art

[0002] Traditional fruit tree canopy recognition is mainly done through manual identification. Due to the low efficiency, large workload and lack of scalability of manual identification methods, the recognition ability of fruit tree canopies is not strong and the recognition range is limited. With the development of image processing technology, especially the application of machine vision technology, such problems will be gradually alleviated. The use of intelligent recognition technology to detect, identify and classify fruit tree canopies provides a theoretical basis for subsequent chemical control work. In addition, diagnostic classification of images extracted from preliminary detection can not only provide direction and data basis for the identification of fruit tree canopies, but also rationally apply fertilizers and pesticides to minimize damage to the ecological environment. The key is to increase farmers' crop yields and product quality and save a lot of labor.

[0003] In the prior art, a patent application with publication number CN111738165A discloses a method for extracting a single plant canopy from a high-resolution drone visible light remote sensing image, which specifically includes the following steps: 1. Preprocessing the acquired drone remote sensing data, cropping, cleaning, and labeling the data set; 2. Using a deep convolutional neural network U-Net to obtain a preliminary canopy segmentation result, using the labeled data set to train the U-Net model, and using the trained U-Net model to segment the image to obtain a probability grayscale map; 3. Based on the marker-controlled watershed algorithm, the probability grayscale map is optimized to obtain the final single plant canopy segmentation result. This method can identify and extract plant canopies, but the plant canopy images collected by this method in a natural environment are affected by factors such as illumination, background (ground film, ground, sky), and weed interference, which causes certain difficulties in the recognition and extraction of fruit tree canopy features. Therefore, the use of traditional image acquisition methods will have a certain impact on the recognition and extraction of features, which reduces the accuracy and effectiveness of the recognition and extraction information.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0005] The purpose of the present invention is to provide a fruit tree canopy information multiple recognition device and recognition method to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A fruit tree canopy information complex identification device, specifically comprising:

[0008] The sample data acquisition module is used to calibrate the binocular camera and infrared camera parameters and obtain the camera parameters, wherein the camera parameters include focal length, principal point position coordinates and lens distortion coefficient, and use a drone carrying the calibrated binocular camera and infrared camera to collect images around the canopies of different fruit trees at different growth stages and with known canopy information, and obtain multiple groups of original color images and infrared images of the fruit tree canopies by duplex acquisition, wherein the canopy information includes the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density;

[0009] An image preprocessing module is used to preprocess the original color image and infrared image of the fruit tree canopy, wherein the preprocessing includes distortion correction and noise reduction enhancement to obtain the original color detection image and infrared detection image of the fruit tree canopy;

[0010] The prediction model training module is used to establish the original color image detection model and the infrared image detection model. The original color detection image and the infrared detection image of the fruit tree canopy are used as sample images, which are input into the original color image detection model and the infrared image detection model respectively. The two models are trained respectively with the known canopy information including the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density as labels.

[0011] The canopy information prediction module is used to collect the canopy original color image and infrared image of the fruit tree canopy to be detected by using a calibrated binocular camera and infrared camera carried by a drone. After preprocessing, the images are input into the trained original color image detection model and infrared image detection model respectively to obtain two sets of fruit tree canopy prediction information data;

[0012] A similarity analysis module is used to generate two groups of fruit tree canopy feature vectors based on the prediction information data of the two groups of fruit tree canopies, calculate the Euclidean distance and cosine similarity between the two groups of feature vectors, and calculate the similarity values ​​of the two groups of fruit tree canopy feature vectors by weighted average method according to the Euclidean distance and cosine similarity;

[0013] The precise information output module is used to set a similarity threshold, compare the similarity values ​​of the two groups of feature vectors with the pre-set similarity threshold, and judge whether the predicted fruit tree canopy information is accurate based on different comparison results. If the similarity value is greater than or equal to the similarity threshold, the mean of the corresponding data in the two groups of fruit tree canopy feature vectors is output as the precise information of the fruit tree canopy. The compound acquisition method is to collect the original color image and infrared image of the canopy of the same target fruit tree at different angles.

[0014] Furthermore, the calibrated binocular camera and infrared camera are installed on the UAV in the following way: the binocular camera and infrared camera are installed at fixed angles and heights, and the installation angles and heights of the binocular camera and infrared camera are different. After the installation is completed, image collection is carried out around the canopies of different fruit trees at different growth stages and with known canopy information. The image collection adopts a duplex collection method to ensure multi-angle monitoring of the same fruit tree canopy.

[0015] Furthermore, the original color image and infrared image of the fruit tree canopy are preprocessed, wherein the distortion correction includes radial distortion correction and tangential distortion correction. The specific logic of radial distortion correction and tangential distortion correction is as follows: three radial distortion coefficients are established and calibrated as , and , the formula for obtaining the ideal image coordinates without tangential distortion is:

[0016]

[0017] Establish two tangential distortion coefficients and calibrate them as and , the model expression without radial distortion is:

[0018]

[0019] By combining the two distortions, the effects of both can be eliminated at the same time. The parameter expression after superposition is:

[0020]

[0021] In the formula, , and are three different radial distortion coefficients, and are two different tangential distortion coefficients, represents the undistorted image coordinates under the ideal optical center system, is the image coordinate of a pixel in the captured image, is the distance from the correction point to the imaging center, where the image coordinate system takes the optical center as the image center, the origin as the intersection of the camera optical axis and the imaging plane, and Axis positive direction, downward The positive direction of the axis is established.

[0022] Furthermore, the method for performing noise reduction and enhancement processing on the original color image and infrared image of the fruit tree canopy is: using a wavelet transform denoising method to perform denoising processing on the distortion-corrected image, and the specific steps of the wavelet transform denoising method include: decomposing the distortion-corrected image through wavelet transform to obtain wavelet coefficients of the image at different scales and directions; performing threshold processing on the wavelet coefficients to set the low-amplitude wavelet coefficients to zero and retain the high-amplitude wavelet coefficients; performing inverse transform on the wavelet coefficients after threshold processing, reconstructing the processed coefficients into an image, and completing image denoising processing;

[0023] Bilateral filtering is used to enhance the details of the fruit tree canopy image. The specific filter transformation is based on the formula:

[0024]

[0025] In the formula, is the coordinate vector in the image coordinate system, is the coordinate vector The gray value at Gray value The gray value after bilateral filtering transformation is are all Gaussian functions, where The formula is:

[0026]

[0027]

[0028] In the formula, is the coordinate vector in the image coordinate system, is the coordinate vector The gray value at and They are The standard deviation of .

[0029] Furthermore, based on the convolutional neural network and long short-term memory network model, an original color image detection model and an infrared image detection model were established. The inputs of the trained original color image detection model and infrared image detection model were the original color detection image and infrared detection image of the fruit tree canopy respectively, and the outputs were canopy information, including the canopy edge contour circumference, canopy surface area, canopy volume and canopy density.

[0030] Furthermore, based on the prediction information data of the two groups of fruit tree canopies, two groups of fruit tree canopy feature vectors are generated respectively: and ,in is the original color feature vector of the fruit tree canopy generated based on the prediction information data output by the original color image detection model. The infrared feature vector of the fruit tree canopy generated based on the prediction information data output by the infrared image detection model. are the original color image prediction feature data, which respectively represent the original color image prediction values ​​of the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density of the fruit tree to be detected. are the infrared image prediction feature data, which respectively represent the infrared image prediction values ​​of the canopy edge contour circumference, canopy surface area, canopy volume and canopy density of the fruit tree to be detected.

[0031] Furthermore, the formulas for calculating the Euclidean distance and cosine similarity between two sets of feature vectors are:

[0032]

[0033]

[0034] In the formula, is the feature vector and The Euclidean distance between is the feature vector and The cosine similarity of Represents the feature vector Middle feature data, Represents the feature vector Middle feature data, is the index of the feature data in the feature vector, and Represents the eigenvectors and The modulus value of

[0035] The formula based on which the similarity values ​​of the two groups of fruit tree canopy feature vectors are calculated by weighted average method based on Euclidean distance and cosine similarity is:

[0036]

[0037] In the formula, is the similarity value; and are the weight coefficients of cosine similarity and Euclidean distance respectively;

[0038] The calibration sets the similarity threshold to , the similarity values ​​of the two sets of feature vectors are compared with the preset similarity threshold, and the logic for judging whether the predicted fruit tree canopy information is accurate is as follows:

[0039] when When , it is judged that the similarity of the fruit tree canopy prediction information data output by the two models meets the requirements, and the mean of the corresponding feature data in the two feature vectors is output as the precise information of the fruit tree canopy.

[0040] The present invention also provides a method for multiple identification of fruit tree canopy information, which is used to control the above-mentioned device for multiple identification of fruit tree canopy information, and the specific steps include:

[0041] Calibrate the binocular camera and infrared camera parameters, and obtain the camera parameters, wherein the camera parameters include focal length, principal point position coordinates, and lens distortion coefficient; use a drone to carry the calibrated binocular camera and infrared camera to collect images around the canopies of different fruit trees at different growth stages and with known canopy information; obtain multiple groups of original color images and infrared images of the fruit tree canopies by duplex collection, wherein the canopy information includes the canopy edge contour circumference, canopy surface area, canopy volume, and canopy density;

[0042] Preprocessing the original color image and the infrared image of the fruit tree canopy, wherein the preprocessing includes distortion correction and noise reduction enhancement, to obtain the original color detection image and the infrared detection image of the fruit tree canopy;

[0043] Establish original color image detection model and infrared image detection model, take original color detection image and infrared detection image of fruit tree canopy as sample images, input them into original color image detection model and infrared image detection model respectively, and use known canopy information including canopy edge contour perimeter, canopy surface area, canopy volume and canopy density as labels to train the two models respectively;

[0044] The canopy original color image and infrared image of the fruit tree to be detected are collected by using a calibrated binocular camera and infrared camera carried by a drone, and the canopy original color image and infrared image of the fruit tree to be detected are respectively input into the trained original color image detection model and infrared image detection model to obtain two sets of prediction information data of the fruit tree canopy;

[0045] Based on the prediction information data of the two groups of fruit tree canopies, two groups of fruit tree canopy feature vectors are generated, the Euclidean distance and cosine similarity between the two groups of feature vectors are calculated, and the similarity values ​​of the two groups of fruit tree canopy feature vectors are calculated by weighted average method according to the Euclidean distance and cosine similarity;

[0046] A similarity threshold is set, and the similarity values ​​of the two sets of feature vectors are compared with the pre-set similarity threshold. Based on different comparison results, it is determined whether the predicted fruit tree canopy information is accurate. If the similarity value is greater than or equal to the similarity threshold, the mean of the corresponding data in the two sets of fruit tree canopy feature vectors is output as the precise information of the fruit tree canopy.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] By using drones carrying binocular cameras and infrared cameras, a compound acquisition method is adopted to collect original color images and infrared images of fruit tree canopies from multiple angles. By simultaneously acquiring visible light images and infrared images, rich canopy information can be provided from multiple dimensions. At the same time, multi-camera acquisition can effectively reduce the influence of factors such as light, background, and weed interference on the fruit tree canopy images collected in natural environments. The collected original images are preprocessed to enhance image details and prepare for subsequent feature extraction. Two canopy information prediction models are established, and a comparison and analysis mode is used, which is equivalent to two sets of standard comparisons, which helps to improve the reliability of the model output data. The binocular camera and the infrared camera are set to different shooting angles. At the same time, there is an angle deviation in the image collected by the binocular camera. The collected photo information is stereo image information of the same plane and different dimensions. This type of image information is closest to reality, which further improves the accuracy and effectiveness of canopy information extraction through the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the overall structure of the device of the present invention;

[0050] Figure 2 It is a schematic diagram of the overall method flow of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0052] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0053] Example:

[0054] See also Figure 1 , the present invention provides a device structure:

[0055] A fruit tree canopy information complex identification device, specifically comprising:

[0056] The sample data acquisition module is used to calibrate the binocular camera and infrared camera parameters and obtain the camera parameters, wherein the camera parameters include focal length, principal point position coordinates and lens distortion coefficient, and use a drone carrying the calibrated binocular camera and infrared camera to collect images around the canopies of different fruit trees at different growth stages and with known canopy information, and obtain multiple groups of original color images and infrared images of the fruit tree canopies by duplex acquisition, wherein the canopy information includes the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density;

[0057] The binocular camera consists of two cameras with exactly the same parameters and placed in parallel. In this ideal state, the optical axes of the two cameras remain parallel and the projection planes are in the same plane. The binocular camera and infrared camera are calibrated using Zhang Zhengyou's calibration method. The specific steps include: making a checkerboard calibration plate; moving the calibration plate and collecting images; extracting corner points; obtaining internal and external parameters and distortion coefficients, and the distortion coefficients include radial distortion coefficients and tangential distortion coefficients. The binocular camera first performs single-target calibration, and then performs binocular calibration based on the single-target calibration data. Zhang Zhengyou's calibration method is a commonly used camera calibration method and will not be described here.

[0058] The calibrated binocular camera and infrared camera are installed on the drone in the following way: the binocular camera and infrared camera are installed in a fixed angle and height manner, and the binocular camera and infrared camera are installed at different angles and heights. At this time, the binocular camera and infrared camera have different shooting angles, and there is an angle deviation in the image collected by the binocular camera. The collected photo information is stereoscopic image information of the same plane and different dimensions. This kind of image information is closest to reality. After the installation is completed, image collection is performed around the canopies of different fruit trees at different growth stages and with known canopy information. The image collection adopts a compound collection method to ensure multi-angle monitoring of the same fruit tree canopy. The compound collection method is to collect the original color image and infrared image of the canopy of the same target fruit tree at different angles;

[0059] The canopy density refers to the leaf area density within the canopy of the fruit tree. The canopy information includes the canopy edge contour circumference, canopy surface area, canopy volume and canopy density. The method for obtaining the canopy density is to screen and extract the canopy information recorded by historical manual identification.

[0060] An image preprocessing module is used to preprocess the original color image and infrared image of the fruit tree canopy, wherein the preprocessing includes distortion correction and noise reduction enhancement to obtain the original color detection image and infrared detection image of the fruit tree canopy;

[0061] The original color image and infrared image of the fruit tree canopy are preprocessed, wherein the distortion correction includes radial distortion correction and tangential distortion correction. The specific logic of radial distortion correction and tangential distortion correction is as follows: three radial distortion coefficients are established and calibrated as , and , the formula for obtaining the ideal image coordinates without tangential distortion is:

[0062]

[0063] Establish two tangential distortion coefficients and calibrate them as and , the model expression without radial distortion is:

[0064]

[0065] By combining the two distortions, the effects of both can be eliminated at the same time. The parameter expression after superposition is:

[0066]

[0067] In the formula, , and are three different radial distortion coefficients, and are two different tangential distortion coefficients. The values ​​of the distortion coefficients are determined according to Zhang Zhengyou’s calibration method. represents the undistorted image coordinates under the ideal optical center system, is the image coordinate of a pixel in the captured image, is the distance from the correction point to the imaging center, where the image coordinate system takes the optical center as the image center, the origin as the intersection of the camera optical axis and the imaging plane, and Axis positive direction, downward The positive direction of the axis is established.

[0068] The method for denoising and enhancing the original color image and infrared image of the fruit tree canopy is as follows: the denoising method of wavelet transform is used to denoise the image after distortion correction. The specific steps of the wavelet transform denoising method include: decomposing the image after distortion correction by wavelet transform to obtain the wavelet coefficients of the image at different scales and directions; thresholding the wavelet coefficients to set the low-amplitude wavelet coefficients to zero and retain the high-amplitude wavelet coefficients; inversely transforming the wavelet coefficients after thresholding, reconstructing the processed coefficients into an image, and completing the image denoising;

[0069] Bilateral filtering is used to enhance the details of the fruit tree canopy image. The specific filter transformation is based on the formula:

[0070]

[0071] In the formula, is the coordinate vector in the image coordinate system, is the coordinate vector The gray value at Gray value The gray value after bilateral filtering transformation is are all Gaussian functions, where The formula is:

[0072]

[0073]

[0074] In the formula, is the coordinate vector in the image coordinate system, is the coordinate vector The gray value at and They are The standard deviation of .

[0075] The prediction model training module is used to establish the original color image detection model and the infrared image detection model. The original color detection image and the infrared detection image of the fruit tree canopy are used as sample images, which are input into the original color image detection model and the infrared image detection model respectively. The two models are trained respectively with the known canopy information including the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density as labels.

[0076] Based on the convolutional neural network and long short-term memory network model, the original color image detection model and the infrared image detection model are established, wherein the convolutional neural network consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, wherein the activation function in the convolutional layer is function, The specific expression of the function is:

[0077]

[0078] in, Indicates The corresponding convolutional layers, It means the The corresponding convolutional layer The feature map eigenvalues;

[0079] For the fully connected layer, the number of neurons in the fully connected layer is set to 32, the initial neural network learning rate is set to 0.001, and the number of training rounds is 100;

[0080] The LSTM model of the long short-term memory network model selects an activation function and an optimization algorithm, wherein the Tanh function is selected as the activation function, and Adam is selected as the optimization algorithm of the LSTM model; the formula of the Tanh function is:

[0081]

[0082] In the formula, Represents the Tanh function, independent variable Represents the weighted sum of the neuron's input, that is, the result of weighted summation of the input received by the neuron from the previous layer;

[0083] At the same time, the hyperparameters of the LSTM model are set, and the hyperparameters of the LSTM model include: the number of network layers, the number of iterations, the learning rate, the batch size, the number of training times, the batch processing number, and the number of hidden layer neurons;

[0084] The number of network layers is set to 3 layers, the number of iterations is set to 200, the learning rate is set to 0.001, the batch size is set to 32, the number of training times is set to 100, the batch size is set to 256, and the number of hidden layer neurons is 32;

[0085] The inputs of the trained original color image detection model and infrared image detection model are the original color detection images and infrared detection images of the fruit tree canopy respectively, and the outputs are canopy information, including the canopy edge contour circumference, canopy surface area, canopy volume and canopy density.

[0086] The canopy information prediction module is used to collect the canopy original color image and infrared image of the fruit tree canopy to be detected by using a calibrated binocular camera and infrared camera carried by a drone. After preprocessing, the images are input into the trained original color image detection model and infrared image detection model respectively to obtain two sets of fruit tree canopy prediction information data;

[0087] A similarity analysis module is used to generate two groups of fruit tree canopy feature vectors based on the prediction information data of the two groups of fruit tree canopies, calculate the Euclidean distance and cosine similarity between the two groups of feature vectors, and calculate the similarity values ​​of the two groups of fruit tree canopy feature vectors by weighted average method according to the Euclidean distance and cosine similarity;

[0088] Based on the prediction information data of two groups of fruit tree canopies, two groups of fruit tree canopy feature vectors are generated respectively. and ,in is the original color feature vector of the fruit tree canopy generated based on the prediction information data output by the original color image detection model. The infrared feature vector of the fruit tree canopy generated based on the prediction information data output by the infrared image detection model. are the original color image prediction feature data, which respectively represent the original color image prediction values ​​of the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density of the fruit tree to be detected. The infrared image prediction feature data represent the infrared images of the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density of the fruit tree to be detected;

[0089] The formulas for calculating the Euclidean distance and cosine similarity between two sets of feature vectors are:

[0090]

[0091]

[0092] In the formula, is the feature vector and The Euclidean distance between is the feature vector and The cosine similarity of Represents the feature vector Middle feature data, Represents the feature vector Middle feature data, is the index of the feature data in the feature vector, and Represents the eigenvectors and The modulus value of

[0093] The formula based on which the similarity values ​​of the two groups of fruit tree canopy feature vectors are calculated by weighted average method based on Euclidean distance and cosine similarity is:

[0094]

[0095] In the formula, is the similarity value; and are the weight coefficients of cosine similarity and Euclidean distance respectively; since Euclidean distance can directly reflect the similarity relationship between two vectors, we set and and are all greater than 0. The larger the Euclidean distance between two vectors, the smaller the data similarity. Euclidean distance Inversely proportional; cosine similarity is a measure of the similarity between two vectors. The larger the cosine similarity, the more similar the two vectors are. The smaller the cosine similarity, the lower the similarity of the vectors. Therefore, cosine similarity Similarity value Inversely proportional.

[0096] The precise information output module is used to set a similarity threshold, compare the similarity values ​​of the two sets of feature vectors with the pre-set similarity threshold, and judge whether the predicted fruit tree canopy information is accurate based on different comparison results. If the similarity value is greater than or equal to the similarity threshold, the mean of the corresponding data in the two sets of fruit tree canopy feature vectors is output as the precise information of the fruit tree canopy.

[0097] The similarity threshold can be set based on the experience of experts and actual conditions, and the calibration similarity threshold is , similarity threshold It is usually set to 0.7-1.0 and can be set according to different accuracy requirements. The similarity values ​​of the two sets of feature vectors are compared with the preset similarity threshold to determine whether the predicted fruit tree canopy information is accurate. The logic is as follows:

[0098] when When , it is judged that the similarity of the fruit tree canopy prediction information data output by the two models meets the requirements, and the mean of the corresponding feature data in the two feature vectors is output as the accurate information of the fruit tree canopy;

[0099] The output canopy information is as follows:

[0100]

[0101] The other canopy information output includes canopy surface area, canopy volume and canopy density. The calculation method is consistent with the precise value of the edge contour perimeter, so I will not elaborate on it here.

[0102] when When the two models output the prediction information data of the fruit tree canopy, it is judged that the similarity is small and does not meet the similarity requirements. The data are collected and identified again until they meet the similarity requirements.

[0103] See also Figure 2 The present invention also provides a method for multiple identification of fruit tree canopy information, which is used to control the above-mentioned device for multiple identification of fruit tree canopy information, and the specific steps include:

[0104] Step 1: Calibrate the binocular camera and infrared camera parameters and obtain the camera parameters, wherein the camera parameters include focal length, principal point position coordinates and lens distortion coefficient, use a drone to carry the calibrated binocular camera and infrared camera to collect images around the canopies of different fruit trees at different growth stages and with known canopy information, and obtain multiple groups of original color images and infrared images of the fruit tree canopies by duplex collection, wherein the canopy information includes the canopy edge contour circumference, canopy surface area, canopy volume and canopy density;

[0105] Step 2: preprocessing the original color image and infrared image of the fruit tree canopy, wherein the preprocessing includes distortion correction and noise reduction enhancement to obtain the original color detection image and infrared detection image of the fruit tree canopy;

[0106] Step 3: Establish original color image detection model and infrared image detection model, take original color detection image and infrared detection image of fruit tree canopy as sample images, input them into original color image detection model and infrared image detection model respectively, and use known canopy information including canopy edge contour perimeter, canopy surface area, canopy volume and canopy density as labels to train the two models respectively;

[0107] Step 4: Use the calibrated binocular camera and infrared camera carried by the drone to collect the canopy original color image and infrared image of the fruit tree canopy to be detected, and input the canopy image and infrared image of the fruit tree to be detected into the trained original color image detection model and infrared image detection model respectively to obtain two sets of prediction information data of the fruit tree canopy;

[0108] Step 5: Based on the prediction information data of the two groups of fruit tree canopies, two groups of fruit tree canopy feature vectors are generated, and the Euclidean distance and cosine similarity between the two groups of feature vectors are calculated. The similarity values ​​of the two groups of fruit tree canopy feature vectors are calculated by weighted average method according to the Euclidean distance and cosine similarity;

[0109] Step 6: Set a similarity threshold, compare the similarity values ​​of the two sets of feature vectors with the pre-set similarity threshold, and judge whether the predicted fruit tree canopy information is accurate based on different comparison results. If the similarity value is greater than or equal to the similarity threshold, output the mean of the corresponding data in the two sets of fruit tree canopy feature vectors as the precise information of the fruit tree canopy.

[0110] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0111] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A fruit tree canopy information multiple identification device, characterized in that: Specifically include: The sample data acquisition module is used to calibrate the binocular camera and infrared camera parameters and obtain the camera parameters, wherein the camera parameters include focal length, principal point position coordinates and lens distortion coefficient, and use a drone carrying the calibrated binocular camera and infrared camera to collect images around the canopies of different fruit trees at different growth stages and with known canopy information, and obtain multiple groups of original color images and infrared images of the fruit tree canopies by duplex acquisition, wherein the canopy information includes the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density; An image preprocessing module is used to preprocess the original color image and infrared image of the fruit tree canopy, wherein the preprocessing includes distortion correction and noise reduction enhancement to obtain the original color detection image and infrared detection image of the fruit tree canopy; The prediction model training module is used to establish the original color image detection model and the infrared image detection model. The original color detection image and the infrared detection image of the fruit tree canopy are used as sample images, which are input into the original color image detection model and the infrared image detection model respectively. The two models are trained respectively with the known canopy information including the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density as labels. The canopy information prediction module is used to collect the canopy original color image and infrared image of the fruit tree canopy to be detected by using a calibrated binocular camera and infrared camera carried by a drone. After preprocessing, the images are input into the trained original color image detection model and infrared image detection model respectively to obtain two sets of fruit tree canopy prediction information data; A similarity analysis module is used to generate two groups of fruit tree canopy feature vectors based on the prediction information data of the two groups of fruit tree canopies, calculate the Euclidean distance and cosine similarity between the two groups of feature vectors, and calculate the similarity values ​​of the two groups of fruit tree canopy feature vectors by weighted average method according to the Euclidean distance and cosine similarity; The precise information output module is used to set a similarity threshold, compare the similarity values ​​of the two sets of feature vectors with the pre-set similarity threshold, and judge whether the predicted fruit tree canopy information is accurate based on different comparison results. If the similarity value is greater than or equal to the similarity threshold, the mean of the corresponding data in the two sets of fruit tree canopy feature vectors is output as the precise information of the fruit tree canopy.

2. A fruit tree canopy information multiple identification device according to claim 1, characterized in that: The calibrated binocular camera and infrared camera are installed on the drone in the following way: the binocular camera and infrared camera are installed at fixed angles and heights, and the installation angles and heights of the binocular camera and infrared camera are different. After the installation is completed, image acquisition is performed around the canopies of different fruit trees at different growth stages and with known canopy information. The image acquisition adopts a duplex acquisition method to ensure multi-angle monitoring of the same fruit tree canopy. The duplex acquisition method is to collect original color images and infrared images of the canopy of the same target fruit tree at different angles.

3. A fruit tree canopy information multiple identification device according to claim 2, characterized in that: The original color image and infrared image of the fruit tree canopy are preprocessed, wherein the distortion correction includes radial distortion correction and tangential distortion correction. The specific logic of radial distortion correction and tangential distortion correction is as follows: three radial distortion coefficients are established and calibrated as , and , the formula for obtaining the ideal image coordinates without tangential distortion is: ； Establish two tangential distortion coefficients and calibrate them as and , the model expression without radial distortion is: ； The two distortions are combined and superimposed to eliminate the effects of both. The parameter expression after superposition is: ； In the formula, , and are three different radial distortion coefficients, and are two different tangential distortion coefficients, represents the undistorted image coordinates under the ideal optical center system, is the image coordinate of a pixel in the captured image, is the distance from the correction point to the imaging center, where the image coordinate system takes the optical center as the image center, the origin as the intersection of the camera optical axis and the imaging plane, and Axis positive direction, downward The positive direction of the axis is established.

4. The device for multiplex identification of fruit tree canopy information according to claim 3, characterized in that: The method for denoising and enhancing the original color image and infrared image of the fruit tree canopy is as follows: the denoising method of wavelet transform is used to denoise the image after distortion correction. The specific steps of the wavelet transform denoising method include: decomposing the image after distortion correction by wavelet transform to obtain the wavelet coefficients of the image at different scales and directions; thresholding the wavelet coefficients to set the low-amplitude wavelet coefficients to zero and retain the high-amplitude wavelet coefficients; inversely transforming the wavelet coefficients after thresholding, reconstructing the processed coefficients into an image, and completing the image denoising; Bilateral filtering is used to enhance the details of the fruit tree canopy image. The specific filter transformation is based on the formula: ； In the formula, is the coordinate vector in the image coordinate system, is the coordinate vector The gray value at Gray value The gray value after bilateral filtering transformation is are all Gaussian functions, where The formula is: ； ； In the formula, is the coordinate vector in the image coordinate system, is the coordinate vector The gray value at and They are The standard deviation of .

5. The device for multiplex identification of fruit tree canopy information according to claim 4, characterized in that: Based on convolutional neural network and long short-term memory network models, original color image detection model and infrared image detection model were established. The input of the trained original color image detection model and infrared image detection model were the original color detection image and infrared detection image of fruit tree canopy respectively, and the output was canopy information, including canopy edge contour circumference, canopy surface area, canopy volume and canopy density.

6. The device for multiplex identification of fruit tree canopy information according to claim 1, characterized in that: Based on the prediction information data of two groups of fruit tree canopies, two groups of fruit tree canopy feature vectors are generated respectively. and ,in is the original color feature vector of the fruit tree canopy generated based on the prediction information data output by the original color image detection model. The infrared feature vector of the fruit tree canopy generated based on the prediction information data output by the infrared image detection model. are the original color image prediction feature data, which respectively represent the original color image prediction values ​​of the canopy edge contour perimeter, canopy surface area, canopy volume and canopy density of the fruit tree to be detected. are the infrared image prediction feature data, which respectively represent the infrared image prediction values ​​of the canopy edge contour circumference, canopy surface area, canopy volume and canopy density of the fruit tree to be detected.

7. The device for multiplex identification of fruit tree canopy information according to claim 6, characterized in that: The formulas for calculating the Euclidean distance and cosine similarity between two sets of feature vectors are: ； ； In the formula, is the feature vector and The Euclidean distance between is the feature vector and The cosine similarity of Represents the feature vector Middle feature data, Represents the feature vector Middle feature data, is the index of the feature data in the feature vector, and Represents the eigenvectors and The modulus value of The formula based on which the similarity values ​​of the two groups of fruit tree canopy feature vectors are calculated by weighted average method based on Euclidean distance and cosine similarity is: ； In the formula, is the similarity value; and are the weight coefficients of cosine similarity and Euclidean distance respectively, where and and All are greater than 0; The calibration sets the similarity threshold to , the similarity values ​​of the two sets of feature vectors are compared with the preset similarity threshold, and the logic for judging whether the predicted fruit tree canopy information is accurate is as follows: when When , it is judged that the similarity of the fruit tree canopy prediction information data output by the two models meets the requirements, and the mean of the corresponding feature data in the two feature vectors is output as the precise information of the fruit tree canopy.

8. A method for complex identification of fruit tree canopy information, characterized in that: The method for multiple identification of fruit tree canopy information is used to control the multiple identification device for fruit tree canopy information according to any one of claims 1 to 7, and the specific steps include: Calibrate the binocular camera and infrared camera parameters, and obtain the camera parameters, wherein the camera parameters include focal length, principal point position coordinates, and lens distortion coefficient; use a drone to carry the calibrated binocular camera and infrared camera to collect images around the canopies of different fruit trees at different growth stages and with known canopy information; obtain multiple groups of original color images and infrared images of the fruit tree canopies by duplex collection, wherein the canopy information includes the canopy edge contour circumference, canopy surface area, canopy volume, and canopy density; Preprocessing the original color image and the infrared image of the fruit tree canopy, wherein the preprocessing includes distortion correction and noise reduction enhancement, to obtain the original color detection image and the infrared detection image of the fruit tree canopy; Establish original color image detection model and infrared image detection model, take original color detection image and infrared detection image of fruit tree canopy as sample images, input them into original color image detection model and infrared image detection model respectively, and use known canopy information including canopy edge contour perimeter, canopy surface area, canopy volume and canopy density as labels to train the two models respectively; The canopy original color image and infrared image of the fruit tree to be detected are collected by using a calibrated binocular camera and infrared camera carried by a drone, and the canopy original color image and infrared image of the fruit tree to be detected are respectively input into the trained original color image detection model and infrared image detection model to obtain two sets of prediction information data of the fruit tree canopy; Based on the prediction information data of the two groups of fruit tree canopies, two groups of fruit tree canopy feature vectors are generated, the Euclidean distance and cosine similarity between the two groups of feature vectors are calculated, and the similarity values ​​of the two groups of fruit tree canopy feature vectors are calculated by weighted average method according to the Euclidean distance and cosine similarity; A similarity threshold is set, and the similarity values ​​of the two sets of feature vectors are compared with the pre-set similarity threshold. Based on different comparison results, it is determined whether the predicted fruit tree canopy information is accurate. If the similarity value is greater than or equal to the similarity threshold, the mean of the corresponding data in the two sets of fruit tree canopy feature vectors is output as the precise information of the fruit tree canopy.

Images