Methods and devices for measuring fruit tree canopy structure information

Abstract

This invention discloses a method and apparatus for measuring fruit tree canopy structure information, belonging to the field of agricultural technology. The method includes: converting a drone aerial photograph into a corresponding binocular canopy foreground RGB image; performing region matching processing on the binocular canopy foreground RGB image based on block similarity, and calculating a binocular canopy height estimation image based on binocular disparity; using a generative adversarial network, generating a binocular canopy height image based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image; reconstructing the canopy structure based on the binocular canopy height image to obtain the three-dimensional canopy structure, and measuring the characteristic parameters of the fruit tree canopy structure. This method saves computation time in the height estimation process, improves the accuracy of canopy height prediction, and provides a reasonable distribution of calculated height values, resulting in a detailed canopy height image that satisfies the requirements for calculating canopy structure parameters.

Description

Technical Field

[0001] This invention relates to the field of agricultural technology, and in particular to a method and apparatus for measuring the canopy structure information of fruit trees. Background Technology

[0002] Among the various structures of fruit trees, the canopy is the first part exposed to sunlight and the external environment, and it is also the main site of photosynthesis and respiration. The structural information of the fruit tree canopy, such as its height, shape, structure, and volume, not only reflects the growth status and yield potential of the fruit tree, but also plays a crucial guiding role in decision-making for key production steps such as canopy pruning, irrigation, fertilization, and precision pesticide application. While traditional manual on-site sampling and measurement of fruit tree canopy structure played an important role for a period of time, with the expansion of fruit tree cultivation areas, traditional manual measurement methods have shown limitations in quickly and accurately measuring canopy parameters, leading to low production efficiency, waste of water, fertilizer, pesticides / herbicides, and other agricultural chemicals, as well as serious pollution. This method can no longer meet the needs of modern precision agriculture. Therefore, how to quickly and accurately obtain structural information of the fruit tree canopy at a lower cost has become a key research focus in current precision orchard management.

[0003] In recent years, with the expansion of fruit tree planting areas, the demand for large-area, high-throughput canopy data collection in orchard management has been increasing to improve orchard production efficiency. With the development of low-altitude drone technology in recent years, the application of drones in agriculture has become increasingly widespread. Drones have the advantages of low-altitude flight and rapid maneuverability, enabling them to acquire high-quality raw data in a short time, making it possible to accurately measure the canopy structure of fruit trees in large-area scenarios.

[0004] Currently, technologies exist for acquiring structural information about fruit tree canopies using drones equipped with LiDAR or RGB cameras. LiDAR is the most direct method for obtaining canopy structure information by scanning with a drone equipped with LiDAR, directly obtaining 3D point cloud data of the canopy. While LiDAR measurement of canopy structure offers high accuracy, the high cost of LiDAR, the dense canopy point cloud data, and the increased computational load during post-processing result in high data acquisition and reconstruction costs, making canopy measurement difficult in large-scale orchards. To reduce the cost of canopy structure data acquisition, many studies use drones equipped with RGB cameras to acquire image sequences of the fruit tree canopy, combining SFM (Structure From Motion) and MVS (Multi-View Stereo) to reconstruct the 3D structure. However, this requires processing a large number of photographic images, the reconstruction process is computationally intensive, and if the 3D reconstruction fails, numerous more images need to be captured for reconstruction, resulting in high data processing costs. It is evident that in practical UAV-based fruit tree canopy structure measurement, both lidar-based and multi-view reconstruction methods incur high data acquisition and processing costs, making it difficult to complete the task of efficiently and cost-effectively measuring fruit tree canopy structure information.

[0005] To achieve low-cost acquisition of the three-dimensional structure of fruit tree canopies, some researchers have used binocular cameras to measure the canopy structure. This method utilizes the principle of binocular parallax to calculate the vertical depth information of the canopy, thereby obtaining the three-dimensional structure of the canopy and greatly reducing the cost of data acquisition and processing. However, the accuracy of binocular measurement is affected by the distance between the binocular baseline and the distance to the object being measured. Since drones typically fly at high altitudes, it is difficult to use drones equipped with long-baseline binocular systems to measure canopy depth, which severely limits the distance and resolution of fruit tree canopy measurements. To improve the depth range and resolution of binocular measurement, Matsuura et al. (Matsuura, Y., Heming, Z., Nakao, K., Qiong, C., Firmansyah, I., Kawai, S., ... & Nobuhara, H. (2023). High-precision plant height measurement by drone with RTK-GNSS and single camera for real-time processing. Scientific Reports, 13(1), 6329.) used drones to take vertical overhead photographs, and used a high-precision RTK-GNSS positioning system to obtain the two drone photography positions, simulated a long-baseline binocular stereo model, and then used a binocular depth estimation method to measure the canopy height, obtaining the CHM (Canopy Height Map). That is, the grayscale image reflecting the canopy height distribution of fruit trees from the drone's overhead view can be considered as the CHM. This method improves the depth range and resolution of drone binocular photogrammetry. The aforementioned binocular canopy measurement methods all employ binocular depth estimation algorithms. These algorithms are based on the disparity of binocular images, and the accuracy of binocular disparity calculation depends on the accuracy of the binocular matching algorithm. However, actual UAV aerial images not only contain background information such as the ground, but also have complex foreground structures in the fruit tree canopy. Factors such as occlusion and changes in lighting can severely affect the accuracy of binocular stereo matching. This results in the CHM obtained by depth estimation using the aforementioned binocular methods often containing many holes and poor accuracy in estimating canopy height, thus limiting the application of binocular depth estimation algorithms in calculating three-dimensional canopy structure information. Summary of the Invention

[0006] In order to solve the problems existing in the prior art, the present invention provides the following technical solution.

[0007] The first aspect of this invention provides a method for measuring the canopy structure information of fruit trees, comprising:

[0008] S101 uses a drone equipped with an RGB camera to capture images or videos of fruit trees from above.

[0009] S102, Remove the background color from fruit tree images or videos and convert it into the corresponding binocular canopy foreground RGB image;

[0010] S103, perform corresponding region matching processing on the binocular canopy foreground RGB image based on block similarity, and calculate the binocular canopy height estimation image based on binocular disparity;

[0011] S104, using a generative adversarial network, a binocular canopy height image is generated based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image;

[0012] S105. The canopy structure is reconstructed based on binocular canopy height images to obtain the three-dimensional canopy structure, and the characteristic parameters of the fruit tree canopy structure are measured in the three-dimensional canopy structure.

[0013] Preferably, the process of matching corresponding regions in the binocular canopy foreground RGB image based on block similarity includes:

[0014] Obtain one block from the left and right canopy foreground RGB images, respectively.

[0015] The corresponding feature maps are obtained by downsampling the two blocks respectively;

[0016] Calculate the distance between two feature maps and determine the canopy structure similarity between the two blocks based on this distance. The smaller the distance between the two feature maps, the higher the canopy structure similarity between the two blocks.

[0017] Match the two blocks with the highest canopy structure similarity.

[0018] Preferably, the step of calculating the binocular canopy height estimation image based on binocular parallax includes:

[0019] Calculate the disparity between the center points of the two matched blocks in the binocular canopy foreground RGB image, and use the disparity to calculate the height value of the blocks;

[0020] Traverse all blocks to obtain the height values ​​of each block in the binocular canopy foreground RGB image, and then obtain the height matrix of the binocular canopy foreground RGB image. Remove outliers in the height matrix and record the maximum value in the height matrix. Normalize the height matrix to a grayscale image of 0-255 levels, and retain only the canopy foreground region in the grayscale image to obtain the binocular canopy height estimation image.

[0021] Preferably, the generative adversarial network includes a generator network and a discriminator network. During the downsampling process, the generator network uses consecutive convolutional modules to simultaneously extract the canopy foreground color features of the input binocular canopy foreground RGB image and the canopy height prediction features of the binocular canopy height estimation image and fuse them. During the upsampling process, the generator network uses deconvolutional modules to progressively reconstruct the image and uses a skip connection structure to skip-connect downsampling feature layers of the same scale.

[0022] Preferably, the generator network calculates pixel-wise loss and canopy height distribution loss during training; wherein, the canopy height distribution loss includes canopy height value quantity distribution loss and canopy height value position distribution loss;

[0023] The number of canopy height values ​​in the foreground region of the canopy is counted in the generated binocular canopy height image and the real binocular canopy height image. The difference between the two counts is used as the canopy height value distribution loss.

[0024] The average distance from each canopy height value to the canopy center in the foreground region of the generated binocular canopy height image and the real binocular canopy height image is used as the canopy height value location distribution loss.

[0025] Preferably, the canopy structure reconstruction based on binocular canopy height images includes:

[0026] The left and right three-dimensional point clouds of the corresponding fruit tree canopy were obtained using binocular canopy height images.

[0027] The left and right 3D point clouds are aligned and merged in 3D space, and the merged point cloud is smoothed to obtain the final canopy 3D point cloud.

[0028] Preferably, removing the background color from the fruit tree image or video and converting it into a corresponding binocular canopy foreground RGB image includes:

[0029] Extract adjacent frames from an image or video;

[0030] Remove the background color of the fruit trees from the extracted adjacent frame images;

[0031] The adjacent frame images are converted into standard stereo images to obtain stereo foreground RGB images of the canopy.

[0032] A second aspect of the present invention provides a device for measuring fruit tree canopy structure information, comprising:

[0033] The data acquisition module is used to obtain images or videos of fruit trees by taking aerial photos with an RGB camera mounted on a drone;

[0034] The image preprocessing module is used to remove the background color from fruit tree images or videos and convert them into corresponding binocular canopy foreground RGB images;

[0035] The binocular canopy height estimation image generation module is used to perform corresponding region matching processing on the binocular canopy foreground RGB image based on block similarity, and to calculate the binocular canopy height estimation image based on binocular disparity;

[0036] A binocular canopy height image generation module is used to generate a binocular canopy height image by utilizing a generative adversarial network based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image.

[0037] The canopy 3D structure reconstruction module is used to reconstruct the canopy structure based on binocular canopy height images to obtain the canopy 3D structure, and to measure the characteristic parameters of the fruit tree canopy structure in the canopy 3D structure.

[0038] The present invention also provides a memory that stores multiple instructions for implementing the fruit tree canopy structure information measurement method as described in the first aspect.

[0039] The present invention also provides an electronic device, including a processor and a memory connected to the processor, the memory storing a plurality of instructions which can be loaded and executed by the processor to enable the processor to perform the fruit tree canopy structure information measurement method as described in the first aspect.

[0040] The beneficial effects of this invention are as follows: The fruit tree canopy structure information measurement method and device provided by this invention utilizes single-lens photography from a drone to simulate binocular imaging. It learns the true canopy height distribution characteristics from the RGB color information of the fruit tree canopy extracted from the foreground RGB image and the height information extracted from the canopy height prediction image, thereby achieving high-precision canopy height calculation. Furthermore, it employs a canopy height prediction method based on block structure similarity. This method uses region similarity as a metric for feature matching, effectively shortening the feature matching time for binocular images and improving the accuracy of canopy height prediction. Secondly, during the canopy height calculation process, this invention uses a canopy height distribution supervision mechanism to judge the rationality of the height value distribution in the calculated canopy height image, ensuring that the final calculated canopy height image is detailed and has a reasonable height distribution, thus meeting the requirements for subsequent canopy structure parameter calculations. Therefore, the technical solution provided by this invention fully considers the needs of practical application scenarios, offering a new possibility for precision orchard management. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the process for measuring fruit tree canopy structure information according to the present invention;

[0042] Figure 2 This is a schematic diagram of the generation process of the binocular CFM-RGB described in this invention;

[0043] Figure 3 This is a schematic diagram illustrating that the same region of the canopy has high similarity between binocular images, while different regions have low similarity between binocular images, as described in this invention.

[0044] Figure 4 This is a flowchart illustrating the canopy height estimation method based on block structure similarity according to the present invention, where (a) represents the binocular block similarity calculation process; and (b) represents the calculation process of canopy height h at the Block.

[0045] Figure 5 In the image, (a) is CFM-RGB, (b) is the Block segmentation image of the canopy foreground region, and (c) is CHM-estimated.

[0046] Figure 6 This is a schematic diagram illustrating the training and inference process of the generative adversarial network described in this invention;

[0047] Figure 7 The images and details of CHM images before and after adding QD_loss to the generator's loss function under the same training rounds are shown in the figure below. (a) is the CHM generated without adding QD_loss, (b) is the CHM generated after adding QD_loss, and (c) is the real CHM.

[0048] Figure 8 This is a schematic diagram of the functional structure of the fruit tree canopy structure information measuring device of the present invention. Detailed Implementation

[0049] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment of the invention provides a method for measuring fruit tree canopy structure information, including:

[0052] S101 uses a drone equipped with an RGB camera to capture images or videos of fruit trees from above.

[0053] S102, Remove the background color from fruit tree images or videos and convert it into the corresponding binocular canopy foreground RGB image;

[0054] S103, perform corresponding region matching processing on the binocular canopy foreground RGB image based on block similarity, and calculate the binocular canopy height estimation image based on binocular disparity;

[0055] S104, using a generative adversarial network, a binocular canopy height image is generated based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image;

[0056] S105. The canopy structure is reconstructed based on binocular canopy height images to obtain the three-dimensional canopy structure, and the characteristic parameters of the fruit tree canopy structure are measured in the three-dimensional canopy structure.

[0057] The method described in this invention is proposed based on existing technologies and considering the needs of practical application scenarios. It is a high-precision, fast, and low-cost method for measuring fruit tree canopy structure information, providing a new possibility for precise orchard management. This invention utilizes single-lens drone photography to simulate binocular imaging, enabling the rapid acquisition of detailed CHM (binocular canopy height image) to reconstruct the three-dimensional structure of the fruit tree canopy. The main innovation of this method lies in the fact that, based on the characteristics of fruit tree canopy exhibiting different color representations at different heights and locations in actual drone overhead images due to factors such as lighting, fruit tree canopy structure, texture, and drone photography angle, this invention proposes a canopy height calculation network based on binocular images. First, the drone single-lens overhead image is converted into a binocular image. Then, the RGB color information of the fruit tree canopy is extracted from the binocular canopy foreground RGB image (CFM-RGB), and the height distribution characteristics of the real canopy are learned from the height information of the binocular canopy height estimation image (CHM-estimated), thereby achieving high-precision CHM calculation. In this process, the present invention also proposes a canopy height prediction method based on block structure similarity. This method uses region similarity as a metric for feature matching, which effectively shortens the feature matching time of binocular images and improves the accuracy of canopy height prediction. Secondly, in the process of canopy height calculation, the present invention also proposes for the first time to adopt a canopy height distribution supervision mechanism to judge the rationality of the height value distribution in the calculated CHM, so that the final calculated CHM is refined and the height distribution is reasonable, thereby satisfying the subsequent calculation of canopy structure parameters.

[0058] In step S101, an overhead shot can be taken using the onboard RGB camera (1-inch CMOS, 84-degree FOV, 8.8mm focal length, and 5472×3648 pixels) on a low-cost industrial drone (DJIPhantom 4RTK, DJI, Shenzhen, China). The drone can fly at an altitude of 30 meters (GSD: 0.82cm / pixel) to obtain overhead images or videos of the fruit trees.

[0059] In step S102, considering the presence of ground background information in the aerial images taken by the drone in the orchard, this invention, to reduce the impact of ground background information on the canopy height prediction process, firstly removes the background information from the aerial images or videos of the fruit trees and generates a binocular CFM-RGB (Canopy Foreground RGB Map) of the same fruit tree object to facilitate subsequent canopy height prediction. It should be noted that the binocular CFM-RGB includes left-eye CFM-RGB and right-eye CFM-RGB. The specific operation process is as follows... Figure 2 As shown, it includes the following steps:

[0060] 1. Adjacent frame extraction: Extracting adjacent frame images from drone overhead video or image sequences;

[0061] 2. Simulated Binocular Transformation: Since the drone's flight path may not be parallel to the CMOS sensor of the camera lens, it is necessary to convert adjacent frame images into a standard binocular image format. This invention can adopt any one of the following two transformation methods to simulate standard binocular photography: a. Known camera lens attitude: The attitude of the camera lens during drone photography can be obtained from the drone's aerial photography log. By rotating adjacent frame images according to the horizontal rotation angle α of the camera lens and cropping out redundant boundaries, a standard binocular image format can be obtained; b. Unknown camera lens attitude: When the attitude of the drone's camera lens is unknown, the rotation angle α can be calculated according to the binocular feature point matching method. By rotating adjacent frame images according to the rotation angle α and cropping out redundant boundaries, a standard binocular image format can be obtained.

[0062] 3. Target canopy matching: High-dimensional feature template matching is used to match the same canopy object in the stereo image, which can determine the region of the same canopy object in the stereo image;

[0063] 4. Matching Region Segmentation: Segmenting matching regions of the same fruit tree canopy in the binocular image;

[0064] 5. Foreground Canopy Segmentation: To reduce the impact of the ground background area on the subsequent fruit tree canopy height prediction process, a foreground segmentation network (such as the U2net network) is used to separate the foreground and background of the fruit tree canopy.

[0065] Step S103 involves first performing corresponding region matching processing on the binocular canopy foreground RGB images based on block similarity. This may include the following steps: acquiring a block from the left and right canopy foreground RGB images respectively; downsampling the two blocks to obtain corresponding feature maps; calculating the distance between the two feature maps and determining the canopy structure similarity between the two blocks based on this distance. The smaller the distance between the two feature maps, the higher the canopy structure similarity between the two blocks; and matching the two blocks with the highest canopy structure similarity. Then, the binocular canopy height estimation image is obtained based on the binocular disparity, which may include the following: calculating the disparity between the center points of the two matched blocks in the binocular canopy foreground RGB image, and using the disparity to calculate the height value of the block; traversing all blocks to obtain the height value of each block in the binocular canopy foreground RGB image, thereby obtaining the height matrix of the binocular canopy foreground RGB image, removing outliers in the height matrix and recording the maximum value in the height matrix, normalizing the height matrix to a grayscale image of 0-255 levels, and retaining only the canopy foreground region in the grayscale image to obtain the binocular canopy height estimation image.

[0066] In step S103, since the calculation effect of canopy height is strongly correlated with the quality of the input canopy height estimation image, improving the speed and accuracy of canopy height estimation will further improve the accuracy of CHM. However, common binocular depth estimation algorithms are usually based on pixel matching. These algorithms typically require generating a large number of feature points for subsequent matching, which reduces the efficiency of canopy height estimation. At the same time, these algorithms are sensitive to changes in texture and lighting conditions, and will produce more false matches in the foreground of complex fruit tree canopies, affecting the accuracy of canopy height estimation.

[0067] In binocular images composed of long baselines, although the texture details of the fruit tree canopy are quite complex, the same region in the canopy shows high structural similarity between binocular images (e.g., Figure 3 As shown, a block in the left canopy foreground image and a block in the right canopy foreground image belong to the same region, thus exhibiting high similarity. Conversely, a block in the left canopy foreground image and another block in the right canopy foreground image belong to different regions, thus exhibiting low similarity. Therefore, the high-dimensional information similarity between binocular canopy structures can be used as a metric for binocular matching. This leads to a proposed canopy height prediction method based on block structure similarity. The core idea is to utilize a downsampling strategy to extract high-dimensional structural information within binocular canopy blocks, thereby comparing structural similarity. Specifically, the following method can be employed:

[0068] In the process of canopy block matching, the accuracy of the similarity between blocks directly affects the calculation of subsequent block height values. To better utilize the structural features of canopy blocks to calculate the similarity between two blocks, this method first downsamples the images of the two blocks to obtain high-dimensional feature maps of the two blocks before calculating the block similarity. Then, it uses these high-dimensional feature maps to calculate the similarity (the calculation process is as follows). Figure 4 As shown in (a) and Table 1). This can increase the contribution of canopy structure features to the block structure similarity and reduce the influence of canopy details on the block structure similarity. Then, the distance between the high-dimensional feature maps of the two blocks (for example, cosine distance can be used) is calculated, and this distance is used as a measure of the canopy structure similarity between the blocks. The smaller the cosine distance between the two high-dimensional feature maps, the higher the canopy structure similarity between them. The formula for calculating the cosine distance is shown in Equation (1):

[0069]

[0070] Where ML is the high-dimensional feature map of the left canopy block for which similarity is to be calculated, and MR is the high-dimensional feature map of the right canopy block for which similarity is to be calculated. m*n The pixel matrix of the high-dimensional feature map of the left canopy region, MR m*n Let ML be the pixel matrix of the high-dimensional feature map of the right canopy region, where m and n are the width and height of the pixel matrix, respectively, and i and j are the x and y coordinates of the pixel matrix, respectively. i,j MR represents the pixel value at position i,j in the high-dimensional feature map of the left canopy region. i,j d represents the pixel value at position i,j in the high-dimensional feature map of the right canopy region. cos (ML,MR) is the cosine distance between ML and MR.

[0071] Therefore, the complete calculation process of similarity for binocular canopy block matching can be shown in Table 1.

[0072] Table 1

[0073]

[0074]

[0075] Based on the aforementioned canopy block structure similarity measurement method, this invention, in the binocular block matching process, first divides the canopy foreground region of one eye image into blocks (called Blocks), then searches for the block in the other eye image that is most similar to the canopy structure of each block as the matching block (called the Matched block), and calculates the center point of the block (e.g., ...). Figure 4 (as shown in the (xb, yb) coordinates) and the center point of the matching block (e.g. Figure 4The disparity between the two eyes (shown in the (xm, ym) coordinates) is used to calculate the height value of the block using the principle of stereo vision (e.g., Figure 4 As shown in (b)). Since the standard binocular form correction has been completed in step S102, the process of finding matching blocks now only occurs in the area at the upper and lower edges of the Block region (e.g., Figure 4 You can do it within the Bar area shown in the image.

[0076] The system iterates through all blocks in the left-eye CFM-RGB region, uses the matching method described above to find the best matching region and calculates its height value, thus obtaining the left-eye canopy foreground height matrix M. Outliers in M ​​are removed, and the maximum value h in M ​​is recorded. max The matrix M is normalized to a grayscale image of 0-255 levels. Only the canopy foreground region in the grayscale image is retained to obtain the left target CHM-estimated (e.g., Figure 5 (As shown in Figure (c)). By swapping the CFM-RGB values ​​of the left and right eyes and reusing the canopy height estimation method based on block structure similarity, the CHM-estimated value for the right eye is obtained. Figure 5 Figure (a) shows CFM-RGB. Figure 5 Figure (b) shows a block-division image of the canopy foreground region. Figure 5 Figure (c) shows the CHM-estimated. Observing the CHM-estimated, the height value of each block region in the image is the overall height of the canopy in that region. The CHM-estimated is not only the input object for subsequent canopy height calculation, but also provides preliminary canopy height values ​​as guidance for the canopy height calculation process, enabling the canopy height calculation network to more accurately calculate the height of various parts of the canopy by combining the RGB information of the foreground. At the same time, each gray value in the CHM-estimated is also a record of the relative height of various parts of the canopy, providing a vertical mapping ratio for subsequent canopy reconstruction work, so that the reconstructed canopy has a reasonable three-dimensional scale.

[0077] In step S104, the generative adversarial network includes a generator network and a discriminator network. During downsampling, the generator network uses consecutive convolutional modules to simultaneously extract and fuse the canopy foreground color features of the input binocular canopy foreground RGB image and the canopy height prediction features of the binocular canopy height estimation image. During upsampling, the generator network uses deconvolutional modules to progressively reconstruct the image and uses a skip connection structure to skip-connect downsampling feature layers of the same scale. During training, the generator network calculates pixel-wise loss and canopy height distribution loss. The canopy height distribution loss includes canopy height value quantity distribution loss and canopy height value position distribution loss. The number of canopy height values ​​in the canopy foreground region of the generated binocular canopy height image and the real binocular canopy height image are counted, and the difference between the two counts is used as the canopy height value quantity distribution loss. The average distance from each canopy height value to the canopy center in the canopy foreground region of the generated binocular canopy height image and the real binocular canopy height image is counted, and the difference between the two averages is used as the canopy height value position distribution loss.

[0078] Since the three-dimensional structure of the fruit tree canopy needs to be reconstructed from the calculated CHM (Content Structure Model), the calculation of canopy height requires not only generating a reasonable canopy height distribution but also ensuring the invariance of the outline shape of the canopy foreground. Based on this consideration, this invention proposes a fruit tree canopy foreground height calculation network based on the correlation between canopy color and height distribution. The training and inference flowcharts of this network model are shown below. Figure 6 As shown, a generative adversarial network (GAN) structure is used. The "dual-input image generator network" is used to calculate the canopy height, accepting CFM-RGB and CHM-estimated as dual inputs. CFM-RGB provides the RGB color information of the canopy foreground, while CHM-estimated provides the estimated canopy height values ​​in each region. The generator network combines the canopy foreground color from CFM-RGB and the canopy height estimates from CHM-estimated to learn a deep mapping relationship between the canopy RGB information, the estimated height information, and the actual canopy height, generating a CHM image with reasonable canopy height distribution characteristics, and outputting the CHM. The canopy height calculation process uses the canopy height distribution loss function proposed in this invention to accelerate the convergence speed of the generator network and improve the reasonableness of the height distribution of the generated CHM; simultaneously, a pixel-wise loss function is used to ensure the invariance of the two-dimensional contour structure of the calculated CHM. Furthermore, the "discriminator network" learns the ability to distinguish between the generated CHM and the real CHM; this process uses adversarial loss to guide the learning of the "discriminator." The ability of the "generator network with dual input images" to calculate canopy height is improved during the adversarial training process between the generator and the discriminator. Such a network structure meets the needs of fruit tree canopy height calculation.

[0079] In this invention, the generator network can be an encoding / decoding network or a feature extraction and reconstruction network. For example, the generator network can be based on U-Net. During the downsampling process, U-Net uses consecutive convolutional layers to simultaneously extract the input CFM-RGB and CHM-estimated features. During downsampling, it fuses the RGB information and height information of the canopy to obtain high-dimensional features rich in canopy color and height information. During the upsampling process, it uses deconvolutional modules to gradually reconstruct the image. During this process, skip connections are used to connect downsampled feature layers of the same scale, enabling the image restoration process to fuse canopy hue, contour, and other feature information at different scales, thereby improving the image reconstruction quality.

[0080] In this embodiment of the invention, the generator uses a pixel-wise loss to ensure the invariance of the two-dimensional contour structure of the generated CHM. However, in actual training, using only this loss will result in the network lacking learning of the characteristics of the canopy height distribution, leading to problems such as slow convergence speed and unreasonable canopy height distribution generated by the generator network. Therefore, this invention proposes for the first time a canopy height distribution loss (QD_loss, Quantity and Distribution loss), which can evaluate the similarity of the height value distribution between the generated CHM and the real CHM, making the canopy height distribution generated by the generator network more reasonable. QD Loss from the number distribution of canopy height values quantity and the location distribution loss of canopy height values distribution The composition, as defined in Formula 2, is as follows:

[0081] loss QD =loss quantity +loss distribution (2)

[0082] One method to analyze the loss is to count the number of height values ​​in the canopy region. quantity The calculation is performed. Specifically, the number of canopy height values ​​in the foreground region of the canopy and the real CHM are counted, and the difference between the two is used as the loss, so that the network can learn the difference between the distribution of canopy height values.

[0083] The loss can be calculated by statistically analyzing the distribution distances of each height value in the canopy region to the canopy center. distribution The calculation is performed. Specifically, the average distance from each height value of the canopy to the center of the canopy is calculated between the generated CHM and the real CHM in the foreground region of the canopy. The difference between the two is used as the loss, which allows the network to learn the differences in the location distribution of canopy height values.

[0084] Loss analysis of canopy height value quantity distribution. quantity and the location distribution loss of canopy height values distribution Before fusion, this invention first divides the loss quantity and loss distribution Perform decimal scaling and normalization to the range [0,1] to ensure that the losses of both are on the same scale. Figure 7 The comparison shows the effect of adding QD_loss to the generator's loss function before and after the same number of training epochs. Figure 7 In the figure, (a) shows the CHM generation effect without QD_loss, (b) shows the CHM generation effect with QD_loss, and (c) shows the actual CHM.

[0085] Compared to Figure 7 Figure (a) in the middle, Figure 7 Image (b) shows a clear height difference within the canopy, and is closer to the true CHM in canopy details. It is evident that, under the same number of training iterations, Figure 7 In Figure (b), the addition of QD loss to the network effectively improves the quality of CHM generation, making the generated CHM image more consistent with... Figure 7 The image shown in Figure (c) is more similar to the real CHM image and accelerates the convergence process of the network.

[0086] Step S105 involves reconstructing the canopy structure based on the binocular canopy height image to obtain a three-dimensional canopy structure, and measuring the characteristic parameters of the fruit tree canopy structure within the three-dimensional canopy structure. The canopy structure reconstruction based on the binocular canopy height image may include: obtaining the left-eye and right-eye three-dimensional point clouds of the corresponding fruit tree canopy using the binocular canopy height image; aligning and merging the left-eye and right-eye three-dimensional point clouds in three-dimensional space; and smoothing the merged point cloud to obtain the final three-dimensional canopy point cloud.

[0087] Specifically, based on the maximum value h in the height matrix recorded during the canopy height prediction process. max The actual height corresponding to each gray level in the calculated left and right eye CHMs is determined, and the canopy structure is restored separately to obtain the left eye canopy point cloud pc_l and the right eye canopy point cloud pc_r. The two point clouds are aligned and merged in three-dimensional space, and the merged point cloud is smoothed (the point clouds at the same horizontal position are subjected to z-direction mean filtering) to obtain the final canopy three-dimensional point cloud pc.

[0088] For the task of measuring fruit tree canopy structure information, the method provided in this invention uses a drone equipped with a monocular camera to simulate a binocular system, which can quickly obtain high-accuracy three-dimensional structure of the fruit tree canopy. Based on this, the three-dimensional feature parameters of the canopy structure are calculated, effectively reducing the cost of fruit tree canopy data acquisition and processing. This method is suitable for high-throughput, accurate measurement of the canopy structure of individual fruit trees in large-scale orchard settings, assisting in the analysis of the growth status of the fruit tree canopy. In future work, the ideas of this invention can be applied to phenotypic measurement tasks of other crops, such as field crops and low shrubs, expanding the application scope of this invention in the agricultural field.

[0089] Example 2

[0090] like Figure 8 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow. That is, the embodiments of the present invention also provide a fruit tree canopy structure information measurement device, including:

[0091] Data acquisition module 201 is used to obtain images or videos of fruit trees by taking aerial photos with an RGB camera mounted on a drone;

[0092] Image preprocessing module 202 is used to remove the background color from fruit tree images or videos and convert them into corresponding binocular canopy foreground RGB images;

[0093] The binocular canopy height estimation image generation module 203 is used to perform corresponding region matching processing on the binocular canopy foreground RGB image based on block similarity, and to calculate the binocular canopy height estimation image based on binocular disparity.

[0094] The binocular canopy height image generation module 204 is used to generate a binocular canopy height image by using a generative adversarial network based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image.

[0095] The canopy 3D structure reconstruction module 205 is used to reconstruct the canopy structure based on binocular canopy height images to obtain the canopy 3D structure, and to measure the characteristic parameters of the fruit tree canopy structure in the canopy 3D structure.

[0096] In the binocular canopy height estimation image generation module, the matching process of corresponding regions in the binocular canopy foreground RGB image based on block similarity includes:

[0097] Obtain one block from the left and right canopy foreground RGB images, respectively.

[0098] The corresponding feature maps are obtained by downsampling the two blocks respectively;

[0099] Calculate the distance between two feature maps and determine the canopy structure similarity between the two blocks based on this distance. The smaller the distance between the two feature maps, the higher the canopy structure similarity between the two blocks.

[0100] Match the two blocks with the highest canopy structure similarity.

[0101] Furthermore, the step of calculating the binocular canopy height estimation image based on binocular parallax includes:

[0102] Calculate the disparity between the center points of the two matched blocks in the binocular canopy foreground RGB image, and use the disparity to calculate the height value of the blocks;

[0103] Traverse all blocks to obtain the height values ​​of each block in the binocular canopy foreground RGB image, and then obtain the height matrix of the binocular canopy foreground RGB image. Remove outliers in the height matrix and record the maximum value in the height matrix. Normalize the height matrix to a grayscale image of 0-255 levels, and retain only the canopy foreground region in the grayscale image to obtain the binocular canopy height estimation image.

[0104] In addition, in the binocular canopy height image generation module, the generative adversarial network may include a generator network and a discriminator network. During the downsampling process, the generator network uses consecutive convolutional modules to simultaneously extract the canopy foreground color features of the input binocular canopy foreground RGB image and the canopy height prediction features of the binocular canopy height estimation image and fuse them. During the upsampling process, the generator network uses deconvolutional modules to progressively reconstruct the image and uses a skip connection structure to skip-connect downsampling feature layers of the same scale.

[0105] Furthermore, the generator network calculates pixel-wise loss and canopy height distribution loss during training; wherein, the canopy height distribution loss includes canopy height value quantity distribution loss and canopy height value position distribution loss; the number of canopy height values ​​in the canopy foreground region of the generated binocular canopy height image and the real binocular canopy height image are counted, and the difference between the two counts is used as the canopy height value quantity distribution loss; the average distance from each canopy height value in the canopy foreground region of the generated binocular canopy height image and the real binocular canopy height image to the canopy center is counted, and the difference between the two averages is used as the canopy height value position distribution loss.

[0106] Furthermore, in the canopy three-dimensional structure reconstruction module, the canopy structure reconstruction based on the binocular canopy height image includes:

[0107] The left and right three-dimensional point clouds of the fruit tree canopy are obtained using binocular canopy height images. The left and right three-dimensional point clouds are aligned and merged in three-dimensional space, and the merged point cloud is smoothed to obtain the final canopy three-dimensional point cloud.

[0108] Furthermore, in the image preprocessing module, removing the background color from the fruit tree image or video and converting it into a corresponding binocular canopy foreground RGB image includes:

[0109] Extract adjacent frames from an image or video;

[0110] Remove the background color of the fruit trees from the extracted adjacent frame images;

[0111] The adjacent frame images are converted into standard stereo images to obtain stereo foreground RGB images of the canopy.

[0112] This device can be implemented using the fruit tree canopy structure information measurement method provided in Embodiment 1 above. For the specific implementation method, please refer to the description in Embodiment 1, which will not be repeated here.

[0113] The present invention also provides a memory that stores multiple instructions for implementing the fruit tree canopy structure information measurement method as described in Embodiment 1.

[0114] The present invention also provides an electronic device, including a processor and a memory connected to the processor, the memory storing a plurality of instructions which can be loaded and executed by the processor to enable the processor to perform the fruit tree canopy structure information measurement method as described in Embodiment 1.

[0115] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for measuring fruit tree canopy structure information based on a single camera from a drone, characterized in that, include: S101 uses a drone equipped with an RGB camera to capture images or videos of fruit trees from above. S102, Remove the background color from fruit tree images or videos and convert it into the corresponding binocular canopy foreground RGB image; S103, perform corresponding region matching processing on the binocular canopy foreground RGB image based on block similarity, and calculate the binocular canopy height estimation image based on binocular disparity; S104, using a generative adversarial network, a binocular canopy height image is generated based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image; S105, the canopy structure is reconstructed based on binocular canopy height images to obtain the three-dimensional canopy structure, and the characteristic parameters of the fruit tree canopy structure are measured in the three-dimensional canopy structure; The step of matching corresponding regions in the binocular canopy foreground RGB image based on block similarity includes: Obtain one block from the left and right canopy foreground RGB images, respectively. The corresponding feature maps are obtained by downsampling the two blocks respectively; Calculate the cosine distance between two feature maps and determine the canopy structure similarity between the two blocks based on this distance. The smaller the cosine distance between the two feature maps, the higher the canopy structure similarity between the two blocks. Match the two blocks with the highest canopy structure similarity; The generative adversarial network includes a generator network and a discriminator network. During the downsampling process, the generator network uses consecutive convolutional modules to simultaneously extract the canopy foreground color features of the input binocular canopy foreground RGB image and the canopy height prediction features of the binocular canopy height estimation image and fuse them. During the upsampling process, the generator network uses deconvolutional modules to progressively reconstruct the image and uses a skip connection structure to skip-connect downsampling feature layers of the same scale. The generator network calculates pixel-wise loss and canopy height distribution loss during training; wherein, the canopy height distribution loss includes canopy height value quantity distribution loss and canopy height value position distribution loss; The number of canopy height values ​​in the foreground region of the canopy is counted in the generated binocular canopy height image and the real binocular canopy height image. The difference between the two counts is used as the canopy height value distribution loss. The average distance from each canopy height value to the canopy center in the foreground region of the generated binocular canopy height image and the real binocular canopy height image is used as the canopy height value location distribution loss.

2. The method for measuring fruit tree canopy structure information based on a single UAV camera as described in claim 1, characterized in that, The binocular canopy height estimation image calculated based on binocular parallax includes: Calculate the disparity between the center points of the two matched blocks in the binocular canopy foreground RGB image, and use the disparity to calculate the height value of the blocks; Traverse all blocks to obtain the height values ​​of each block in the binocular canopy foreground RGB image, and then obtain the height matrix of the binocular canopy foreground RGB image. Remove outliers in the height matrix and record the maximum value in the height matrix. Normalize the height matrix to a grayscale image of 0-255 levels, and retain only the canopy foreground region in the grayscale image to obtain the binocular canopy height estimation image.

3. The method for measuring fruit tree canopy structure information based on a single UAV camera as described in claim 1, characterized in that, The canopy structure reconstruction based on binocular canopy height images includes: The left and right three-dimensional point clouds of the corresponding fruit tree canopy were obtained using binocular canopy height images. The left and right 3D point clouds are aligned and merged in 3D space, and the merged point cloud is smoothed to obtain the final canopy 3D point cloud.

4. The method for measuring fruit tree canopy structure information based on a single UAV camera as described in claim 1, characterized in that, The process of removing the background color from fruit tree images or videos and converting them into corresponding binocular canopy foreground RGB images includes: Extract adjacent frames from fruit tree images or videos; Remove the background color of the fruit trees from the extracted adjacent frame images; The adjacent frame images are converted into standard stereo images to obtain stereo foreground RGB images of the canopy.

5. A device for measuring fruit tree canopy structure information based on a single camera from a drone, characterized in that, include: The data acquisition module is used to obtain images or videos of fruit trees by taking aerial photos with an RGB camera mounted on a drone; The image preprocessing module is used to remove the background color from fruit tree images or videos and convert them into corresponding binocular canopy foreground RGB images; The binocular canopy height estimation image generation module is used to perform corresponding region matching processing on the binocular canopy foreground RGB image based on block similarity, and to calculate the binocular canopy height estimation image based on binocular disparity; A binocular canopy height image generation module is used to generate a binocular canopy height image by utilizing a generative adversarial network based on the canopy foreground color information of the binocular canopy foreground RGB image and the canopy height prediction information of the binocular canopy height estimation image. The canopy 3D structure reconstruction module is used to reconstruct the canopy structure based on binocular canopy height images to obtain the canopy 3D structure, and to measure the characteristic parameters of the fruit tree canopy structure in the canopy 3D structure; The step of matching corresponding regions in the binocular canopy foreground RGB image based on block similarity includes: Obtain one block from the left and right canopy foreground RGB images, respectively. The corresponding feature maps are obtained by downsampling the two blocks respectively; Calculate the cosine distance between two feature maps and determine the canopy structure similarity between the two blocks based on this distance. The smaller the cosine distance between the two feature maps, the higher the canopy structure similarity between the two blocks. Match the two blocks with the highest canopy structure similarity; The generative adversarial network includes a generator network and a discriminator network. During the downsampling process, the generator network uses consecutive convolutional modules to simultaneously extract the canopy foreground color features of the input binocular canopy foreground RGB image and the canopy height prediction features of the binocular canopy height estimation image and fuse them. During the upsampling process, the generator network uses deconvolutional modules to progressively reconstruct the image and uses a skip connection structure to skip-connect downsampling feature layers of the same scale. The generator network calculates pixel-wise loss and canopy height distribution loss during training; wherein, the canopy height distribution loss includes canopy height value quantity distribution loss and canopy height value position distribution loss; The number of canopy height values ​​in the foreground region of the canopy is counted in the generated binocular canopy height image and the real binocular canopy height image. The difference between the two counts is used as the canopy height value distribution loss. The average distance from each canopy height value to the canopy center in the foreground region of the generated binocular canopy height image and the real binocular canopy height image is used as the canopy height value location distribution loss.

6. A memory, characterized in that, The system stores multiple instructions for implementing the method for measuring fruit tree canopy structure information based on a single UAV camera, as described in any one of claims 1-4.

7. An electronic device, characterized in that, The device includes a processor and a memory connected to the processor. The memory stores multiple instructions that can be loaded and executed by the processor to enable the processor to perform the method for measuring fruit tree canopy structure information based on a single UAV lens as described in any one of claims 1-4.

Images