Method and device for extracting fruit tree canopy

Abstract

The present invention provides a method and device for extracting a fruit tree canopy, and relates to the field of image processing technology. The method includes: performing image preprocessing based on drone aerial images to generate a digital orthophoto and a canopy height model image of the fruit tree; cropping the digital orthophoto and the canopy height model image respectively; using the SAM image segmentation model to fully segment the cropped digital orthophoto to obtain an image pre-segmentation result; based on prior knowledge of fruit trees, the cropped digital orthophoto, and the canopy height model image, screening the image spots in the image pre-segmentation result to obtain partial image spots, generating segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial image spots, and interactively segmenting the cropped digital orthophoto based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy. Based on the technical solution provided by the embodiment of the present invention, the applicability of fruit tree canopy extraction can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for extracting a fruit tree canopy. Background Art

[0002] At present, accurately obtaining the canopy distribution of fruit trees is not only an important basis for tree growth monitoring, yield estimation, agricultural financial insurance application and disaster assessment, but also the basis for macro-control of fruit tree growth and regulation of fruit trade.

[0003] In recent years, UAV low-altitude remote sensing technology and sensing technology have been widely used in the fields of crop growth, pest and disease detection, etc., and can provide new technical means and data basis for the precise management of tree canopies.

[0004] In related technologies, most fruit tree canopy extraction methods are highly dependent on sample quantity and quality, requiring precise delineation and annotation of tree crown contours as training sample sets, and the training process requires a large data set. Fruit tree canopy extraction methods are mostly customized for specific experimental scenarios and specific types of fruit trees in terms of feature selection, feature fusion, and threshold parameter setting. The extraction model is often not applicable to different fruit tree types and different regions, and the tree crown contours need to be redrawn as a sample data set. Therefore, the applicability of the fruit tree canopy extraction methods in related technologies is poor. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and device for extracting the fruit tree canopy in order to solve the problem of poor applicability of the fruit tree canopy extraction method in the prior art.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A first aspect of an embodiment of the present invention provides a method for extracting a fruit tree canopy, which performs image preprocessing based on drone aerial images to generate a digital orthophoto and a canopy height model image of the fruit tree; the digital orthophoto and the canopy height model image are cropped respectively; the cropped digital orthophoto is fully segmented using a SAM (Segment Anything Model) to obtain an image pre-segmentation result; based on prior knowledge of fruit trees, the cropped digital orthophoto and the cropped canopy height model image, the spots in the image pre-segmentation result are screened to obtain partial spots, segmentation prompt information for interactive segmentation of the fruit tree canopy is generated based on the partial spots, and the cropped digital orthophoto is interactively segmented based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy; wherein the prior knowledge of fruit trees includes spatial characteristics, spectral characteristics and morphological characteristics of fruit trees.

[0008] Optionally, image preprocessing is performed based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees, including: generating digital orthophotos, digital surface model images and digital terrain model images based on drone aerial image stitching and processing software and drone aerial images; unifying the resolution of digital surface model images and digital terrain model images; generating a canopy height model image based on the difference in grayscale values ​​between the first image and the second image; wherein the first image is a digital surface model image after unified resolution, and the second image is a digital terrain model image after unified resolution.

[0009] Optionally, the digital orthophoto image and the canopy height model image are cropped separately, including: aligning the coordinate systems and spatial positions of the digital orthophoto image and the canopy height model image; cropping the aligned digital orthophoto image and the canopy height model image into multiple small-size images; wherein the cropped digital orthophoto image and the canopy height model image of the same scene have the same geographical location and have overlapping areas.

[0010] Optionally, the spatial feature indicates that the height distribution range of the fruit tree is a first height distribution range; based on prior knowledge of the fruit tree, the cropped digital orthophoto and the cropped canopy height model image, the spots in the image pre-segmentation result are filtered to obtain partial spots, and segmentation prompt information for interactive segmentation of the fruit tree canopy is generated based on the partial spots, including: performing local maximum point detection based on the cropped canopy height model image, taking the local maximum point as the crown vertex, and taking the pixel value at the crown vertex as the height value of the tree; filtering the crown vertices whose height values ​​are within the first height distribution range, and counting the pixel values ​​at each filtered crown vertex to obtain a height distribution histogram of the tree; based on the height distribution histogram, calculating the mean height and height standard deviation of the fruit tree to determine a second height distribution range of the fruit tree.

[0011] Optionally, the spectral characteristics indicate that the target vegetation index grayscale value of the fruit tree is in a first grayscale range, and the target vegetation index is a green leaf index; based on prior knowledge of the fruit tree, the cropped digital orthophoto and the cropped canopy height model image, the spots in the image pre-segmentation result are screened to obtain partial spots, and segmentation prompt information for interactive segmentation of the fruit tree canopy is generated based on the partial spots, including: based on the bands of the cropped digital orthophoto, determining the vegetation index grayscale image of the target vegetation index; based on the vegetation index grayscale image, obtaining the vegetation index grayscale value at the crown vertex filtered based on the second height distribution range; screening the crown vertices whose vegetation index grayscale values ​​are within the first grayscale range, and counting the vegetation index grayscale values ​​at each screened crown vertex to obtain a vegetation index distribution histogram of the fruit tree; based on the vegetation index distribution histogram, calculating the vegetation index mean and vegetation index standard deviation of the fruit tree to determine the vegetation index distribution range of the fruit tree; wherein the segmentation prompt information includes the vegetation index distribution range of the fruit tree.

[0012] Optionally, the morphological characteristics indicate that the canopy aspect ratio of the fruit tree is within a first aspect ratio range; based on prior knowledge of fruit trees, the cropped digital orthophoto and the cropped canopy height model image, the spots in the image pre-segmentation results are screened to obtain partial spots, and segmentation prompt information for interactive segmentation of the fruit tree canopy is generated based on the partial spots, including: detecting the minimum circumscribed rectangle of the spots in the image pre-segmentation results to obtain the aspect ratio of each spot; screening the spots in the image pre-segmentation results whose aspect ratio is within the first aspect ratio range; detecting the set of pixel points within the spot outline after the aspect ratio is screened to obtain an area distribution histogram of the fruit tree spots; based on the area distribution histogram of the fruit tree spots, calculating the area mean of the fruit tree spots and the area standard deviation of the fruit tree spots to obtain the canopy area distribution range of the fruit tree.

[0013] Optionally, based on prior knowledge of fruit trees, the cropped canopy height model image and the cropped digital orthophoto, the image pre-segmentation results are filtered to obtain partial spots, including: filtering a first group of spots whose tree heights are within a second height distribution range from the image pre-segmentation results; filtering a second group of spots whose vegetation index grayscale values ​​are within the vegetation index grayscale value range from the first group of spots; filtering a third group of spots whose aspect ratios are within a third range and whose crown areas are within the crown area distribution range from the second group of spots, the third group of spots being partial spots of the fruit tree canopy.

[0014] Optionally, generating segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial patches, including: calculating the center coordinate point of each patch in the partial patches based on the coordinates of the four corner points of the circumscribed rectangle of each patch in the partial patches of the fruit tree canopy; using the center coordinate point of each patch in the partial patches as the segmentation prompt information for the first round of interactive segmentation, performing the first round of interactive segmentation on the cropped digital orthophoto with the specific target being the fruit tree canopy based on the segmentation prompt information for the first round of interactive segmentation and the SAM image segmentation model, and obtaining the first round of interactive segmentation results; using the i-th round of interactive segmentation as the segmentation prompt information for the first round of interactive segmentation; The center coordinate point of each patch in the mutual segmentation result is used as the segmentation prompt information of the i+1th round of interactive segmentation of the fruit tree canopy. Based on the segmentation prompt information of the i+1th round of interactive segmentation and the SAM image segmentation model, the cropped digital orthoimage is subjected to the i+1th round of interactive segmentation with the specific target being the fruit tree canopy, and the i+1th round of interactive segmentation result is obtained; i is a positive integer less than K, K is the number of iterations, and K is a positive integer greater than 1; the segmentation prompt information generated in the K-1th round is used as the learning result of the segmentation prompt information of the interactive segmentation with the specific target being the fruit tree canopy.

[0015] Optionally, the cropped digital orthophoto is interactively segmented based on the segmentation prompt information and the SAM image segmentation model to obtain the extraction result of the fruit tree canopy, including: performing the K-th round of interactive segmentation of the cropped digital orthophoto with the specific target being the fruit tree canopy based on the segmentation prompt information obtained in the K-1 round of learning and the SAM image segmentation model to obtain the final extraction map of the fruit tree canopy.

[0016] According to a second aspect of an embodiment of the present invention, a device for extracting a fruit tree canopy is provided. The device for extracting a fruit tree canopy includes: an image preprocessing module, a cropping module, a SAM image segmentation module, a screening module, and a generation module; the image preprocessing module is used to perform image preprocessing based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees; the cropping module is used to crop the digital orthophotos and canopy height model images respectively; the SAM image segmentation module is used to perform full segmentation on the cropped digital orthophotos using a SAM image segmentation model to obtain image pre-segmentation results; the screening module is used to filter the spots in the image pre-segmentation results to obtain partial spots based on prior knowledge of fruit trees, the cropped digital orthophotos, and the cropped canopy height model images; the generation module is used to generate segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial spots; the SAM image segmentation module is also used to perform interactive segmentation on the cropped digital orthophotos based on the segmentation prompt information and the SAM image segmentation model to obtain the extraction result of the fruit tree canopy; wherein the prior knowledge of fruit trees includes spatial characteristics, spectral characteristics, and morphological characteristics of fruit trees.

[0017] Optionally, the image preprocessing module is specifically used to: generate digital orthophotos, digital surface model images and digital terrain model images based on drone aerial image stitching and processing software and drone aerial images; unify the resolution of digital surface model images and digital terrain model images; generate a canopy height model image of the fruit tree based on the difference in grayscale values ​​between the first image and the second image; wherein the first image is a digital surface model image after unified resolution, and the second image is a digital terrain model image after unified resolution.

[0018] Optionally, the cropping module is specifically used to: align the coordinate system and spatial position of the digital orthophoto image and the canopy height model image; and crop the aligned digital orthophoto image and the canopy height model image into multiple small-size images respectively; wherein the cropped digital orthophoto image and the canopy height model image of the same scene have the same geographical location and have overlapping areas.

[0019] Optionally, the spatial feature indicates that the height distribution range of the fruit tree is a first height distribution range; the generation module is specifically used to: perform local maximum point detection based on the cropped canopy height model image, take the local maximum point as the crown vertex, and take the pixel value at the crown vertex as the tree height value; filter the crown vertices whose height values ​​are within the first height distribution range, and count the pixel values ​​at each filtered crown vertex to obtain the tree height distribution histogram; based on the height distribution histogram, calculate the mean height and height standard deviation of the fruit tree to determine the second height distribution range of the fruit tree.

[0020] Optionally, the spectral characteristics indicate that the target vegetation index grayscale value of the fruit tree is in a first grayscale range, and the target vegetation index is a green leaf index; the generation module is specifically used to: determine a vegetation index grayscale image of the target vegetation index based on the band of the cropped digital orthophoto; based on the vegetation index grayscale image, obtain the vegetation index grayscale value at the crown vertex after filtering based on the second height distribution range; filter the crown vertices whose vegetation index grayscale values ​​are within the first grayscale range, and count the vegetation index grayscale values ​​at each filtered crown vertex to obtain a vegetation index distribution histogram of the fruit tree; based on the vegetation index distribution histogram, calculate the vegetation index mean and vegetation index standard deviation of the fruit tree to determine the vegetation index distribution range of the fruit tree; wherein, the segmentation prompt information includes the vegetation index distribution range of the fruit tree.

[0021] Optionally, the morphological features indicate that the canopy aspect ratio of the fruit tree is within a first aspect ratio range; the generation module is specifically used to: detect the minimum circumscribed rectangle of the spots in the image pre-segmentation results to obtain the aspect ratio of each spot; filter the spots in the image pre-segmentation results whose aspect ratios are within the first aspect ratio range; detect the set of pixel points within the spot outline after aspect ratio filtering to obtain an area distribution histogram of the fruit tree spots; based on the area distribution histogram of the fruit tree spots, calculate the area mean and area standard deviation of the fruit tree spots to obtain the canopy area distribution range of the fruit tree.

[0022] Optionally, the screening module is specifically used to: screen a first group of spots whose tree heights are within a second height distribution range from the image pre-segmentation results; screen a second group of spots whose vegetation index grayscale values ​​are within the vegetation index grayscale value range from the first group of spots; screen a third group of spots whose aspect ratios are within a third range and whose crown areas are within the crown area distribution range from the second group of spots, and the third group of spots are partial spots of the fruit tree canopy.

[0023] Optionally, the generation module is used to calculate the center coordinate point of each patch in the partial patch of the fruit tree canopy based on the coordinates of the four corner points of the circumscribed rectangle of each patch in the partial patch; use the center coordinate point of each patch in the partial patch as the segmentation prompt information of the first round of interactive segmentation, and perform the first round of interactive segmentation with the specific target being the fruit tree canopy on the cropped digital orthophoto based on the segmentation prompt information of the first round of interactive segmentation and the SAM image segmentation model to obtain the first round of interactive segmentation result; use the center coordinate point of each patch in the i-th round interactive segmentation result as the segmentation prompt information of the i+1-th round of interactive segmentation of the fruit tree canopy, and perform the i+1-th round of interactive segmentation with the specific target being the fruit tree canopy on the cropped digital orthophoto based on the segmentation prompt information of the i+1-th round of interactive segmentation and the SAM image segmentation model to obtain the i+1-th round of interactive segmentation result; i is a positive integer less than K, K is the number of iterations, and K is a positive integer greater than 1; and use the segmentation prompt information generated in the K-1th round as the learning result of the segmentation prompt information of the interactive segmentation with the specific target being the fruit tree canopy.

[0024] Optionally, the SAM image segmentation module is also used to perform the K-th round of interactive segmentation of the cropped digital orthoimage with the specific target being the fruit tree canopy based on the segmentation hint information obtained from the K-1 round of learning, and the SAM image segmentation model to obtain the final extracted image patch of the fruit tree canopy.

[0025] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor comprises: a memory for storing instructions executable by the processor; wherein the processor is configured to: perform image preprocessing based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees; crop the digital orthophotos and canopy height model images respectively; perform full segmentation of the cropped digital orthophotos using a SAM image segmentation model to obtain image pre-segmentation results; based on prior knowledge of fruit trees, the cropped digital orthophotos, and the cropped canopy height model images, filter the spots in the image pre-segmentation results to obtain partial spots, generate segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial spots, and interactively segment the cropped digital orthophotos based on the segmentation prompt information and the SAM image segmentation model to obtain fruit tree canopy extraction results; wherein the prior knowledge of fruit trees comprises spatial characteristics, spectral characteristics, and morphological characteristics of fruit trees.

[0026] In a fourth aspect of the embodiments of the present invention, the embodiments of the present invention provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method for extracting the fruit tree canopy as described in the first aspect are implemented.

[0027] In a fifth aspect of the embodiments of the present invention, an embodiment of the present invention provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of a method for extracting a fruit tree canopy as described in the first aspect.

[0028] The method for extracting the fruit tree canopy provided by the embodiment of the present invention can perform image preprocessing based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees; then crop the digital orthophotos and canopy height model images respectively; then use the SAM image segmentation model to fully segment the cropped digital orthophotos to obtain image pre-segmentation results; based on fruit tree prior knowledge, the cropped canopy height model images and the cropped digital orthophotos, screen some of the image patches obtained by the image pre-segmentation results, generate segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial patches, and interactively segment the cropped digital orthophotos based on the segmentation prompt information and the SAM image segmentation model to obtain the extraction results of the fruit tree canopy. Since the fruit tree prior knowledge used includes the spatial characteristics, spectral characteristics and morphological characteristics of the fruit tree, the segmentation prompt information in the digital orthophotos can be accurately learned, and based on the learned segmentation prompt information, the cropped digital image is interactively segmented with the specific target of the fruit tree canopy, thereby accurately and quickly extracting the fruit tree canopy. Therefore, unlike other deep learning-based methods, this method can accurately extract the canopy of specific trees (in this case, fruit trees) without the need for precise tree crown outline samples or manual threshold setting. This method can overcome the problems of existing fruit tree canopy extraction methods, which rely heavily on the number and quality of manually outlined samples, and have poor model transferability and applicability in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is one of the flow charts of the method for extracting the fruit tree canopy provided in an embodiment of the present invention.

[0030] Figure 2 A logical diagram of learning segmentation prompt information provided by an embodiment of the present invention.

[0031] Figure 3 A screening logic diagram provided by an embodiment of the present invention.

[0032] Figure 4 This is the second flow chart of the method for extracting the fruit tree canopy provided in an embodiment of the present invention.

[0033] Figure 5 Schematic diagram of experimental results of different scenarios of the fruit tree canopy extraction method provided by an embodiment of the present invention.

[0034] Figure 6Schematic diagram of experimental comparison results between the fruit tree canopy extraction method provided in an embodiment of the present invention and related methods.

[0035] Figure 7 Schematic diagram of the extraction results of the citrus tree canopy provided in an embodiment of the present invention.

[0036] Figure 8 This is a schematic diagram of the extraction results of the orange tree canopy provided by an embodiment of the present invention.

[0037] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present invention.

[0038] Figure 10 A schematic structural diagram of a device for extracting the fruit tree canopy provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible implementations consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0040] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present invention. The singular forms "a," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, third, etc. may be used to describe various information in embodiments of the present invention, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0042] The embodiment of the present invention provides a method for extracting the canopy of a fruit tree. It uses common sense prior knowledge such as the approximate height distribution range of fruit trees, the existence of local elevation vertices at the location of the fruit trees, the fruit tree vegetation index within a certain threshold range, and the fruit tree crown shape being approximately circular to automatically generate fruit tree canopy segmentation prompt information. Combined with the SAM image segmentation model and the sampling iterative learning method, it does not require precise outline of tree crown contour samples like other deep learning-based methods, and does not require precise manual setting of thresholds. It can also achieve accurate extraction of specific tree canopies (fruit trees in the present invention). It can solve the problems of existing fruit tree canopy extraction methods that are heavily dependent on the number and quality of manually outlined samples, and have poor model transferability and applicability in different scenarios.

[0043] The method and device for extracting the fruit tree canopy provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] Example 1

[0045] Figure 1 The flow chart of a method for extracting the canopy of a fruit tree provided by an embodiment of the present invention is as follows. Figure 1 As shown in FIG, the following steps 101 to 104 may be included:

[0046] Step 101: The electronic device performs image preprocessing based on the drone aerial image to generate a digital orthophoto and a canopy height model image of the fruit tree.

[0047] Typically, drone aerial images are remote sensing images with orthographic projection properties.

[0048] Among them, digital orthophotos are aerial photographs that have been geometrically corrected (for example, to have a uniform scale). They are equivalent to orthographically projected aerial photographs. They are digitized aerial photographs / remote sensing images (monochrome / color) scanned and processed using digital elevation models. The projection errors are corrected pixel by pixel, and then the images are mosaicked and cropped according to the map range to generate image data.

[0049] A Canopy Height Model (CHM) is a surface model that represents the height of vegetation above the ground. It can usually be generated based on a Digital Surface Model (DSM) and a Digital Elevation Model (DEM). Specifically, CHM = DSM-DEM. A Digital Surface Model can also be called a Digital Canopy Model (DCM).

[0050] Step 102: The electronic device crops the digital orthophoto and the canopy height model image respectively.

[0051] The digital orthophoto image and the canopy height model image are cropped separately. The canopy height model image represents the height information of the objects in the digital orthophoto image.

[0052] Step 103: The electronic device uses the SAM image segmentation model to fully segment the cropped digital orthoimage to obtain an image pre-segmentation result.

[0053] Specifically, the electronic device inputs the cropped digital orthoimage into an image segmentation model (SegmentAnything Model, SAM), performs image pre-segmentation on the cropped digital orthoimage, and outputs a result of the image pre-segmentation.

[0054] Specifically, the full segmentation characteristics of the SAM segmentation model can be used to obtain image pre-segmentation results.

[0055] Step 104: The electronic device screens some patches obtained from the image pre-segmentation results based on prior knowledge of fruit trees, the cropped canopy height model image, and the cropped digital orthophoto, generates segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial patches, and interactively segments the cropped digital orthophoto based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy.

[0056] Among them, the prior knowledge of fruit trees includes the spatial characteristics, spectral characteristics and morphological characteristics of fruit trees.

[0057] Specifically, spatial characteristics indicate the height distribution range of fruit trees; spectral characteristics indicate the range of fruit tree vegetation index; and morphological characteristics indicate the shape, canopy aspect ratio, and area range of fruit trees.

[0058] For example, common sense prior knowledge 1 (spatial characteristics): The height of fruit trees is typically distributed within the range [0m, 3m]. Common sense prior knowledge 2 (spectral characteristics): The grayscale values ​​of vegetation GLI are typically distributed within the range [0, 1]. Common sense prior knowledge 3 (morphological characteristics): The shape of a fruit tree crown is approximately circular, and its canopy aspect ratio (the ratio of the major axis to the minor axis of the minimum circumscribed rectangle) is close to 1, typically distributed within the range [0.8, 1.2]. These ranges can be adjusted based on the actual scenario.

[0059] In this embodiment of the present invention, the tree height distribution range in an image can be learned based on spatial features, and the crown apex position of the fruit tree can be detected to obtain the tree height. The distribution range of the crown vegetation index can be learned based on spectral features. The area distribution range of the trees in an image can be learned based on morphological features. Furthermore, spatial, spectral, and morphological features can be used to select patches from the pre-segmentation results that meet the characteristics of fruit trees.

[0060] An embodiment of the present invention provides a method for extracting a fruit tree canopy. An electronic device can perform image preprocessing based on drone aerial images to generate a digital orthophoto and a canopy height model image of a fruit tree; then, the digital orthophoto and the canopy height model image are cropped respectively; then, the cropped digital orthophoto is fully segmented using a SAM image segmentation model to obtain an image pre-segmentation result; based on prior knowledge of fruit trees, the cropped canopy height model image, and the cropped digital orthophoto, partial patches obtained from the image pre-segmentation result are screened, and segmentation prompt information for interactive segmentation of the fruit tree canopy is generated based on the partial patches. The cropped digital orthophoto is interactively segmented based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy. Because the prior knowledge of the fruit trees used includes the spatial, spectral, and morphological characteristics of the fruit trees, the segmentation prompt information in the digital orthophoto can be accurately learned. Based on the learned segmentation prompt information, the cropped digital image is interactively segmented with the specific target of the fruit tree canopy, thereby accurately and quickly extracting the fruit tree canopy. Therefore, unlike other deep learning-based methods, this method can accurately extract the canopy of specific trees (in this case, fruit trees) without the need for precise tree crown outline samples or manual threshold setting. This method can overcome the problems of existing fruit tree canopy extraction methods, which rely heavily on the number and quality of manually outlined samples, and have poor model transferability and applicability in different scenarios.

[0061] Optionally, in a method for extracting a fruit tree canopy provided by an embodiment of the present invention, the above-mentioned step 101 may specifically include steps 11 to 13:

[0062] Step 11: The electronic device generates a digital orthophoto, a digital surface model image, and a digital terrain model image based on the drone aerial image stitching and processing software and the drone aerial image.

[0063] A digital surface model (DSM) is a digital representation of the surface morphology of an object. It is a database representing the spatial distribution of ground features. It typically consists of a data array consisting of a series of ground point coordinates (x, y, z) and surface attributes (such as object type and characteristics).

[0064] Digital Terrain Models (DTMs) include digital elevation models, digital slope models, and digital aspect models. Terrain is short for "surface morphology" or "geomorphology." Terrain can be described by elevation, or by information such as slope and aspect. A DTM is a simulation of a continuous surface using a large number of selected points with known x, y, and z coordinates in an arbitrary coordinate system. x and y represent the plane coordinates of the point, while the z value can represent information such as elevation, slope, and temperature. When z represents elevation, it is a Digital Elevation Model (DEM).

[0065] Step 12: The electronic device unifies the resolutions of the digital surface model image and the digital terrain model image.

[0066] Step 13: The electronic device generates a canopy height model image based on the difference in grayscale values ​​between the first image and the second image.

[0067] The first image is a digital surface model image with a unified resolution, and the second image is a digital terrain model image with a unified resolution.

[0068] In the embodiment of the present invention, the resolution and coordinate system of the digital orthoimage and the canopy height model image need to be unified before cropping, so as to ensure that the geographical ranges of the two images in the same scene are consistent.

[0069] Based on this solution, the electronic device can generate the digital orthophoto and the corresponding digital surface model image and digital terrain model image based on the drone aerial image through image preprocessing; based on the difference in grayscale values ​​of the digital surface model image and the digital terrain model image after unified resolution, the canopy height model image of the fruit tree is generated, so that the segmentation prompt information of the interactive segmentation of the fruit tree canopy can be accurately learned based on the digital orthophoto and the canopy height model image, so that the fruit tree canopy can be accurately extracted based on the segmentation prompt information.

[0070] Optionally, in a method for extracting a fruit tree canopy provided by an embodiment of the present invention, step 102 may be specifically performed through the following steps 21 to 23:

[0071] Step 21: The electronic device aligns the coordinate system and spatial position of the digital orthophoto image and the canopy height model image.

[0072] Step 22: The electronic device crops the aligned digital orthophoto and canopy height model image into multiple small-size images.

[0073] Among them, the cropped digital orthophoto and canopy height model images of the same scene have the same geographical location, and the two adjacent small images have a certain degree of overlap.

[0074] It is understandable that a digital orthophoto is a high-resolution, large-size image that is difficult to use on computer equipment. Therefore, a large image can be cropped and cut into multiple small images as the images to be detected for the input SAM segmentation model.

[0075] It should be noted that, in the embodiment of the present invention, the electronic device cuts a large image into multiple small images, which are placed in a folder when used, and the SAM segmentation model processes them in batches in sequence.

[0076] In the above step 22, overlapping cropping is performed to prevent the incompleteness of the fruit tree target during image segmentation. There is a certain overlapping distance (overlapping degree) between two adjacent small images after cropping.

[0077] Specifically, electronic devices follow the same cutting rules and Figure 1 and Figure 2 Cut. Figure 1 This large picture is cut into multiple small pictures, and then Figure 2 This large picture is divided into several smaller pictures. Figure 1 Digital orthophotos can be understood as real objects that can be seen by the naked eye. Figure 2 Image representation of canopy elevation model Figure 1 The height information of the object in the image is represented by each pixel value.

[0078] Based on this solution, after the electronic device acquires the canopy high-rise model image, it can align the resolution, coordinate system, and spatial position of the digital orthophoto and canopy height model images. Then, using the same cropping method, the two images are cropped into multiple fixed-size small images. These serve as the input images for the SAM segmentation model, facilitating the operation and learning of the electronic device. There is also a certain degree of overlap between adjacent small images to prevent incomplete fruit tree targets during image segmentation.

[0079] Based on spatial features, the height distribution range of trees in the image is learned.

[0080] Optionally, in a method for extracting a fruit tree canopy provided in an embodiment of the present invention, the spatial feature indicates that the height distribution range of the fruit tree is a first height distribution range. In step 104, steps 411 to 414 may be included to learn the height distribution range of the fruit tree canopy.

[0081] Step 411: The electronic device detects local maximum points based on the canopy height model image, uses the local maximum points as the tree crown vertices, and uses the pixel values ​​at the tree crown vertices as the tree height values.

[0082] The canopy height model image has a high resolution. It is understood that after obtaining the image pre-segmentation result, the image pre-segmentation result can be converted into an image, and then the local maximum point is calculated based on the canopy height model image.

[0083] It can be understood that through this step, the positions of the vertices of the crowns of some fruit trees and the height values ​​of some fruit trees can be obtained.

[0084] Step 412: The electronic device selects tree crown vertices whose height values ​​are within the first height distribution range, and counts the pixel values ​​at each selected tree crown vertex to obtain a tree height distribution histogram.

[0085] Exemplarily, the first height distribution range is [0m, 3m], and the pixel value at the top of the tree crown is the tree height value.

[0086] Step 413: The electronic device calculates the mean height and the standard deviation of the fruit trees based on the height distribution histogram, and determines a second height distribution range of the fruit trees.

[0087] For example, based on prior knowledge of fruit trees, we filter vertices with heights between 0 and 3 meters, count the pixel values ​​at these vertices, and obtain a tree height distribution histogram corresponding to the drone aerial image. Based on the tree height distribution histogram in the drone aerial image, we calculate the mean μ1 and standard deviation σ1 of the tree heights, and obtain the tree height distribution range in the drone aerial image based on formula (1).

[0088] H=μ1±1.96σ1 Formula (1)

[0089] It should be noted that the mean plus or minus the standard deviation is used to describe the distribution range of the data. Generally, the mean plus or minus two standard deviations can include 95% of the data points. This range is called the standard deviation range.

[0090] Based on this scheme, after the electronic device generates a canopy height model image corresponding to the digital orthophoto, local maximum detection can be performed on the canopy height model image, and each local maximum point is used as the crown vertex, and the pixel value at each crown vertex is used as the tree height value. Then, based on prior knowledge, the crown vertices within the first height distribution range are screened, and the pixel values ​​of the crown vertices that meet the first height distribution range are counted to obtain the tree height distribution histogram of the digital orthophoto, so that the second height distribution range of the fruit tree can be learned; the fruit tree spots in the fully segmented image can be accurately screened based on the learned second height distribution range of the fruit tree, thereby improving the extraction accuracy.

[0091] Based on spectral characteristics, learn the distribution range of tree crown vegetation index.

[0092] It should be noted that, based on the characteristics of fruit tree crowns belonging to vegetation, taking into account the applicability of visible light data (containing only red, green and blue bands), in the embodiment of the present invention, GLI (Green Leaf Index) is selected as the vegetation index for fruit tree canopy extraction.

[0093] Optionally, in a method for extracting a fruit tree canopy provided by an embodiment of the present invention, the spectral feature indicates that the grayscale value of a target vegetation index of the fruit tree is in a first grayscale range, and the target vegetation index is a green leaf index.

[0094] Furthermore, the above step 104 may include the following steps 420 to 423:

[0095] Step 420: The electronic device determines a vegetation index grayscale image of the target vegetation index based on the bands of the cropped digital orthophoto.

[0096] Step 421: The electronic device obtains the vegetation index grayscale value at the crown vertex after filtering within the second height distribution range based on the vegetation index grayscale image.

[0097] Step 422: The electronic device selects the crown vertices whose vegetation index grayscale values ​​are within the first grayscale range, and counts the vegetation index grayscale values ​​at each selected crown vertex to obtain a vegetation index distribution histogram of the fruit tree.

[0098] Step 423: The electronic device calculates the mean value and standard deviation of the vegetation index of the fruit trees based on the vegetation index distribution histogram, and determines the distribution range of the vegetation index of the fruit trees.

[0099] First, the vegetation index grayscale image is obtained by calculating the digital orthophoto band. The GLI of the digital orthophoto can be determined based on formula (2).

[0100]

[0101] Among them, Green represents the green pixel value, Red represents the red pixel value, and Blue represents the blue pixel value.

[0102] For example, the GLI grayscale image is superimposed on the fruit tree vertices filtered according to fruit tree prior knowledge 1 to obtain the grayscale values ​​of the fruit tree vertices. The fruit tree vertices with GLI values ​​between [0, 1] are filtered according to fruit tree prior knowledge 2. The GLI values ​​of the filtered vertices are counted to obtain a fruit tree GLI distribution histogram. The mean μ2 and standard deviation σ2 of the fruit tree GLI are calculated, and the GLI distribution range of the fruit tree is obtained based on formula (3).

[0103] G=μ2±1.96σ2 Formula (3)

[0104] Based on this scheme, the electronic device can generate a GLI grayscale image through the digital orthophoto based on the range of the green leaf index in the prior knowledge, and then obtain the GLI grayscale value at the crown vertex that has been filtered through the learned second height distribution range. Based on the GLI grayscale value at the crown vertex, the GIL distribution histogram in the digital orthophoto is determined, so that the green leaf index distribution range of the fruit trees in the digital orthophoto can be learned, and the fruit tree spots in the segmented image can be accurately screened based on the learned green leaf index distribution range, thereby improving the extraction accuracy.

[0105] Based on morphological features, learn the distribution range of tree crown areas in images.

[0106] Optionally, in a method for extracting a fruit tree canopy provided in an embodiment of the present invention, the morphological feature indicates that the canopy aspect ratio of the fruit tree is within a first aspect ratio range. Typically, the shape of a fruit tree canopy is approximately circular, and its canopy aspect ratio (the ratio of the major axis to the minor axis of the minimum circumscribed rectangle) is close to 1. This prior knowledge is used to set the aspect ratio between 0.5 and 1.5 to select patches to ensure that they better match the fruit tree canopy shape, thereby obtaining more accurate fruit tree patches.

[0107] Furthermore, the above step 104 may include steps 430 to 433:

[0108] Step 430: The electronic device detects the minimum bounding rectangle of the image spots in the image pre-segmentation result to obtain the aspect ratio of each image spot.

[0109] Step 431: The electronic device screens the image pre-segmentation results for patches whose aspect ratios are within a first aspect ratio range.

[0110] Step 432: The electronic device detects the pixel point set within the outline of the pattern after aspect ratio screening to obtain an area distribution histogram of the fruit tree pattern.

[0111] Step 433: The electronic device calculates the area mean and area standard deviation of the fruit tree patches based on the area distribution histogram of the fruit tree patches, and obtains the distribution range of the crown area of ​​the fruit trees.

[0112] For example, under normal circumstances, the shape of the fruit tree crown is approximately circular, and the aspect ratio of the crown (the ratio of the major axis to the minor axis of the minimum circumscribed rectangle) is close to 1. The electronic device screens the spots with an aspect ratio between [0.5, 1.5]. The pixel point set within the outline of the screened spots is detected, and the area of ​​the spots is counted to obtain a histogram of the spot area distribution. According to the histogram of the spot area distribution, the mean μ3 of the spot area and the standard deviation σ3 of the spot area are calculated. Based on formula (4), the distribution range of the crown spot area is obtained according to the mean μ3 of the spot area and the standard deviation σ3 of the spot area.

[0113] S=μ3±1.96σ3 Formula (4)

[0114] The pixel point set is multiplied by the ground resolution (meter / pixel) and converted into square meters to represent the crown area.

[0115] Based on this scheme, the electronic device can detect the minimum circumscribed rectangle of the spots in the segmented image after fully segmenting the cropped digital orthophoto, and obtain the aspect ratio of each spot; then, based on the first aspect ratio range indicated by the prior knowledge of fruit trees, screen the spots with aspect ratios within the first aspect ratio range; then, detect the set of pixel points within the outline of the fruit tree spots that meet the range to obtain the area distribution histogram of the fruit tree spots, so as to learn the crown area distribution range of the fruit trees in the digital orthophoto, and subsequently accurately screen the fruit tree spots in the segmented image based on the learned crown area distribution range of the fruit trees, thereby improving the extraction accuracy.

[0116] Example:

[0117] Figure 2 A logical diagram of learning segmentation prompt information provided by an embodiment of the present invention, such as Figure 2 As shown in , digital orthophotos are obtained based on drone aerial images. Preprocessing and image pre-segmentation yield a pre-segmentation result image and a canopy height model image. Maximum value detection can be performed based on the pre-segmentation result and canopy height model image. Then, the vertex points are detected and their height values ​​are obtained. The vertices are filtered using the height distribution range of prior knowledge 1, and the height values ​​of the filtered vertices are counted to obtain the height distribution range. A vegetation index is calculated based on the digital orthophotos, and a vegetation index grayscale map is determined. Based on the height-filtered vertices and the vegetation index grayscale map, the grayscale values ​​of the filtered vertices are determined. Vertices whose grayscale values ​​fall within the range are filtered using prior knowledge 2. The vegetation index at the filtered vertices is counted to obtain the vegetation index distribution range. The aspect ratio of the patches is calculated based on the preset segmentation result image. Prior knowledge 3 is used to filter patches whose aspect ratios fall within the range. The area of ​​each filtered patch is then counted to obtain the canopy area distribution range. Afterwards, the patches in the image pre-segmentation results are filtered according to the learned tree height distribution range, vegetation index distribution range, and crown area distribution range to obtain partial patches of the fruit tree canopy (intermediate results).

[0118] Optionally, in an embodiment of the present invention, based on the learned tree height value range, vegetation index grayscale value range, and crown area distribution range, the fruit tree spots in the image pre-segmentation results are screened to obtain partial spots of the extracted fruit tree canopy.

[0119] Furthermore, in the method for extracting the fruit tree canopy provided by an embodiment of the present invention, the above-mentioned step 104 may further include the following steps 440 to 442:

[0120] Step 440: The electronic device selects a first group of image spots whose tree heights are within a second height distribution range from the image pre-segmentation result.

[0121] Step 441: The electronic device selects a second group of image spots whose vegetation index grayscale values ​​are within a vegetation index grayscale value range from the first group of image spots.

[0122] Step 442: The electronic device selects a third group of spots from the second group of spots whose aspect ratios are within a third range and whose crown areas are within the crown area distribution range. The third group of spots is a portion of the fruit tree canopy.

[0123] For example, Figure 3 A screening logic diagram provided by an embodiment of the present invention, such as Figure 3 As shown in , for the spots in the image pre-segmentation results, the tree height value is first used to filter the spots, then the crown vegetation index is used to filter the spots, and finally the crown aspect ratio and area are used to filter the spots to obtain partial spots of the fruit tree canopy. That is, after the last filtering condition is filtered, the final extraction result of the partial spots of the fruit tree canopy is obtained.

[0124] Based on this scheme, the electronic device can first filter the spots in the image pre-segmentation results based on the learned tree height distribution range, and then continue to filter from the spots filtered by height based on the learned GLI range, and finally continue to filter from the spots filtered by GLI based on the learned crown area distribution range, so as to obtain partial spots of the fruit tree canopy.

[0125] Optionally, in a method for extracting a fruit tree canopy provided by an embodiment of the present invention, after the above step 442, the above step 104 may further include the following steps 443 to 445:

[0126] Step 443: The electronic device calculates the coordinates of the center point of each patch in the partial patch of the fruit tree canopy based on the coordinates of the four corner points of the circumscribed rectangle of each patch.

[0127] Exemplarily, the horizontal center coordinates of the pattern spot are determined based on formula (5), and the vertical center coordinates of the pattern spot are determined based on formula (6).

[0128]

[0129]

[0130] Among them, X min Indicates the maximum horizontal coordinate of the bounding rectangle of the spot, Xmax Indicates the minimum horizontal coordinate of the bounding rectangle of the spot; Y min Indicates the maximum vertical coordinate of the bounding rectangle of the spot, Y max Indicates the minimum vertical coordinate of the bounding rectangle of the image spot.

[0131] Step 444: The electronic device uses the center coordinate points of the partial image patches of the fruit tree canopy as segmentation prompt information for the first round of interactive segmentation, and performs the first round of interactive segmentation on the cropped digital orthoimage with the specific target being the fruit tree canopy based on the SAM image segmentation model to obtain the first round of interactive segmentation results.

[0132] Among them, the interactive segmentation of specific targets in the present invention refers to the image segmentation of the cropped digital orthophoto using the SAM segmentation model interactive segmentation based on the coordinates of each center point of the fruit tree patch determined in the previous round as segmentation prompt information.

[0133] Step 445: The electronic device uses the center coordinate point of each patch in the i-th round interactive segmentation result as the segmentation prompt information of the i+1-th round interactive segmentation of the fruit tree canopy, and performs the i+1-th round interactive segmentation of the cropped digital orthoimage with the specific target being the fruit tree canopy based on the segmentation prompt information of the i+1-th round interactive segmentation and the SAM image segmentation model to obtain the i+1-th round interactive segmentation result.

[0134] Wherein, i is a positive integer less than K, K is the number of iterations, and K is a positive integer greater than 1.

[0135] Step 446: Use the segmentation hint information generated in the K-1th round as the learning result of the segmentation hint information for interactive segmentation of the fruit tree canopy as the specific target.

[0136] Specifically, the above step 445 may be repeated multiple times to obtain segmentation prompt information of the fruit tree canopy.

[0137] It can be understood that the electronic device can learn the segmentation prompt information extracted from the canopy of the fruit tree through multiple iterations, which can make the obtained results more accurate.

[0138] Example:

[0139] Round 1: Based on the image pre-segmentation results and prior knowledge training, the screening information of the fruit tree canopy patches is obtained. The image pre-segmentation results are filtered based on the screening information to obtain partial patches of the fruit tree canopy. The center point of this partial patch is used as the segmentation prompt information 1 of the first round of interactive segmentation. Based on the segmentation prompt information 1, the cropped digital orthoimage is subjected to the first round of interactive segmentation with the specific target of the fruit tree canopy as the target, and the first round of fruit tree canopy extraction patches is obtained. Segmentation prompt information 2 is obtained based on the center point of the extracted patches of the fruit tree canopy in the first round.

[0140] Round 2: Based on segmentation hint information 2, the cropped digital orthophoto is interactively segmented with the fruit tree canopy as the second specific target to obtain the second round of fruit tree canopy extraction patches; the center point of the second round of fruit tree canopy extraction patches is used as the third segmentation hint information.

[0141] Repeat the above second round of steps K times, using the center point of the patch in the segmentation result of the previous round as the segmentation hint for the next round of interactive segmentation of the specific target, to obtain the final extraction result. K can be the maximum number of iterations set or the number of iterations when the iteration conditions are met.

[0142] Based on this scheme, the electronic device can use the center point of the image patch obtained in the previous round of image segmentation as the segmentation prompt information, and re-segment the cropped digital orthophoto with the specific target being the fruit tree canopy. Then, based on the segmentation prompt information of the fruit tree canopy learned in the previous round and the results of the interactive segmentation of the specific target in this round, new segmentation prompt information is obtained through multiple iterative learning, thereby learning accurate segmentation prompt information of the fruit tree patch.

[0143] Optionally, in the fruit tree canopy extraction method provided in an embodiment of the present invention, after the above step 446, the following step 447 may be further included:

[0144] Step 447: Based on the segmentation prompt information obtained through the K-1 round of learning, the electronic device uses the SAM image segmentation model to perform the K-th round of interactive segmentation on the cropped digital orthoimage with the specific target being the fruit tree canopy, to obtain the final extracted image patch of the fruit tree canopy.

[0145] Based on this scheme, after the electronic device iteratively learns the segmentation prompt information, it can input the learned segmentation prompt information and the cropped digital orthophoto into the SAM image segmentation model to perform interactive segmentation with the specific target being the fruit tree canopy, thereby accurately obtaining the extracted image patches of the fruit tree canopy in the drone aerial image.

[0146] Figure 4 This is a schematic diagram of the overall logic of a fruit tree canopy extraction method provided by an embodiment of the present invention. Based on common sense prior knowledge and drone aerial imagery, online learning is used to obtain the distribution range of fruit tree height values, vegetation index grayscale values, and area values. Fruit tree canopy patches are then filtered and extracted based on segmentation hints, resulting in the final fruit tree canopy patch extraction results.

[0147] Experimental data:

[0148] Experimental environment: Windows 10 operating system, computer graphics card is Intel (R) Xeon (R) CPU E5-2620v4 @ 2.10GHz (32 CPUs), equipped with two processors (a total of 32 CPU cores) 8GB video memory, equipped with NVIDIV Quadro P4000 GPU, based on the Pytorch 2.1.0 framework, using Python 3.9 and TorchVision 0.8, CUDA version 12.1, and CUDNN version 8801.

[0149] This method was experimentally validated by extracting different types of fruit tree canopies in two different geographical environments. Scenario 1 involved a citrus orchard, while Scenario 2 involved an orange orchard. These two comparative experiments demonstrated the feasibility and versatility of the method, demonstrating its ability to extract fruit tree canopies under zero-sample conditions. The experimental results showed that the accuracy of fruit tree canopy extraction reached 99% under both conditions.

[0150] Scenario 1: The image size input to the SAM image segmentation model is 3072×3072;

[0151] Scenario 2: The image size input to the SAM image segmentation model is 512×512.

[0152] Experiment 1: Based on the fruit tree canopy extraction method of the present invention, common sense prior knowledge is used to automatically generate segmentation prompt information for fruit tree canopy extraction for scene 1 and scene 2 respectively, and the fruit tree canopy is extracted in combination with the SAM image segmentation model.

[0153] Experiment 2: Based on the commonly used deep learning model U-net, the fruit tree canopy is extracted for scene 1 and scene 2 respectively.

[0154] The present invention was experimentally verified through experiments on extracting canopies of different types of fruit trees in two different scenes. In the above experiments, the zero-sample fruit tree canopy segmentation method proposed in the present invention was used to obtain fruit tree canopy extraction results for different types of fruit trees in two different scenes.

[0155] Figure 5 The experimental results of the fruit tree canopy extraction method provided by the embodiment of the present invention are as follows. Figure 5 As shown in , the first row is the experimental results of scene one, the second row is the experimental results of scene two, the first column is the digital orthophoto, the second column is the image pre-segmentation result, the third column is the screened part of the fruit tree spots, the fourth column is the screened part of the spots, and the fifth column is the fruit tree canopy extraction result.

[0156] As the number of iterations increases, the accuracy of the segmentation information gradually improves and increases. The method continuously learns and adapts to the characteristics of the fruit tree canopy during the iteration process, allowing for more accurate identification of subtle differences between the fruit tree canopy and other ground features, leading to more precise interactive segmentation. As the number of iterations increases, the understanding and capture of features such as the boundaries and shape of the fruit tree canopy gradually deepens, resulting in clearer and more reliable segmentation results.

[0157] Figure 6 This is a schematic diagram of an experimental comparison result provided by an embodiment of the present invention. The image segmentation of the test area using the fruit tree canopy extraction method and the U-net method proposed in the present invention is used to obtain the fruit tree canopy extraction results. Figure 6 As shown in , the first row is the experimental comparison results of scene one, and the second row is the experimental comparison results of scene two, where the first column is different digital orthoimages, the second column is the extraction results of the present invention, and the third column is the extraction results of the U-net method.

[0158] It should be noted that in the color comparison results, Figure 6 The fruit tree patches extracted from the results of the present invention are larger overall, while the fruit tree patches extracted from the results of the U-net method are smaller overall. Figure 6 In the last two images in the last row, the spots marked as white circles are not accurately extracted. It can be seen that the number of spots not accurately extracted by the present invention is much smaller than the number of spots not accurately extracted by the U-net method.

[0159] The classic deep learning semantic segmentation model U-net is used to extract fruit tree canopies. When the pixel difference between the fruit tree canopy and weeds is not obvious, obvious under-segmentation and missed segmentation problems often occur. A small amount of tree canopy may be mistakenly classified as background, affecting the accuracy of the segmentation results.

[0160] In comparison, the method proposed in this invention is significantly superior to the classic deep learning semantic segmentation model U-net method, and can better preserve the canopy structure and ensure that the integrity of the fruit trees is effectively maintained. This result is reflected not only in the segmentation accuracy, but also in the integrity of the canopy shape and structure.

[0161] The number of extracted fruit tree canopies was visually counted and combined with the number of true targets obtained visually to analyze the accuracy of fruit tree canopy extraction. A polygon that completely covered the fruit tree canopy was considered a correctly extracted canopy. Incomplete coverage and polygons that incorrectly covered the outside of the canopy were considered incorrectly extracted canopies, and a canopy with no polygon coverage inside was considered a missed canopy.

[0162] Figure 7 and Figure 8The results of fruit tree canopy extraction for two scenes using the present invention and their distribution in the study area are shown.

[0163] Figure 7 Shows the citrus tree canopy extraction results for the entire experimental park in scene 1. Figure 8 This figure shows the results of extracting the orange tree canopy for the entire experimental park in Scene 2. The first column is the digital orthophoto image, and the second column is the extracted patches of the fruit tree canopy.

[0164] In order to evaluate the accuracy of the method proposed in this study in the fruit tree canopy extraction task, a confusion matrix was constructed based on the difference between the actual number of fruit trees in the entire experimental park and the experimental extraction results in two scenarios, and the accuracy rate was calculated to perform a statistical analysis of the accuracy of fruit tree counting in the entire park.

[0165] Using scenario 1 of the present invention, there are a total of 1,852 trees in the entire park, 1,852 trees were extracted correctly, 2 trees were extracted incorrectly, and 0 trees were missed, with an accuracy rate of 99.9%; in scenario 2, there are a total of 3,122 trees in the entire park, 3,118 trees were extracted correctly, 9 trees were extracted incorrectly, and 5 trees were missed, with an accuracy rate of 99.8%.

[0166] The U-net deep learning method showed good accuracy, but still lower than the method of the present invention. There were also minor extraction errors and omissions, resulting in lower accuracy. Numerically, the extraction results of the method of the present invention were highly accurate and could meet the needs of practical application in orchards.

[0167] The fruit tree canopy extraction method provided by the embodiment of the present invention provides a novel zero-sample fruit tree canopy segmentation method, which can extract the fruit tree canopy without manually outlining samples or accurately setting thresholds. Since the image segmentation process does not require manual parameter adjustment, canopy extraction is not restricted by specific parameters. It has high portability and applicability for canopy extraction of different types of fruit trees in different scenes, and can greatly improve the efficiency of canopy extraction. The zero-sample fruit tree canopy segmentation method of the present invention is simple to operate, has good practicality, and is easy to apply and promote.

[0168] Example 2:

[0169] Corresponding to the embodiments of the aforementioned methods, the embodiments of the present invention also provide embodiments of the device and the terminal to which it is applied. The embodiments of the method for extracting the canopy of a fruit tree according to the embodiments of the present invention can be applied to electronic devices, such as servers or terminal devices. The device embodiments can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the file processing in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, such as Figure 9The figure shows a hardware structure diagram of an electronic device in which the method for extracting the canopy of a fruit tree according to an embodiment of the present invention is used. Figure 9 In addition to the processor 910, memory 930, network interface 920, and non-volatile memory 940 shown, the server or electronic device where the device 931 is located in the embodiment may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0170] like Figure 10 As shown, Figure 10 The present invention provides a block diagram of a device 1000 for extracting a fruit tree canopy, which includes an image preprocessing module 1001, a cropping module 1002, a SAM image segmentation module 1003, a screening module 1004, and a generation module 1005. The image preprocessing module 1001 is used to perform image preprocessing based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees; the cropping module 1002 is used to crop the digital orthophotos and canopy height model images respectively; the SAM image segmentation module 1003 is used to use the SAM image segmentation model to segment the cropped digital orthophotos. The digital orthophoto is fully segmented to obtain an image pre-segmentation result; a screening module 1004 is used to screen the spots in the image pre-segmentation result to obtain partial spots based on prior knowledge of fruit trees, the cropped digital orthophoto and the cropped canopy height model image; a generation module 1005 is used to generate segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial spots; the SAM image segmentation module 1003 is also used to interactively segment the cropped digital orthophoto based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy; wherein the prior knowledge of fruit trees includes the spatial characteristics, spectral characteristics and morphological characteristics of the fruit trees.

[0171] Optionally, the image preprocessing module 1001 is specifically used to: generate digital orthophotos, digital surface model images and digital terrain model images based on drone aerial image stitching and processing software and drone aerial images; unify the resolution of digital surface model images and digital terrain model images; generate a canopy height model image based on the difference in grayscale values ​​between the first image and the second image; wherein the first image is a digital surface model image with unified resolution, and the second image is a digital terrain model image with unified resolution.

[0172] Optionally, the cropping module 1002 is specifically used to: align the coordinate system and spatial position of the digital orthophoto image and the canopy height model image; and crop the aligned digital orthophoto image and the canopy height model image into multiple small-size images respectively; wherein the cropped digital orthophoto image and the canopy height model image of the same scene have the same geographical location and have overlapping areas.

[0173] Optionally, the spatial feature indicates that the height distribution range of the fruit tree is a first height distribution range; the screening module 1004 is specifically used to: perform local maximum point detection based on the cropped canopy height model image, use the local maximum point as the crown vertex, and use the pixel value at the crown vertex as the tree height value; screen the crown vertices whose height values ​​are within the first height distribution range, and count the pixel values ​​at each screened crown vertex to obtain a height distribution histogram of the tree; based on the height distribution histogram, calculate the mean height and height standard deviation of the fruit tree to determine a second height distribution range of the fruit tree.

[0174] Optionally, the spectral characteristics indicate that the target vegetation index grayscale value of the fruit tree is in a first grayscale range, and the target vegetation index is a green leaf index; the generation module 1005 is specifically used to: determine the vegetation index grayscale image of the target vegetation index based on the band of the cropped digital orthophoto; based on the vegetation index grayscale image, obtain the vegetation index grayscale value at the crown vertex after filtering based on the second height distribution range; filter the crown vertices whose vegetation index grayscale values ​​are within the first grayscale range, and count the vegetation index grayscale values ​​at each filtered crown vertex to obtain a vegetation index distribution histogram of the fruit tree; based on the vegetation index distribution histogram, calculate the vegetation index mean and vegetation index standard deviation of the fruit tree to determine the vegetation index distribution range of the fruit tree.

[0175] Optionally, the morphological features indicate that the canopy aspect ratio of the fruit tree is within a first aspect ratio range; the generation module 1005 is specifically used to: detect the minimum circumscribed rectangle of the spots in the image pre-segmentation result to obtain the aspect ratio of each spot; filter the spots in the image pre-segmentation result whose aspect ratio is within the first aspect ratio range; detect the set of pixel points within the spot outline after aspect ratio filtering to obtain an area distribution histogram of the fruit tree spots; based on the area distribution histogram of the fruit tree spots, calculate the area mean of the fruit tree spots and the area standard deviation of the fruit tree spots to obtain the canopy area distribution range of the fruit tree.

[0176] Optionally, the screening module 1004 is specifically used to: screen a first group of spots whose tree heights are within a second height distribution range from the image pre-segmentation results; screen a second group of spots whose vegetation index grayscale values ​​are within the vegetation index grayscale value range from the first group of spots; screen a third group of spots whose aspect ratios are within a third range and whose crown areas are within the crown area distribution range from the second group of spots, wherein the third group of spots are partial spots of the fruit tree canopy.

[0177] Optionally, the generation module 1005 is specifically used to: calculate the center coordinate point of each patch in the partial patch of the fruit tree canopy based on the coordinates of the four corner points of the circumscribed rectangle of each patch; use the center coordinate point of each patch in the partial patch as the segmentation prompt information of the first round of interactive segmentation, and perform the first round of interactive segmentation of the cropped digital orthophoto with the specific target being the fruit tree canopy based on the segmentation prompt information of the first round of interactive segmentation and the SAM image segmentation model to obtain the first round of interactive segmentation result; use the center coordinate point of each patch in the i-th round interactive segmentation result as the segmentation prompt information of the i+1-th round of interactive segmentation of the fruit tree canopy, and perform the i+1-th round of interactive segmentation of the cropped digital orthophoto with the specific target being the fruit tree canopy based on the segmentation prompt information of the i+1-th round of interactive segmentation and the SAM image segmentation model to obtain the i+1-th round of interactive segmentation result; i is a positive integer less than K, K is the number of iterations, and K is a positive integer greater than 1; and use the segmentation prompt information generated in the K-1th round as the learning result of the segmentation prompt information of the interactive segmentation with the specific target being the fruit tree canopy.

[0178] Optionally, the SAM image segmentation module 1003 is also used to perform the K-th round of interactive segmentation of the cropped digital orthoimage with the specific target being the fruit tree canopy based on the segmentation hint information obtained from the K-1 round of learning and the SAM image segmentation model to obtain the final extracted image patch of the fruit tree canopy.

[0179] It should be noted that the fruit tree canopy extraction device provided in the embodiment of the present invention can implement the steps of each method embodiment in the above embodiment 1, and thus can also achieve the effect of the above method, which will not be repeated here.

[0180] Correspondingly, an embodiment of the present invention also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: perform image preprocessing based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees; crop the digital orthophotos and canopy height model images into multiple small images respectively; use the SAM image segmentation model to fully segment the cropped digital orthophotos to obtain image pre-segmentation results; based on prior knowledge of fruit trees, the cropped digital orthophotos and the cropped canopy height model images, filter the spots in the image pre-segmentation results to obtain partial spots, generate segmentation prompt information for interactive segmentation of the fruit tree canopy based on the partial spots, and interactively segment the cropped digital orthophotos based on the segmentation prompt information and the SAM image segmentation model to obtain fruit tree canopy extraction results; wherein the prior knowledge of fruit trees includes spatial characteristics, spectral characteristics and morphological characteristics of fruit trees.

[0181] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0182] An embodiment of the present invention also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned fruit tree canopy extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0183] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0184] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned fruit tree canopy extraction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0185] It should be understood that the chip mentioned in the embodiment of the present invention can also be called a system-on-chip, a system-on-chip, a chip system, or a system-on-chip chip, etc.

[0186] An embodiment of the present invention provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned method for extracting the fruit tree canopy and achieve the same technical effect. To avoid repetition, it will not be described here.

[0187] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present invention. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0188] The foregoing description describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] Other embodiments of the present invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow from the general principles of the invention and include common knowledge or customary techniques in the art not claimed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0190] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for extracting fruit tree canopy, characterized in that: The method comprises: Image preprocessing is performed based on drone aerial images to generate digital orthophotos and canopy height model images of fruit trees; Cropping the digital orthophoto and the canopy height model image respectively to align the coordinate systems and spatial positions so that the digital orthophoto and the canopy height model image of the same scene have the same geographical location and have overlapping areas after cropping; The SAM image segmentation model is used to fully segment the cropped digital orthophoto to obtain the image pre-segmentation result; Based on prior knowledge of fruit trees, the cropped digital orthophoto and the cropped canopy height model image, the spots in the image pre-segmentation result are screened to obtain partial spots, segmentation prompt information for interactive segmentation of the fruit tree canopy is generated based on the partial spots, and the cropped digital orthophoto is interactively segmented based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy; The prior knowledge of fruit trees includes spatial characteristics, spectral characteristics and morphological characteristics of fruit trees; the spatial characteristics indicate the height distribution range of fruit trees; the spectral characteristics indicate the range of fruit tree vegetation index; the morphological characteristics indicate the shape, canopy aspect ratio and area range of fruit trees; the interactive segmentation indicates the segmentation prompt information of the fruit tree patches determined in the previous round, and uses the SAM image segmentation model to perform image segmentation on the cropped digital orthophoto in this round; Based on prior knowledge of fruit trees, the cropped digital orthophoto, and the cropped canopy height model image, the spots in the image pre-segmentation result are screened to obtain partial spots, including: learning the height distribution range of trees in the drone aerial image based on the spatial features and the cropped canopy height model image; Based on the spectral characteristics and the cropped digital orthophoto, learning the distribution range of the tree crown vegetation index in the drone aerial image; Based on the morphological features and the image pre-segmentation result, the tree crown area distribution range in the UAV aerial image is learned.

2. The method according to claim 1, wherein The image preprocessing based on the drone aerial image to generate a digital orthophoto and a canopy height model image of the fruit tree includes: Generate digital orthophotos, digital surface model images, and digital terrain model images based on drone aerial image stitching and processing software and drone aerial images; unifying the resolutions of the digital surface model image and the digital terrain model image; generating the canopy height model image based on a difference in grayscale values ​​between the first image and the second image; The first image is a digital surface model image with a unified resolution, and the second image is a digital terrain model image with a unified resolution.

3. The method according to claim 2, wherein The step of separately cropping the digital orthophoto and the canopy height model image comprises: aligning the coordinate systems and spatial positions of the digital orthophoto and the canopy height model image; respectively cropping the aligned digital orthophoto and the canopy height model image into a plurality of small-size images; Among them, the cropped digital orthophoto and canopy height model images of the same scene have the same geographical location and overlapping areas.

4. The method according to claim 1, wherein The spatial feature indicates that the height distribution range of the fruit trees is a first height distribution range; The method of screening the image pre-segmentation results to obtain partial image patches based on the prior knowledge of fruit trees, the cropped canopy height model image, and the cropped digital orthophoto includes: Based on the cropped canopy height model image, local maximum point detection is performed, and the local maximum point is used as the crown vertex, and the pixel value at the crown vertex is used as the tree height value; Screening tree crown vertices whose height values ​​are within the first height distribution range, and counting the pixel values ​​at each screened tree crown vertex to obtain a tree height distribution histogram; Based on the height distribution histogram, the mean height and the standard deviation of the fruit trees are calculated to determine a second height distribution range of the fruit trees.

5. The method according to claim 4, wherein The spectral feature indicates that the grayscale value of the target vegetation index of the fruit tree is in a first grayscale range, and the target vegetation index is a green leaf index; The method of screening the image pre-segmentation results to obtain partial image patches based on the prior knowledge of fruit trees, the cropped canopy height model image, and the cropped digital orthophoto includes: Determining a vegetation index grayscale image of the target vegetation index based on the bands of the cropped digital orthophoto; Based on the vegetation index grayscale image, obtaining a vegetation index grayscale value at a tree crown vertex after filtering based on the second height distribution range; Screening the crown vertices whose vegetation index grayscale values ​​are within the first grayscale range, and counting the vegetation index grayscale values ​​at each screened crown vertex to obtain a vegetation index distribution histogram of the fruit trees; Based on the vegetation index distribution histogram, the mean value and standard deviation of the vegetation index of the fruit trees are calculated to determine the distribution range of the second vegetation index of the fruit trees.

6. The method according to claim 5, wherein The morphological characteristics indicate that the canopy length-to-width ratio of the fruit tree is within a first length-to-width ratio range; The method of screening the image pre-segmentation results to obtain partial image patches based on the prior knowledge of fruit trees, the cropped canopy height model image, and the cropped digital orthophoto includes: Detecting the minimum circumscribed rectangle of the image spots in the image pre-segmentation result to obtain the aspect ratio of each image spot; screening the image pre-segmentation results for patches whose aspect ratios are within the first aspect ratio range; Detect the pixel point set within the outline of the patch after aspect ratio screening to obtain the area distribution histogram of the fruit tree patch; Based on the area distribution histogram of the fruit tree spots, the area mean of the fruit tree spots and the area standard deviation of the fruit tree spots are calculated to obtain the distribution range of the crown area of ​​the fruit trees.

7. The method according to claim 6, wherein The method of screening the image pre-segmentation results to obtain partial image patches based on the prior knowledge of fruit trees, the cropped canopy height model image, and the cropped digital orthophoto includes: Filtering a first group of image spots whose tree heights are within the second height distribution range from the image pre-segmentation result; Selecting a second group of spots whose vegetation index grayscale values ​​are within the vegetation index grayscale value range from the first group of spots; A third group of spots whose aspect ratios are within a third range and whose crown areas are within the crown area distribution range are selected from the second group of spots, and the third group of spots are partial spots of the fruit tree canopy.

8. The method according to claim 7, wherein The segmentation prompt information for interactive segmentation of the fruit tree canopy layer generated based on the partial image patches includes: Calculating the center coordinate point of each of the partial spots in the fruit tree canopy based on the coordinates of the four corner points of the circumscribed rectangle of each spot; Using the center coordinate point of each patch in the partial patch as segmentation hint information of the first round of interactive segmentation, performing the first round of interactive segmentation on the cropped digital orthoimage with the specific target being the fruit tree canopy based on the segmentation hint information of the first round of interactive segmentation and the SAM image segmentation model, and obtaining a first round of interactive segmentation result; The center coordinate point of each patch in the i-th round interactive segmentation result is used as the segmentation prompt information of the i+1-th round interactive segmentation of the fruit tree canopy, and the i+1-th round interactive segmentation of the cropped digital orthoimage with the specific target being the fruit tree canopy is performed based on the segmentation prompt information of the i+1-th round interactive segmentation and the SAM image segmentation model to obtain the i+1-th round interactive segmentation result; i is a positive integer less than K, K is the number of iterations, and K is a positive integer greater than 1; The segmentation hint information generated in the K-1th round is used as the learning result of the segmentation hint information for interactive segmentation with the specific target being the fruit tree canopy.

9. The method according to claim 8, wherein The interactive segmentation of the cropped digital orthoimage based on the segmentation prompt information and the SAM image segmentation model to obtain the extraction result of the fruit tree canopy includes: Based on the segmentation hint information obtained from the K-1 round of learning, the cropped digital orthoimage is subjected to the K-round interactive segmentation with the specific target being the fruit tree canopy, and the final extracted image patch of the fruit tree canopy is obtained.

10. A device for extracting fruit tree canopy, characterized in that: The device for extracting the fruit tree canopy comprises: an image preprocessing module, a cropping module, a SAM image segmentation module, a screening module and a generation module; The image preprocessing module is used to perform image preprocessing based on the drone aerial image to generate a digital orthophoto and a canopy height model image; The cropping module crops the digital orthophoto and the canopy height model image respectively to align the coordinate system and spatial position, so that the digital orthophoto and the canopy height model image of the same scene have the same geographical location and an overlapping area after cropping; The SAM image segmentation module is used to perform full segmentation on the cropped digital orthophoto using the SAM image segmentation model to obtain an image pre-segmentation result; The screening module is used to screen the spots in the image pre-segmentation result to obtain partial spots based on prior knowledge of fruit trees, the cropped digital orthophoto and the cropped canopy height model image; The generating module is used to generate segmentation prompt information of interactive segmentation of fruit tree canopies based on the partial image patches; The SAM image segmentation module is further used to interactively segment the cropped digital orthophoto based on the segmentation prompt information and the SAM image segmentation model to obtain an extraction result of the fruit tree canopy; The prior knowledge of fruit trees includes spatial characteristics, spectral characteristics and morphological characteristics of fruit trees; the spatial characteristics indicate the height distribution range of fruit trees; the spectral characteristics indicate the range of fruit tree vegetation index; the morphological characteristics indicate the shape, canopy aspect ratio and area range of fruit trees; the interactive segmentation indicates the segmentation prompt information of the fruit tree patches determined in the previous round, and uses the SAM image segmentation model to perform image segmentation on the cropped digital orthophoto in this round; The screening module is further used to learn the tree height distribution range in the UAV aerial image based on the spatial features and the cropped canopy height model image; learn the tree crown vegetation index distribution range in the UAV aerial image based on the spectral features and the cropped digital orthophoto; and learn the tree crown area distribution range in the UAV aerial image based on the morphological features and the image pre-segmentation results.

Images