Intelligent prediction method and device for mango fruit falling damage based on unmanned aerial vehicle image

Abstract

The invention discloses an unmanned aerial vehicle image-based mango fruit falling damage intelligent prediction method and device, and belongs to the technical field of image processing. The method comprises the steps of obtaining a canopy orthographic image of a fruit tree three-dimensional model of a target fruit tree, and determining height information of the target fruit tree; determining a first flower region set of the target fruit tree in the canopy orthographic image; based on the first flower region set, predicting fruits of the target fruit tree to obtain fruit prediction information of the target fruit tree; and predicting the fruit falling risk of the target fruit tree based on the fruit prediction information and the height information. According to the method, the fruit falling risk of the fruit tree can be accurately predicted.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a method and device for intelligently predicting mango fruit drop hazards based on drone images. Background Art

[0002] In orchards or green belts, fruits may fall from the trees due to various reasons such as ripening, wind or precipitation. It is necessary to conduct risk prediction for this phenomenon in order to control the impact and harm to agriculture and ecosystems.

[0003] At present, the main method is to observe the color of the fruit to judge the fruit's maturity and then predict the risk of the fruit falling. This risk prediction method is not accurate. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a method and device for intelligently predicting mango fruit drop hazards based on drone images, which can accurately predict the risk of fruit drop on fruit trees.

[0005] In a first aspect, the present application provides a method for intelligently predicting mango fruit drop hazards based on drone images, the method comprising:

[0006] Obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree, and determining height information of the target fruit tree;

[0007] Determine a first flower region set of the target fruit tree in the canopy orthographic image;

[0008] Based on the first flower region set, predicting the fruit of the target fruit tree to obtain fruit prediction information of the target fruit tree;

[0009] Based on the fruit prediction information and the height information of the target fruit tree, the fruit drop risk of the target fruit tree is predicted.

[0010] According to the intelligent prediction method for mango fruit drop hazards based on drone images of the present application, the fruits of the target fruit trees are predicted and estimated based on the conversion relationship between flowers and fruits through the first flower area set in the canopy orthography, and combined with the height information, accurate prediction of the fruit drop risk is achieved at an early stage, providing sufficient time and data support for the subsequent reasonable management of factors affecting the fruit drop risk.

[0011] According to an embodiment of the present application, the step of predicting the fruit of the target fruit tree based on the first flower region set to obtain the fruit prediction information of the target fruit tree includes:

[0012] Determining a mean area value and a mean diameter value of the flower regions in the first flower region set;

[0013] Determine a second flower region set in the first flower region set according to a numerical relationship between the region area of ​​each flower region and the area mean, and a numerical relationship between the region diameter and the diameter mean;

[0014] The fruit prediction information is determined according to the second flower region set.

[0015] According to an embodiment of the present application, determining the fruit prediction information according to the second flower region set includes:

[0016] determining a third flower area set corresponding to the second flower area set according to the flower loss coefficient of the target fruit tree;

[0017] The fruit prediction information is obtained based on the flower-fruit conversion rate of the target fruit tree and the third flower area set.

[0018] According to an embodiment of the present application, predicting the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree includes:

[0019] constructing a fruit risk item according to the fruit prediction information;

[0020] Based on the fruit risk item and the height information, the fruit falling risk is obtained.

[0021] According to an embodiment of the present application, determining the height information of the target fruit tree includes:

[0022] Obtaining the trunk height of the target fruit tree;

[0023] Binarization processing is performed on the canopy orthographic image;

[0024] Perform pixel traversal on the binarized canopy orthographic image to obtain the canopy height of the target fruit tree;

[0025] The height information is determined according to the canopy height and the trunk height.

[0026] According to one embodiment of the present application, determining the first flower region set of the target fruit tree in the canopy orthographic image includes:

[0027] The flower regions in the canopy orthographic image are segmented by a deep semantic segmentation network to obtain the first flower region set, and the deep semantic segmentation network is trained by a canopy sample set.

[0028] According to one embodiment of the present application, the step of obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree includes:

[0029] Acquire multiple fruit tree images of the target fruit tree;

[0030] Based on the fruit tree image, the target fruit tree is three-dimensionally reconstructed to obtain the three-dimensional model of the fruit tree, so as to determine the fruit tree orthographic image of the three-dimensional model of the fruit tree;

[0031] The orthographic image of the fruit tree is segmented to obtain the orthographic image of the canopy.

[0032] In a second aspect, the present application provides an intelligent prediction device for mango fruit falling hazards based on drone images, the device comprising:

[0033] An acquisition module, used to acquire a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree, and determine height information of the target fruit tree;

[0034] A first processing module, configured to determine a first flower region set of the target fruit tree in the canopy orthographic image;

[0035] A second processing module is used to predict the fruit of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree;

[0036] The third processing module is used to predict the fruit falling risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree.

[0037] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the method for intelligently predicting mango fruit falling hazards based on drone images as described in the first aspect above is implemented.

[0038] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent prediction method for mango fruit falling hazards based on drone images as described in the first aspect above.

[0039] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the intelligent prediction method for mango fruit falling hazards based on drone images as described in the first aspect.

[0040] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the intelligent prediction method for mango fruit falling hazards based on drone images as described in the first aspect above.

[0041] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 This is one of the flow charts of the intelligent prediction method of mango fruit falling hazard based on drone images provided in the embodiment of the present application;

[0044] Figure 2 This is the second flow chart of the intelligent prediction method of mango fruit falling hazard based on drone images provided in the embodiment of the present application;

[0045] Figure 3 It is a structural schematic diagram of a mango fruit falling hazard intelligent prediction device based on drone images provided in an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0048] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0049] In combination with the accompanying drawings, the intelligent prediction method for mango fruit falling hazard based on drone images, the intelligent prediction device for mango fruit falling hazard based on drone images, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0050] Among them, the intelligent prediction method of mango fruit falling damage based on drone images can be applied to the terminal, and can be specifically executed by hardware or software in the terminal.

[0051] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0052] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the intelligent prediction method for mango fruit drop hazard based on drone images provided in the embodiments of the present application, the execution subject of the intelligent prediction method for mango fruit drop hazard based on drone images can be an electronic device or a functional module or functional entity in an electronic device that can implement the intelligent prediction method for mango fruit drop hazard based on drone images. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The intelligent prediction method for mango fruit drop hazard based on drone images provided in the embodiments of the present application is described below using electronic devices as the execution subject as an example.

[0053] like Figure 1 As shown, the intelligent prediction method for mango fruit falling hazards based on drone images includes: steps 110 to 140.

[0054] Step 110: Obtain a canopy orthographic image of the three-dimensional fruit tree model of the target fruit tree, and determine the height information of the target fruit tree.

[0055] It can be understood that this application is to predict the risk of fruit drop for fruit trees in the flowering period, which can provide important data support for optimizing agricultural management and production plans.

[0056] It should be noted that the target fruit tree can be a fruit tree with flower characteristics during the flowering period, for example, it can be a mango tree, a peach tree or an apple tree, etc. The subsequent embodiments of this application all take the mango tree as an example to illustrate the scheme of this application, which is not regarded as limiting the scope of protection of this application.

[0057] The canopy orthographic image may be an orthographic projection of the canopy of the target fruit tree in the side view direction at multiple angles, which can cover the full range of features of the canopy of the target fruit tree.

[0058] The three-dimensional model of the fruit tree is a three-dimensional model of the target fruit tree, which can be obtained by performing a three-dimensional scan on the target fruit tree.

[0059] The height information may include the height of the target fruit tree in actual space, which is obtained by measuring the height of the target fruit tree, and may also include the canopy height of the target fruit tree, which is obtained by measuring the height from the bottom of the canopy of the target fruit tree to the ground.

[0060] Step 120: Determine a first flower region set of the target fruit tree in the canopy orthographic image.

[0061] The first flower region set includes all flower regions from the canopy orthographic image. The flower regions can be achieved through target segmentation and can be used to characterize the flower density of the target fruit tree.

[0062] It can be understood that the canopy orthographic images in multiple directions have richer spatial information, can more accurately characterize the canopy growth status of the target fruit tree, and the obtained first flower area set is more accurate.

[0063] In this step, a category label is assigned to each pixel in the canopy orthographic image, and the category label may be flower or non-flower. The flower region in the canopy orthographic image is segmented, and a set of multiple flower regions obtained is taken as a first flower region set.

[0064] Step 130: Based on the first flower region set, predict the fruit of the target fruit tree to obtain fruit prediction information of the target fruit tree.

[0065] It can be understood that the fruit of the target fruit tree is predicted by the first flower area set in the blooming period, combined with the pollination rate and fruit rate of the flowers, as well as other influencing factors such as water and nutrients.

[0066] In this step, the number of flowers is obtained through the first flower area set and the size of individual flowers of the target fruit tree, and then the number of flowers is multiplied by the pollination rate and the fruit rate, and can also be multiplied by the coefficients of other influencing factors to obtain fruit prediction information.

[0067] Step 140: predict the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information.

[0068] The fruit prediction information may include the predicted number of fruit trees or fruit density of the target fruit trees.

[0069] It is understandable that the higher the fruit of the target fruit tree, the greater the risk of fruit drop, and the greater the number of fruit trees or the fruit density, the greater the risk of fruit drop.

[0070] In actual implementation, the fruit drop risk of the target fruit tree is predicted based on the fruit prediction information and height information through the numerical relationship between the fruit prediction information and height information and the fruit drop risk.

[0071] Among them, the fruit drop risk can be used as data support for management measures such as spraying inhibition results and fruit drop warnings, to conduct early intervention on target fruit trees and achieve more effective management of fruit drop risks.

[0072] In the related art, the risk of mangoes falling is predicted by observing the color of the fruit and judging the degree of yellowing of the fruit. This technology can only be used for predictions in the late stage of fruit growth and cannot provide sufficient time to take corresponding preventive measures.

[0073] According to the intelligent prediction method for mango fruit drop hazard based on drone images provided in the embodiment of the present application, the fruits of the target fruit trees are predicted and estimated based on the conversion relationship between flowers and fruits through the first flower area set in the canopy orthography, and combined with the height information, accurate prediction of the fruit drop risk is achieved at an early stage, providing sufficient time and data support for the subsequent reasonable management of the factors affecting the fruit drop risk.

[0074] In some embodiments, obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree includes:

[0075] Acquire multiple fruit tree images of target fruit trees;

[0076] Based on the fruit tree image, the target fruit tree is reconstructed in three dimensions to obtain a three-dimensional model of the fruit tree, so as to determine the orthographic image of the fruit tree three-dimensional model;

[0077] The orthographic image of the fruit tree is segmented to obtain the canopy orthographic image.

[0078] Among them, the multiple fruit tree images can be obtained by photographing the target fruit tree when the mango flowers are in full bloom, including the top image and the side image of the target fruit tree. The resolution of the fruit tree image can distinguish the area where the flowers of the target fruit tree are located.

[0079] It is understandable that while taking the image of each mango tree, the image acquisition device can also be calibrated to determine the internal parameters of the image acquisition device such as focal length and principal point, as well as external parameters such as position and direction, as the information carried by the fruit tree image, so as to ensure the accurate mapping relationship between the pixels in the three-dimensional model of the fruit tree and the physical measurement units in the real world during the three-dimensional reconstruction stage.

[0080] In actual implementation, during the peak period of mango flowering, drones equipped with image acquisition devices are used to collect high-resolution fruit tree images of each mango tree, with the resolution of the fruit tree images being no less than 0.25mm.

[0081] In order to obtain multi-perspective information and more accurately reconstruct the three-dimensional shape and structure of the target fruit tree, for a mango tree, no less than five images that can completely cover the mango tree are required from the side and top.

[0082] The three-dimensional reconstruction of a single mango tree can be achieved through the OpenDroneMap software, which generates a three-dimensional model of the mango tree, obtains the size mapping relationship between the pixels in each three-dimensional model of the fruit tree and the actual space, and determines the pixel size of each three-dimensional model of the fruit tree in cm.

[0083] The three-dimensional reconstruction is completed for all individual mango trees in turn, and a collection of fruit tree orthographic images P and a collection of pixel sizes S of the three-dimensional models of the fruit trees corresponding to all mango trees are obtained.

[0084] The fruit tree orthographic image of each mango tree in the collection P is input into the canopy recognition model. The canopy recognition model performs canopy recognition and segmentation on the fruit tree orthographic image to obtain the canopy orthographic image and trunk orthographic image of each mango tree.

[0085] The canopy recognition model may be obtained by training a neural network model with a plurality of fruit tree training samples in a fruit tree sample set, and the fruit tree training samples may be sample fruit tree orthographic images with canopy orthographic image labels and trunk orthographic image labels.

[0086] The canopy recognition model can be constructed based on a deep semantic segmentation network. For example, the deep semantic segmentation network can be a feature reconstruction network (Feature Reconstruction Network, FR-Net).

[0087] The collection of canopy orthographic images of each mango tree is taken as C all , the set of orthographic images of each mango tree trunk is taken as T all .

[0088] In this embodiment, by performing three-dimensional reconstruction on the target fruit tree and segmenting the orthographic image of the fruit tree, a canopy orthographic image can be obtained, which provides a basis for obtaining accurate height information.

[0089] In some embodiments, determining a first set of flower regions of a target fruit tree in a canopy orthographic image includes:

[0090] The flower regions in the canopy orthographic image are segmented by a deep semantic segmentation network to obtain a first flower region set. The deep semantic segmentation network is trained by the canopy sample set.

[0091] It can be understood that the deep semantic segmentation network is used to predict the semantic category to which each pixel belongs, so as to identify the category corresponding to each pixel area in the image and distinguish the boundaries.

[0092] Among them, the deep semantic segmentation network can be built based on FR-Net, and the FR-Net is trained by multiple canopy training samples in the canopy sample set. The canopy training samples can be sample canopy orthographic images with flower area labels.

[0093] In actual execution, the canopy orthographic image is input into the deep semantic segmentation network, which performs feature extraction and semantic information fusion on the canopy orthographic image, and performs semantic segmentation prediction on each pixel to obtain and output the pixel area corresponding to each flower area, and takes the set of pixel areas corresponding to multiple flower areas as the first flower area set F.

[0094] In this embodiment, the flower region in the canopy orthographic image is segmented by a deep semantic segmentation network, which can effectively capture the geometric shape and position information of the flower region and improve the segmentation accuracy of the flower region.

[0095] In some embodiments, based on the first flower region set, predicting the fruit of the target fruit tree to obtain the fruit prediction information of the target fruit tree includes:

[0096] Determine the mean area and the mean diameter of the flower regions in the first flower region set;

[0097] Determine a second flower region set in the first flower region set according to a numerical relationship between the region area of ​​each flower region and the area mean, and a numerical relationship between the region diameter of each flower region and the diameter mean;

[0098] Fruit prediction information is determined based on the second flower region set.

[0099] It can be understood that the regional area of ​​the flower region can be the number of pixels in the flower region, the area mean can be the average of the number of pixels corresponding to the regional area of ​​each flower region in the first flower region set, and the diameter mean can be the average of the regional diameter of each flower region in the first flower region set.

[0100] Among them, the area diameter of the flower area can be represented by the number of pixels. The area diameter of the flower area can be the diameter of the largest inscribed circle of the flower area, or can be the diagonal of the smallest circumscribed matrix of the flower area, or a boundary search can be performed on the flower area, and the distance between the two farthest points is taken as the area diameter of the flower area.

[0101] Due to problems such as overlap between mango flowers, it is necessary to further determine the flower area from the segmented flower area to remove anomalies caused by low segmentation accuracy and overlapping problems.

[0102] The numerical relationship can be a ratio relationship, a difference relationship, a square difference relationship, etc.

[0103] In actual implementation, the area and diameter of each flower region in the first flower region set F are averaged to obtain the area mean M S and mean diameter M D , and calculate the area of ​​each region and the mean area M S The ratio m between the diameter of each area and the mean diameter M is calculated. D The ratio n between them is used to re-determine whether each flower area in the first flower area set is a flower, and the determination result is F f1 , the judgment formula is:

[0104]

[0105] All the judgment results F f1 =m is taken as the second flower region set.

[0106] According to the conversion rate between flowers and fruits of historical mango trees and the second flower region set, the fruit of the target fruit tree can be predicted to obtain the fruit prediction information of the target fruit tree.

[0107] In this embodiment, by determining the second flower area set in the first flower area set, the problem of fewer flowers due to overlapping flowers in the first flower area set can be overcome. At the same time, the situation in which the flower areas segmented by the deep semantic segmentation model are in abnormal shapes such as long strips, which lead to incorrect identification as flowers, can be eliminated.

[0108] In some embodiments, determining fruit prediction information based on the second flower region set includes:

[0109] Determining a third flower region set corresponding to the second flower region set according to the flower loss coefficient of the target fruit tree;

[0110] Fruit prediction information is obtained based on the flower-fruit conversion rate of the target fruit tree and the third flower area set.

[0111] Among them, the flower loss coefficient can be determined based on factors that affect flower loss, such as climate, pests and diseases, and fertilization strategies.

[0112] During the growth of mango trees, affected by external factors such as wind, the flowers will naturally fall off to a certain extent. In order to improve the accuracy of fruit prediction information, it is necessary to calculate the loss of flowers during the growth process.

[0113] For example, taking the flower loss coefficient as 0.5, we can obtain the flower quantity F of each flower area in the second flower area set after considering the loss. f2 =0.5×F f1 , F f2 The collection is used as the third flower area set.

[0114] It is understandable that the flower-to-fruit conversion rate is related to factors such as pollination efficiency, pest and disease and fertilization strategies, and the variety of the target fruit tree.

[0115] For example, if the flower-fruit conversion rate is 0.3, the fruit prediction information y corresponding to each flower area in the third flower area set can be obtained. f :

[0116] y f =0.3×F f2 +5

[0117] In this embodiment, the accuracy of fruit prediction information can be improved by using the flower loss coefficient and the flower-fruit conversion rate.

[0118] In some embodiments, determining the height information of the target fruit tree includes:

[0119] Get the trunk height of the target fruit tree;

[0120] Binarization of canopy orthographic images;

[0121] Perform pixel traversal on the binary canopy orthographic image to obtain the canopy height of the target fruit tree;

[0122] Determine height information based on canopy height and trunk height.

[0123] Among them, binarization processing can simplify the canopy orthographic image, reduce the amount of processed data, and retain important information in the canopy orthographic image.

[0124] In actual implementation, the set of canopy orthographic images C all The canopy orthographic image C of the target fruit tree in is converted into a grayscale image, and the grayscale value gray of each pixel is obtained. The canopy orthographic image is binarized by setting the grayscale threshold.

[0125] For example, the binarized image C of the canopy orthographic image bin for:

[0126]

[0127] Among them, gray is the grayscale value.

[0128] For the binarized image C of the canopy orthographic image bin, along the row direction, in each column according to the binary map C bin Each pixel is traversed in order from the bottom to the top, and the number of pixel values ​​1 in each column direction is recorded. After completing the traversal of all columns, a set C_heighPixels is constructed according to the number of pixels obtained.

[0129] Determine the maximum number of pixels in C_heighPixels, determine the pixel size of the target fruit tree in the pixel size collection S, multiply the maximum value by the pixel size of the target fruit tree, and obtain the canopy height C of the target fruit tree. H .

[0130] The set of tree trunk orthographic images T all The trunk orthographic image T of the target fruit tree in is converted into a grayscale image, and the grayscale value gray of each pixel is obtained. The trunk orthographic image is binarized by setting the grayscale threshold.

[0131] For example, the binarized image T of the trunk orthographic image bin for:

[0132]

[0133] Among them, gray is the grayscale value.

[0134] For the binary map T of the trunk orthographic image bin , along the row direction, in each column according to the binary map T bin Traverse each pixel from bottom to top, record the number of pixels with a value of 1 in each column direction, and after completing the traversal of all columns, construct a set T_heighPixels based on the number of pixels obtained.

[0135] Determine the maximum number of pixels in T_heighPixels, determine the pixel size of the target fruit tree in the pixel size collection S, multiply the maximum value by the pixel size of the target fruit tree, and obtain the trunk height T of the target fruit tree H .

[0136] The canopy height C of the target fruit tree H and trunk height T H Sum and get the height information of the target fruit tree.

[0137] In this embodiment, the canopy height is calculated by pixel traversal, and the obtained height information is more accurate and suitable for fruit trees of more shapes.

[0138] In some embodiments, based on the fruit prediction information and the height information of the target fruit tree, predicting the fruit drop risk of the target fruit tree includes:

[0139] According to the fruit prediction information, construct the fruit risk item;

[0140] Based on the fruit risk item and height information, the fruit falling risk is obtained.

[0141] It can be understood that the risk of fruit dropping of the target fruit tree is positively correlated with the height of the target fruit tree. The taller the target fruit tree, the greater the risk of fruit dropping. The risk of fruit dropping of the target fruit tree is also related to the number and density of fruits. The greater the number or density of fruits of the target fruit tree, the greater the risk of fruit dropping.

[0142] The height information can be T H +C H .

[0143] Fruit drop risk is used to characterize the risk level of fruit drop of target fruit trees.

[0144] In actual implementation, according to the fruit prediction information y f , construct fruit risk items

[0145] The fruit drop risk Risk is calculated as follows:

[0146]

[0147] In this embodiment, by considering the height information and the fruit prediction information, two factors that have a greater impact on the risk of fruit falling, an accurate risk of fruit falling can be obtained.

[0148] An embodiment is described below.

[0149] like Figure 2 As shown, during the mango flowering period, a drone equipped with an image acquisition device collects high-resolution fruit tree images of each mango tree, and the resolution of the fruit tree images is not less than 0.25 mm.

[0150] In order to obtain multi-perspective information and more accurately reconstruct the three-dimensional shape and structure of the target fruit tree, for a mango tree, no less than five images that can completely cover the mango tree are required from the side and top.

[0151] The three-dimensional reconstruction of a single mango tree can be achieved through the OpenDroneMap software, which generates a three-dimensional model of the mango tree, obtains the size mapping relationship between the pixels in each three-dimensional model of the fruit tree and the actual space, and determines the pixel size of each three-dimensional model of the fruit tree in cm.

[0152] The three-dimensional reconstruction is completed for all individual mango trees in turn, and a collection of orthographic images P and a collection of pixel sizes S of the three-dimensional models of all mango trees are obtained.

[0153] The fruit tree orthographic image of each mango tree in the collection P is input into the canopy recognition model. The canopy recognition model performs canopy recognition and segmentation on the fruit tree orthographic image to obtain the canopy orthographic image and trunk orthographic image of each mango tree.

[0154] The collection of canopy orthographic images of each mango tree is taken as C all , the set of orthographic images of each mango tree trunk is taken as T all .

[0155] C all Any canopy orthographic image in is input into the deep semantic segmentation network, which extracts features and fuses semantic information on the canopy orthographic image, and performs semantic segmentation prediction on each pixel to obtain and output the pixel area corresponding to each flower area, and takes the set of pixel areas corresponding to multiple flower areas as the first flower area set F.

[0156] The area and diameter of each flower region in the first flower region set F are averaged to obtain the area mean M S and mean diameter M D , and calculate the area of ​​each region and the mean area M S The ratio m between the diameter of each area and the mean diameter M is calculated. D The ratio n between them is used to re-determine whether each flower area in the first flower area set is a flower, and the determination result is F f1 , the judgment formula is:

[0157]

[0158] All the judgment results F f1 =m is taken as the second flower region set.

[0159] Taking the flower loss coefficient as 0.5, we can get the flower quantity F of each flower area in the second flower area set after considering the loss. f2 =0.5×F f1 , F f2 The collection is used as the third flower area set.

[0160] Taking the flower-fruit conversion rate as 0.3, we can get the fruit prediction information y corresponding to each flower area in the third flower area set. f :

[0161] y f =0.3×F f2 +5

[0162] The set of canopy orthophotos C allThe canopy orthographic image C of the target fruit tree in is converted into a grayscale image, and the grayscale value gray of each pixel is obtained. The canopy orthographic image is binarized by setting the grayscale threshold.

[0163] For example, the binarized image C of the canopy orthographic image bin for:

[0164]

[0165] Among them, gray is the grayscale value.

[0166] For the binarized image C of the canopy orthographic image bin , along the row direction, in each column according to the binary map C bin Each pixel is traversed in order from the bottom to the top, and the number of pixel values ​​1 in each column direction is recorded. After completing the traversal of all columns, a set C_heighPixels is constructed according to the number of pixels obtained.

[0167] Determine the maximum number of pixels in C_heighPixels, determine the pixel size of the target fruit tree in the pixel size collection S, multiply the maximum value by the pixel size of the target fruit tree, and obtain the canopy height C of the target fruit tree. H .

[0168] The set of tree trunk orthographic images T all The trunk orthographic image T of the target fruit tree in is converted into a grayscale image, and the grayscale value gray of each pixel is obtained. The trunk orthographic image is binarized by setting the grayscale threshold.

[0169] For example, the binarized image T of the trunk orthographic image bin for:

[0170]

[0171] Among them, gray is the grayscale value.

[0172] For the binary map T of the trunk orthographic image bin , along the row direction, in each column according to the binary map T bin Traverse each pixel from bottom to top, record the number of pixels with a value of 1 in each column direction, and after completing the traversal of all columns, construct a set T_heighPixels based on the number of pixels obtained.

[0173] Determine the maximum number of pixels in T_heighPixels, determine the pixel size of the target fruit tree in the pixel size collection S, multiply the maximum value by the pixel size of the target fruit tree, and obtain the trunk height T of the target fruit tree H .

[0174] The canopy height C of the target fruit tree H and trunk height T H Sum and get the height information of the target fruit tree.

[0175] The fruit drop risk Risk is calculated as follows:

[0176]

[0177] In this embodiment, the first flower area set in the canopy orthography is used to predict and estimate the fruits of the target fruit trees based on the conversion relationship between flowers and fruits. Combined with the height information, accurate prediction of the risk of fruit drop is achieved at an early stage, providing sufficient time and data support for the subsequent reasonable management of factors affecting the risk of fruit drop.

[0178] The intelligent prediction method for mango fruit drop hazard based on drone images provided in the embodiment of the present application can be executed by an intelligent prediction device for mango fruit drop hazard based on drone images. In the embodiment of the present application, the intelligent prediction method for mango fruit drop hazard based on drone images is executed by an intelligent prediction device for mango fruit drop hazard based on drone images as an example to illustrate the intelligent prediction device for mango fruit drop hazard based on drone images provided in the embodiment of the present application.

[0179] The embodiment of the present application also provides an intelligent prediction device for mango fruit falling hazards based on drone images.

[0180] like Figure 3 As shown, the intelligent prediction device for mango fruit falling hazards based on drone images includes: an acquisition module 310, a first processing module 320, a second processing module 330 and a third processing module 340.

[0181] The acquisition module 310 is used to acquire a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree and determine the height information of the target fruit tree;

[0182] A first processing module 320 is used to determine a first flower region set of a target fruit tree in the canopy orthographic image;

[0183] The second processing module 330 is used to predict the fruit of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree;

[0184] The third processing module 340 is used to predict the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree.

[0185] According to the intelligent prediction device for mango fruit drop hazards based on drone images provided in the embodiment of the present application, the fruits of the target fruit trees are predicted and estimated based on the conversion relationship between flowers and fruits through the first flower area set in the canopy orthography, and combined with the height information, accurate prediction of the risk of fruit drop is achieved in the early stage, providing sufficient time and data support for the subsequent reasonable management of the factors affecting the risk of fruit drop.

[0186] In some embodiments, the second processing module 330 is further configured to:

[0187] Determine the mean area and the mean diameter of the flower regions in the first flower region set;

[0188] Determine a second flower region set in the first flower region set according to a numerical relationship between the region area and the area mean of each flower region, and a numerical relationship between the region diameter and the diameter mean;

[0189] Fruit prediction information is determined based on the second flower region set.

[0190] In some embodiments, the second processing module 330 is further configured to:

[0191] Determining a third flower region set corresponding to the second flower region set according to the flower loss coefficient of the target fruit tree;

[0192] Fruit prediction information is obtained based on the flower-fruit conversion rate of the target fruit tree and the third flower area set.

[0193] In some embodiments, the third processing module 340 is further configured to:

[0194] According to the fruit prediction information, construct the fruit risk item;

[0195] Based on the fruit risk item and height information, the fruit falling risk is obtained.

[0196] In some embodiments, the acquisition module 310 is further configured to:

[0197] Get the trunk height of the target fruit tree;

[0198] Binarization of canopy orthographic images;

[0199] Perform pixel traversal on the binary canopy orthographic image to obtain the canopy height of the target fruit tree;

[0200] Determine height information based on canopy height and trunk height.

[0201] In some embodiments, the first processing module 320 is further configured to:

[0202] The flower regions in the canopy orthographic image are segmented by a deep semantic segmentation network to obtain a first flower region set, wherein the deep semantic segmentation network is trained by the canopy sample set.

[0203] In some embodiments, the acquisition module 310 is further configured to:

[0204] Acquire multiple fruit tree images of target fruit trees;

[0205] Based on the fruit tree image, the target fruit tree is reconstructed in three dimensions to obtain a three-dimensional model of the fruit tree, so as to determine the orthographic image of the fruit tree three-dimensional model;

[0206] The orthographic image of the fruit tree is segmented to obtain the canopy orthographic image.

[0207] The intelligent prediction device for mango fruit falling hazard based on drone images in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (AugmentedReality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, a super mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (PersonalDigital Assistant, PDA), etc., and can also be a server, a network attached storage (Network AttachedStorage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0208] The intelligent prediction device for mango fruit drop hazard based on drone images in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0209] The intelligent prediction device for mango fruit falling hazard based on drone images provided in the embodiment of the present application can achieve Figure 1 and Figure 2 To avoid repetition, the various processes implemented by the embodiment of the intelligent prediction method for mango fruit falling hazards based on drone images are not repeated here.

[0210] In some embodiments, Figure 4 As shown, the embodiment of the present application also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the embodiment of the intelligent prediction method for mango fruit falling hazards based on drone images is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0211] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0212] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned embodiment of the intelligent prediction method for mango fruit falling hazards based on drone images are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0213] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0214] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned intelligent prediction method for mango fruit falling hazards based on drone images.

[0215] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0216] The embodiment of the present application further provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the intelligent prediction method for mango fruit falling hazards based on drone images, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0217] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0218] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0219] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the intelligent prediction method for mango fruit drop hazards based on drone images of each embodiment of the present application.

[0220] In the description of this application, "first feature" or "second feature" may include one or more of the features.

[0221] In the description of the present application, “plurality” means two or more.

[0222] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0223] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0224] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. An intelligent prediction method for mango fruit drop hazards based on drone images, characterized in that: include: Obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree, and determining height information of the target fruit tree; Determine a first flower region set of the target fruit tree in the canopy orthographic image; Based on the first flower region set, predicting the fruit of the target fruit tree to obtain fruit prediction information of the target fruit tree; Based on the fruit prediction information and the height information of the target fruit tree, the fruit drop risk of the target fruit tree is predicted.

2. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 1, characterized in that: The step of predicting the fruit of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree includes: Determining a mean area value and a mean diameter value of the flower regions in the first flower region set; Determine a second flower region set in the first flower region set according to a numerical relationship between the region area of ​​each flower region and the area mean, and a numerical relationship between the region diameter and the diameter mean; The fruit prediction information is determined according to the second flower region set.

3. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 2, characterized in that: The step of determining the fruit prediction information according to the second flower region set includes: determining a third flower area set corresponding to the second flower area set according to the flower loss coefficient of the target fruit tree; The fruit prediction information is obtained based on the flower-fruit conversion rate of the target fruit tree and the third flower area set.

4. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 1, characterized in that: The predicting the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree comprises: constructing a fruit risk item according to the fruit prediction information; Based on the fruit risk item and the height information, the fruit falling risk is obtained.

5. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 1, characterized in that: The step of determining the height information of the target fruit tree comprises: Obtaining the trunk height of the target fruit tree; Binarization processing is performed on the canopy orthographic image; Perform pixel traversal on the binarized canopy orthographic image to obtain the canopy height of the target fruit tree; The height information is determined according to the canopy height and the trunk height.

6. The method for intelligent prediction of mango fruit falling hazards based on drone images according to any one of claims 1 to 5, characterized in that: Determining the first flower region set of the target fruit tree in the canopy orthographic image comprises: The flower regions in the canopy orthographic image are segmented by a deep semantic segmentation network to obtain the first flower region set, and the deep semantic segmentation network is trained by a canopy sample set.

7. The method for intelligent prediction of mango fruit falling hazards based on drone images according to any one of claims 1 to 5, characterized in that: The step of obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree comprises: Acquire multiple fruit tree images of the target fruit tree; Based on the fruit tree image, the target fruit tree is three-dimensionally reconstructed to obtain the three-dimensional model of the fruit tree, so as to determine the fruit tree orthographic image of the three-dimensional model of the fruit tree; The orthographic image of the fruit tree is segmented to obtain the orthographic image of the canopy.

8. An intelligent prediction device for mango fruit falling hazards based on drone images, characterized in that: include: An acquisition module, used to acquire a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree, and determine height information of the target fruit tree; A first processing module, configured to determine a first flower region set of the target fruit tree in the canopy orthographic image; A second processing module is used to predict the fruit of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree; The third processing module is used to predict the fruit falling risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for intelligently predicting mango fruit falling hazards based on drone images as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for intelligently predicting mango fruit falling hazards based on drone images as described in any one of claims 1 to 7 is implemented.

Images