Intelligent Prediction Method and Device for Mango Fruit Drop Hazard Based on UAV Images

Abstract

The present application discloses an intelligent prediction method and device for mango fruit drop hazards based on drone images, belonging to the field of image processing technology. The method includes: obtaining an orthophoto of the canopy of the three-dimensional model of the target fruit tree and determining the height information of the target fruit tree; determining a first flower region set of the target fruit tree in the orthophoto of the canopy; predicting the fruits of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree; predicting the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information. This method can accurately predict the fruit drop risk of fruit trees.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to an intelligent prediction method and device for mango fruit drop hazards based on UAV images. Background Art

[0002] For fruit trees in orchards or green belts, fruits may fall from the trees due to various reasons such as ripening, wind, or precipitation. It is necessary to predict this phenomenon to control the impacts and hazards on the agricultural and ecological systems.

[0003] Currently, mainly by observing the color of the fruits to judge their maturity, and then predicting the fruit drop risk. This method of risk prediction is not accurate. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art. For this purpose, this application proposes an intelligent prediction method and device for mango fruit drop hazards based on UAV images, which can accurately predict the fruit drop risk of fruit trees.

[0005] In a first aspect, this application provides an intelligent prediction method for mango fruit drop hazards based on UAV images, the method comprising:

[0006] Obtain an orthophoto of the canopy of the three-dimensional model of the target fruit tree, and determine the height information of the target fruit tree;

[0007] Determine a first flower region set of the target fruit tree in the orthophoto of the canopy;

[0008] Based on the first flower region set, predict the fruits of the target fruit tree to obtain fruit prediction information of the target fruit tree;

[0009] Based on the fruit prediction information and the height information of the target fruit tree, predict the fruit drop risk of the target fruit tree.

[0010] According to the intelligent prediction method for mango fruit drop hazards based on UAV images of this application, through the first flower region set in the orthophoto of the canopy, according to the conversion relationship between flowers and fruits, the fruits of the target fruit tree are predicted and estimated, and combined with the height information, accurate prediction of the fruit drop risk is achieved in the early stage, providing sufficient time and data support for subsequent reasonable management of factors affecting the fruit drop risk.

[0011] According to an embodiment of this application, the predicting the fruits of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree includes:

[0012] Determine the area mean and diameter mean of the concentrated flower regions in the first flower region set;

[0013] According to the numerical relationship between the area of each flower region and the area mean, and the numerical relationship between the region diameter and the diameter mean, determine a second flower region set in the first flower region set;

[0014] Determine the fruit prediction information according to the second flower region set.

[0015] According to an embodiment of the present application, the determining the fruit prediction information according to the second flower region set includes:

[0016] Determine a third flower region set corresponding to the second flower region set according to the flower loss coefficient of the target fruit tree;

[0017] Based on the flower-fruit conversion rate of the target fruit tree and the third flower region set, obtain the fruit prediction information.

[0018] According to an embodiment of the present application, the predicting the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree includes:

[0019] Construct a fruit risk item according to the fruit prediction information;

[0020] Based on the fruit risk item and the height information, obtain the fruit drop risk.

[0021] According to an embodiment of the present application, the determining the height information of the target fruit tree includes:

[0022] Obtain the trunk height of the target fruit tree;

[0023] Perform binarization processing on the orthophoto of the canopy;

[0024] Perform pixel traversal on the binarized orthophoto of the canopy to obtain the canopy height of the target fruit tree;

[0025] According to the canopy height and the trunk height, determine the height information.

[0026] According to an embodiment of the present application, the determining the first flower region set of the target fruit tree in the orthophoto of the canopy includes:

[0027] Segment the flower regions in the orthophoto of the canopy through a deep semantic segmentation network to obtain the first flower region set, and the deep semantic segmentation network is trained through a canopy sample set.

[0028] According to an embodiment of the present application, obtaining the orthophoto of the canopy of the fruit tree three-dimensional model of the target fruit tree includes:

[0029] Obtaining a plurality of fruit tree images of the target fruit tree;

[0030] Based on the fruit tree images, performing three-dimensional reconstruction on the target fruit tree to obtain the fruit tree three-dimensional model, so as to determine the orthophoto of the fruit tree of the fruit tree three-dimensional model;

[0031] Segmenting the orthophoto of the fruit tree to obtain the orthophoto of the canopy.

[0032] In a second aspect, the present application provides an intelligent prediction device for mango fruit drop hazards based on UAV images, and the device includes:

[0033] An acquisition module, configured to acquire the orthophoto of the canopy of the fruit tree three-dimensional model of the target fruit tree and determine the height information of the target fruit tree;

[0034] A first processing module, configured to determine a first flower region set of the target fruit tree in the orthophoto of the canopy;

[0035] A second processing module, configured to predict the fruits of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree;

[0036] A third processing module, configured to predict the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree.

[0037] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent prediction method for mango fruit drop hazards based on UAV images as described in the first aspect above.

[0038] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the intelligent prediction method for mango fruit drop hazards based on UAV images as described in the first aspect above.

[0039] In a fifth aspect, the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the intelligent prediction method for mango fruit drop hazards based on UAV images as described in the first aspect.

[0040] In a sixth aspect, the present application provides a computer program product, including a computer program which, when executed by a processor, implements the intelligent prediction method for mango fruit drop hazards based on drone images as described in the first aspect above.

[0041] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0042] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:

[0043] Figure 1 is one of the schematic flowcharts of the intelligent prediction method for mango fruit drop hazards based on drone images provided by an embodiment of the present application;

[0044] Figure 2 is another schematic flowchart of the intelligent prediction method for mango fruit drop hazards based on drone images provided by an embodiment of the present application;

[0045] Figure 3 is the schematic structural diagram of the intelligent prediction device for mango fruit drop hazards based on drone images provided by an embodiment of the present application;

[0046] Figure 4 is the schematic structural diagram of the electronic device provided by an embodiment of the present application. Detailed Embodiments

[0047] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application belong to the scope of protection of the present application.

[0048] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0049] The following will, in conjunction with the accompanying drawings, through specific embodiments and their application scenarios, elaborate in detail on the intelligent prediction method for mango fruit drop hazards based on drone images, the intelligent prediction device for mango fruit drop hazards based on drone images, electronic devices, and readable storage media provided by the embodiments of the present application.

[0050] Among them, the intelligent prediction method for mango fruit drop hazards based on drone images can be applied to a terminal, and specifically can be executed by hardware or software in the terminal.

[0051] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with a touch-sensitive surface (for example, a touch screen display and / or a touchpad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (for example, a touch screen display and / or a touchpad).

[0052] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that for the intelligent prediction method for mango fruit drop hazards based on drone images provided by the embodiments of the present application, the execution subject of the intelligent prediction method for mango fruit drop hazards based on drone images can be an electronic device or a functional module or functional entity in the electronic device that can implement the intelligent prediction method for mango fruit drop hazards based on drone images. The electronic devices mentioned in the embodiments of the present application include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices, etc. Hereinafter, taking the electronic device as the execution subject, the intelligent prediction method for mango fruit drop hazards based on drone images provided by the embodiments of the present application will be described.

[0053] As Figure 1 shown, the intelligent prediction method for mango fruit drop hazards based on drone images includes: Step 110 to Step 140.

[0054] Step 110: Obtain an orthographic view of the canopy of the three-dimensional model of the target fruit tree, and determine the height information of the target fruit tree.

[0055] It can be understood that the present application predicts the risk of fruit drop for fruit trees during the flowering period, which can provide important data support for optimizing agricultural management and production plans.

[0056] It should be noted that the target fruit tree can be a fruit tree with flower characteristics during the flowering period. For example, it can be a mango tree, a peach tree, or an apple tree, etc. In the subsequent embodiments of the present application, the mango tree is used as an example to illustrate the solution of the present application, which is not regarded as a limitation on the protection scope of the present application.

[0057] Among them, the orthographic canopy image can be the orthographic projection of the canopy of the target fruit tree in the side view direction at multiple angles, which can cover the all-round characteristics of the canopy of the target fruit tree.

[0058] Among them, the 3D fruit tree model is the 3D model of the target fruit tree, which can be obtained by 3D scanning the target fruit tree.

[0059] The height information can include the height of the target fruit tree in the actual space, which is obtained by measuring the height of the target fruit tree, or can include the canopy height of the target fruit tree, which is obtained by measuring the height from the bottom of the canopy of the target fruit tree to the ground.

[0060] Step 120: Determine the first flower region set of the target fruit tree in the orthographic canopy image.

[0061] Among them, the first flower region set includes all the flower regions in the orthographic canopy image. The flower regions can be achieved through target segmentation and can be used to characterize the flower density of the target fruit tree.

[0062] It can be understood that the orthographic canopy images in multiple orientations have richer spatial information, can more accurately characterize the growth state of the canopy of the target fruit tree, and the obtained first flower region set is more accurate.

[0063] In this step, each pixel in the orthographic canopy image is assigned a class label, which can be flower and non-flower. The flower regions in the orthographic canopy image are segmented, and the set of multiple obtained flower regions is used as the first flower region set.

[0064] Step 130: Predict the fruits of the target fruit tree based on the first flower region set to obtain the fruit prediction information of the target fruit tree.

[0065] It can be understood that the fruits of the target fruit tree are predicted through the first flower region set in the full bloom period, combined with the pollination rate and fruit rate of the flowers, as well as other influencing factors such as water and nutrients.

[0066] In this step, through the first flower region set and the size of a single flower of the target fruit tree, the number of flowers is obtained, and then the number of flowers is multiplied by the pollination rate and the fruit rate, and can also be multiplied by the coefficients of other influencing factors to obtain the fruit prediction information.

[0067] Step 140: Predict the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information.

[0068] Among them, the fruit prediction information can include the predicted number of fruit trees or fruit density of the target fruit tree.

[0069] It is understandable that the higher the fruits of the target fruit tree, the greater the risk of fruit drop, and the greater the number of fruit trees or the fruit density, the greater the risk of fruit drop.

[0070] In actual implementation, based on the numerical relationship between the fruit prediction information and height information and the fruit drop risk, the fruit drop risk of the target fruit tree is predicted according to the fruit prediction information and height information.

[0071] Among them, the fruit drop risk can be used as data support for management measures such as spraying inhibition results and fruit drop warnings, to perform early intervention on the target fruit tree and achieve more effective management of the fruit drop risk.

[0072] In the related art, by observing the color of the fruit and judging the degree of yellowing of the fruit to predict the possible risk of mango drop, this technology can only be used for the prediction in the later stage of fruit growth and cannot provide sufficient time to take corresponding preventive measures.

[0073] According to the intelligent prediction method for mango fruit drop hazards based on drone images provided by the embodiments of the present application, through the first flower region set in the canopy orthophoto, according to the conversion relationship between the flowers and the fruits, the fruits of the target fruit tree are predicted and estimated, and combined with the height information, accurate prediction of the fruit drop risk is achieved in the early stage, providing sufficient time and data support for reasonably managing the factors affecting the fruit drop risk subsequently.

[0074] In some embodiments, obtaining the canopy orthophoto of the three-dimensional model of the target fruit tree includes:

[0075] Obtaining multiple fruit tree images of the target fruit tree;

[0076] Based on the fruit tree images, performing three-dimensional reconstruction on the target fruit tree to obtain a three-dimensional model of the fruit tree, so as to determine the orthophoto of the fruit tree in the three-dimensional model of the fruit tree;

[0077] Segmenting the orthophoto of the fruit tree to obtain the canopy orthophoto.

[0078] Among them, the multiple fruit tree images can be obtained by photographing the target fruit tree during the vigorous period of mango flowers, including the top image and the side image of the target fruit tree, and the resolution of the fruit tree images can distinguish the flower area of the target fruit tree.

[0079] It is understandable that while photographing the images of each mango tree, the image acquisition device can also be calibrated to determine the internal parameters such as the focal length and principal point of the image acquisition device, as well as the external parameters such as the position and direction, as the information carried by the fruit tree images, so as to ensure the accurate mapping relationship between the pixels in the three-dimensional model of the fruit tree and the physical measurement units in the real world during the three-dimensional reconstruction stage.

[0080] In actual implementation, during the peak period of mango blossoms, a drone equipped with an image acquisition device is used to collect high-resolution fruit tree images of each mango tree, and the resolution of the fruit tree images is not less than 0.25 mm.

[0081] To obtain multi-view information and more accurately reconstruct the three-dimensional shape and structure of the target fruit tree, for one mango tree, at least 5 images that can completely cover the mango tree need to be obtained from the side and top directions.

[0082] The three-dimensional reconstruction of a single mango tree can be achieved through the OpenDroneMap software, generating a three-dimensional model of the fruit tree of the mango tree, and obtaining the size mapping relationship between the pixels in each three-dimensional model of the fruit tree and the actual space, and determining the pixel size of each three-dimensional model of the fruit tree, with the unit being cm.

[0083] The three-dimensional reconstruction of all single mango trees is completed in sequence to obtain a collection P of orthophotos of the fruit trees and a collection S of pixel sizes corresponding to all mango trees in the three-dimensional models of the fruit trees.

[0084] Input the orthophoto of the fruit tree of each mango tree in the collection P into the canopy recognition model. The canopy recognition model performs canopy recognition and segmentation on the orthophoto of the fruit tree to obtain the canopy orthophoto and the trunk orthophoto of each mango tree.

[0085] Among them, the canopy recognition model can be obtained by training a neural network model with multiple fruit tree training samples in the fruit tree sample set. The fruit tree training samples can be sample fruit tree orthophotos with canopy orthophoto labels and trunk orthophoto labels.

[0086] The canopy recognition model can be constructed based on a deep semantic segmentation network. For example, the deep semantic segmentation network can be a Feature Reconstruction Network (FR-Net).

[0087] Take the set of canopy orthophotos of each mango tree as C all and take the set of trunk orthophotos of each mango tree as T all .

[0088] In this embodiment, by performing three-dimensional reconstruction on the target fruit tree and segmenting the orthophoto of the fruit tree, a canopy orthophoto can be obtained, providing a basis for obtaining accurate height information.

[0089] In some embodiments, determining a first flower region set of the target fruit tree in the canopy orthophoto includes:

[0090] Using a deep semantic segmentation network to segment the flower regions in the canopy orthophoto to obtain a first flower region set. The deep semantic segmentation network is trained through a canopy sample set.

[0091] It is understandable that the deep semantic segmentation network is used to predict the semantic category to which each pixel belongs, so as to identify and distinguish the boundaries of the categories corresponding to each pixel region in the image.

[0092] Among them, the deep semantic segmentation network can be constructed based on FR-Net and trained by multiple canopy training samples in the canopy sample set. The canopy training samples can be sample canopy orthophotos with flower region labels.

[0093] In actual execution, the canopy orthophoto is input into the deep semantic segmentation network. The deep semantic segmentation network extracts features and fuses semantic information from the canopy orthophoto, and performs semantic segmentation prediction on each pixel to obtain the pixel region corresponding to each flower region and output it. The set of pixel regions corresponding to multiple flower regions is used as the first flower region set F.

[0094] In this embodiment, by segmenting the flower regions in the canopy orthophoto through the deep semantic segmentation network, the geometric shape and position information of the flower regions can be effectively captured, and the segmentation accuracy of the flower regions can be improved.

[0095] In some embodiments, based on the first flower region set, the fruits of the target fruit tree are predicted to obtain the fruit prediction information of the target fruit tree, including:

[0096] Determine the area mean and diameter mean of the flower regions in the first flower region set;

[0097] According to the numerical relationship between the area of each flower region and the area mean, and the numerical relationship between the diameter of each flower region and the diameter mean, determine the second flower region set in the first flower region set;

[0098] Determine the fruit prediction information according to the second flower region set.

[0099] It is understandable that the area of the flower region can be the number of pixels in the flower region. The area mean can be obtained by taking the mean of the number of pixels corresponding to the area of each flower region in the first flower region set. The diameter mean can be obtained by taking the mean of the diameters of each flower region in the first flower region set.

[0100] Among them, the diameter of the flower region can be characterized by the number of pixels. The diameter of the flower region can be the diameter of the largest inscribed circle of the flower region, or can be the diagonal of the smallest circumscribed matrix of the flower region, or the distance between the two farthest points obtained by searching the boundary of the flower region can be used as the diameter of the flower region.

[0101] Due to problems such as overlap between mango flowers, it is necessary to further determine the flower area from the segmented flower area to remove anomalies caused by low segmentation accuracy and overlapping problems.

[0102] The numerical relationship can be a ratio relationship, a difference relationship, a square difference relationship, etc.

[0103] In actual implementation, the area and diameter of each flower region in the first flower region set F are averaged to obtain the area mean M S and mean diameter M D , and calculate the area of ​​each region and the mean area M S The ratio m between the diameter of each area and the mean diameter M is calculated. D The ratio n between them is used to re-determine whether each flower area in the first flower area set is a flower, and the determination result is F f1 , the judgment formula is:

[0104]

[0105] All the judgment results F f1 =m is taken as the second flower region set.

[0106] According to the conversion rate between flowers and fruits of historical mango trees and the second flower region set, the fruit of the target fruit tree can be predicted to obtain the fruit prediction information of the target fruit tree.

[0107] In this embodiment, by determining the second flower area set in the first flower area set, the problem of fewer flowers due to overlapping flowers in the first flower area set can be overcome. At the same time, the situation in which the flower areas segmented by the deep semantic segmentation model are in abnormal shapes such as long strips, which lead to incorrect identification as flowers, can be eliminated.

[0108] In some embodiments, determining fruit prediction information based on the second flower region set includes:

[0109] Determining a third flower region set corresponding to the second flower region set according to the flower loss coefficient of the target fruit tree;

[0110] Fruit prediction information is obtained based on the flower-fruit conversion rate of the target fruit tree and the third flower area set.

[0111] Among them, the flower loss coefficient can be determined based on factors that affect flower loss, such as climate, pests and diseases, and fertilization strategies.

[0112] During the growth of mango trees, affected by external factors such as wind, the flowers will naturally fall off to a certain extent. In order to improve the accuracy of fruit prediction information, it is necessary to calculate the loss of flowers during the growth process.

[0113] For example, taking the flower loss coefficient as 0.5, the flower quantity F of each flower area in the second concentrated flower area after considering the loss is obtained. f2 = 0.5 × F f1 , and the set of F f2 is used as the third flower area set.

[0114] It can be understood that the flower - fruit conversion rate is related to factors such as pollination efficiency, pests and diseases, fertilization strategies, and the variety of the target fruit tree.

[0115] For example, taking the flower - fruit conversion rate as 0.3, the fruit prediction information y corresponding to each flower area in the third concentrated flower area can be obtained. f :

[0116] y f = 0.3 × F f2 + 5

[0117] In this embodiment, through the flower loss coefficient and the flower - fruit conversion rate, the accuracy of the fruit prediction information can be improved.

[0118] In some embodiments, determining the height information of the target fruit tree includes:

[0119] Obtaining the trunk height of the target fruit tree;

[0120] Performing binarization processing on the orthophoto of the canopy;

[0121] Performing pixel traversal on the binarized orthophoto of the canopy to obtain the canopy height of the target fruit tree;

[0122] Determining the height information according to the canopy height and the trunk height.

[0123] Among them, the binarization processing can simplify the orthophoto of the canopy, reduce the amount of data to be processed, and retain the important information in the orthophoto of the canopy.

[0124] In actual execution, the orthophoto of the canopy C of the target fruit tree in the set C all is converted into a grayscale image, and the grayscale value gray of each pixel is obtained. The binarization of the orthophoto of the canopy is achieved by setting a grayscale threshold.

[0125] For example, the binarized image C bin of the orthophoto of the canopy is:

[0126]

[0127] where gray is the grayscale value.

[0128] For the binarized image C bin, along the row direction, traverse each pixel in each column in the order from the bottom to the top of the binarized image C bin , record the number of pixels with a value of 1 in each column direction. After completing the traversal of all columns, construct the set C_heighPixels based on the obtained number of pixels.

[0129] Determine the maximum value of the number of pixels in C_heighPixels, determine the pixel size of the target fruit tree in the pixel size collection S, and multiply the maximum value by the pixel size of the target fruit tree to obtain the canopy height C of the target fruit tree H .

[0130] Convert the orthographic image T of the trunk of the target fruit tree in the set T all into a grayscale image to obtain the grayscale value gray of each pixel, and perform binarization on the orthographic image of the trunk by setting a grayscale threshold.

[0131] For example, the binarized image T of the orthographic image of the trunk is bin as follows:

[0132]

[0133] where gray is the grayscale value.

[0134] For the binarized image T of the orthographic image of the trunk bin , along the row direction, traverse each pixel in each column in the order from the bottom to the top of the binarized image T bin , record the number of pixels with a value of 1 in each column direction. After completing the traversal of all columns, construct the set T_heighPixels based on the obtained number of pixels.

[0135] Determine the maximum value of the number of pixels in T_heighPixels, determine the pixel size of the target fruit tree in the pixel size collection S, and multiply the maximum value by the pixel size of the target fruit tree to obtain the trunk height T of the target fruit tree H .

[0136] Sum the canopy height C H and the trunk height T H of the target fruit tree to obtain the height information of the target fruit tree.

[0137] In this embodiment, by traversing pixels to calculate the canopy height, the obtained height information is more accurate and suitable for fruit trees of more shapes.

[0138] In some embodiments, based on the fruit prediction information and the height information of the target fruit tree, predict the fruit drop risk of the target fruit tree, including:

[0139] Construct a fruit risk term according to the fruit prediction information;

[0140] Based on the fruit risk item and height information, the fruit drop risk is obtained.

[0141] It can be understood that the fruit drop risk of the target fruit tree is positively correlated with the tree height of the target fruit tree. The higher the target fruit tree, the greater the fruit drop risk. The fruit drop risk of the target fruit tree is also related to the quantity and density of the fruits. The greater the quantity or density of the fruits on the target fruit tree, the greater the fruit drop risk.

[0142] Among them, the height information can be T H +C H .

[0143] The fruit drop risk is used to characterize the risk degree of fruit drop of the target fruit tree.

[0144] In actual implementation, according to the fruit prediction information y f , a fruit risk item is constructed

[0145] The fruit drop risk Risk is calculated as follows:

[0146]

[0147] In this embodiment, by considering the two factors of height information and fruit prediction information that have a greater impact on the fruit drop risk, an accurate fruit drop risk can be obtained.

[0148] The following introduces an embodiment.

[0149] As Figure 2 shown, during the peak period of mango flowers, a high-resolution fruit tree image is collected for each mango tree by a drone equipped with an image acquisition device, and the resolution of the fruit tree image is not lower than 0.25 mm.

[0150] In order to obtain multi-view information and more accurately reconstruct the three-dimensional shape and structure of the target fruit tree, for one mango tree, at least 5 images that can completely cover the mango tree need to be obtained from the side and top directions.

[0151] The three-dimensional reconstruction of a single mango tree can be realized by OpenDroneMap software, generating a three-dimensional model of the fruit tree of this mango tree, and obtaining the size mapping relationship between the pixels in each three-dimensional model of the fruit tree and the actual space, and determining the pixel size of each three-dimensional model of the fruit tree, with the unit of cm.

[0152] The three-dimensional reconstruction of all single mango trees is completed in sequence, obtaining a collection P of orthophotos of the fruit trees of all mango trees and a collection S of pixel sizes.

[0153] Input the orthophoto of each mango tree in the collection P into the canopy recognition model. The canopy recognition model performs canopy recognition and segmentation on the orthophoto of the fruit tree to obtain the orthophoto of the canopy and the orthophoto of the tree trunk for each mango tree.

[0154] Take the set of orthophotos of the canopy of each mango tree as C all , and take the set of orthophotos of the tree trunk of each mango tree as T all .

[0155] Input any orthophoto in C all into the deep semantic segmentation network. The deep semantic segmentation network performs feature extraction and semantic information fusion on the orthophoto of the canopy, and performs semantic segmentation prediction on each pixel to obtain and output the pixel region corresponding to each flower region. Take the set of pixel regions corresponding to multiple flower regions as the first flower region set F.

[0156] Calculate the mean value of the area and diameter of each flower region in the first flower region set F respectively to obtain the area mean value M S and the diameter mean value M D , and calculate the ratio m between each regional area and the area mean value M S , calculate the ratio n between each regional diameter and the diameter mean value M D , and re-determine whether each flower region in the first flower region set is a flower. The determination result is F f1 , and the judgment formula is:

[0157]

[0158] Take the set of flower regions where all determination results F f1 =m as the second flower region set.

[0159] Take the flower loss coefficient as 0.5 to obtain the flower quantity F f2 =0.5×F f1 for each flower region in the second flower region set after considering the loss. Take the set of F f2 as the third flower region set.

[0160] Take the flower-fruit conversion rate as 0.3, and the fruit prediction information y f corresponding to each flower region in the third flower region set can be obtained:

[0161] y f =0.3×F f2 +5

[0162] Take the set of orthophotos of the canopy C allThe orthophoto C of the target fruit tree's canopy is converted to a grayscale image to obtain the grayscale value gray of each pixel, and the orthophoto of the canopy is binarized by setting a grayscale threshold.

[0163] For example, the binarized image C of the orthophoto of the canopy bin is:

[0164]

[0165] where gray is the grayscale value.

[0166] For the binarized image C of the orthophoto of the canopy bin , along the row direction, traverse each pixel in each column in the order from the bottom to the top of the binarized image C bin , record the number of pixels with a value of 1 in each column direction. After completing the traversal of all columns, construct the set C_heighPixels based on the obtained number of pixels.

[0167] Determine the maximum value of the number of pixels in C_heighPixels, determine the pixel size of the target fruit tree in the pixel size set S, and multiply the maximum value by the pixel size of the target fruit tree to obtain the canopy height C of the target fruit tree H .

[0168] Convert the orthophoto T of the trunk of the target fruit tree in the set T all of the orthophoto of the trunk to a grayscale image to obtain the grayscale value gray of each pixel, and binarize the orthophoto of the trunk by setting a grayscale threshold.

[0169] For example, the binarized image T of the orthophoto of the trunk bin is:

[0170]

[0171] where gray is the grayscale value.

[0172] For the binarized image T of the orthophoto of the trunk bin , along the row direction, traverse each pixel in each column in the order from the bottom to the top of the binarized image T bin , record the number of pixels with a value of 1 in each column direction. After completing the traversal of all columns, construct the set T_heighPixels based on the obtained number of pixels.

[0173] Determine the maximum value of the number of pixels in T_heighPixels, determine the pixel size of the target fruit tree in the pixel size set S, and multiply the maximum value by the pixel size of the target fruit tree to obtain the trunk height T of the target fruit tree H .

[0174] The canopy height C of the target fruit tree H and the trunk height T H are summed to obtain the height information of the target fruit tree.

[0175] The fruit drop risk Risk is calculated as follows:

[0176]

[0177] In this embodiment, through the first flower region set in the canopy orthophoto, according to the conversion relationship between flowers and fruits, the fruits of the target fruit tree are predicted and estimated, and combined with the height information, accurate prediction of the fruit drop risk is achieved at an early stage, providing sufficient time and data support for subsequent reasonable management of factors affecting the fruit drop risk.

[0178] In the embodiment of the present application, the intelligent prediction method for mango fruit drop hazards based on UAV images, the execution subject can be an intelligent prediction device for mango fruit drop hazards based on UAV images. In the embodiment of the present application, taking the intelligent prediction device for mango fruit drop hazards based on UAV images executing the intelligent prediction method for mango fruit drop hazards based on UAV images as an example, the intelligent prediction device for mango fruit drop hazards based on UAV images provided by the embodiment of the present application is described.

[0179] The embodiment of the present application also provides an intelligent prediction device for mango fruit drop hazards based on UAV images.

[0180] As Figure 3 shown, the intelligent prediction device for mango fruit drop hazards based on UAV images includes: an acquisition module 310, a first processing module 320, a second processing module 330, and a third processing module 340.

[0181] The acquisition module 310 is used to acquire the canopy orthophoto of the three-dimensional model of the target fruit tree and determine the height information of the target fruit tree;

[0182] The first processing module 320 is used to determine the first flower region set of the target fruit tree in the canopy orthophoto;

[0183] The second processing module 330 is used to predict the fruits of the target fruit tree based on the first flower region set to obtain the fruit prediction information of the target fruit tree;

[0184] The third processing module 340 is used to predict the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree.

[0185] According to the intelligent prediction device for mango fruit drop hazards based on UAV images provided by the embodiments of the present application, through the first flower region set in the canopy orthophoto, according to the conversion relationship between flowers and fruits, the fruits of the target fruit tree are predicted and estimated, and combined with the height information, accurate prediction of the fruit drop risk is achieved in the early stage, providing sufficient time and data support for subsequent reasonable management of the factors affecting the fruit drop risk.

[0186] In some embodiments, the second processing module 330 is further configured to:

[0187] Determine the area mean and diameter mean of the flower regions in the first flower region set;

[0188] According to the numerical relationship between the area of each flower region and the area mean, and the numerical relationship between the region diameter and the diameter mean, determine a second flower region set in the first flower region set;

[0189] Determine fruit prediction information according to the second flower region set.

[0190] In some embodiments, the second processing module 330 is further configured to:

[0191] According to the flower loss coefficient of the target fruit tree, determine a third flower region set corresponding to the second flower region set;

[0192] Based on the flower-fruit conversion rate of the target fruit tree and the third flower region set, obtain fruit prediction information.

[0193] In some embodiments, the third processing module 340 is further configured to:

[0194] Construct a fruit risk item according to the fruit prediction information;

[0195] Based on the fruit risk item and the height information, obtain the fruit drop risk.

[0196] In some embodiments, the acquisition module 310 is further configured to:

[0197] Acquire the trunk height of the target fruit tree;

[0198] Perform binarization processing on the canopy orthophoto;

[0199] Perform pixel traversal on the binarized canopy orthophoto to obtain the canopy height of the target fruit tree;

[0200] Determine the height information according to the canopy height and the trunk height.

[0201] In some embodiments, the first processing module 320 is further configured to:

[0202] The flower regions in the orthophoto of the canopy are segmented by a deep semantic segmentation network to obtain a first set of flower regions, and the deep semantic segmentation network is trained by a canopy sample set.

[0203] In some embodiments, the obtaining module 310 is further configured to:

[0204] Obtain multiple fruit tree images of the target fruit tree;

[0205] Based on the fruit tree images, perform three-dimensional reconstruction on the target fruit tree to obtain a three-dimensional model of the fruit tree, so as to determine the orthophoto of the fruit tree in the three-dimensional model of the fruit tree;

[0206] Segment the orthophoto of the fruit tree to obtain the orthophoto of the canopy.

[0207] The intelligent prediction device for mango fruit drop hazards based on UAV images in the embodiments of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an Augmented Reality (AR) / Virtual Reality (VR) device, a robot, a wearable device, an Ultra-Mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), etc., and can also be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0208] The intelligent prediction device for mango fruit drop hazards based on UAV images in the embodiments of the present application can be a device with an operating system. The operating system can be the Android operating system, the iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0209] The intelligent prediction device for mango fruit drop hazards based on UAV images provided in the embodiments of the present application can implement Figure 1 and Figure 2 Each process implemented by the embodiment of the intelligent prediction method for mango fruit drop hazards based on UAV images. To avoid repetition, it will not be elaborated here.

[0210] In some embodiments, such as Figure 4 shown, an embodiment of the present application further provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored on the memory 402 and executable on the processor 401. When the program is executed by the processor 401, it implements each process of the above-mentioned embodiment of the intelligent prediction method for mango fruit drop hazards based on drone images, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0211] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0212] An embodiment of the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the above-mentioned embodiment of the intelligent prediction method for mango fruit drop hazards based on drone images, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0213] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disc, etc.

[0214] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the above-mentioned intelligent prediction method for mango fruit drop hazards based on drone images.

[0215] Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk, or optical disc, etc.

[0216] Another embodiment of the present application provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement each process of the above-mentioned embodiment of the intelligent prediction method for mango fruit drop hazards based on drone images, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0217] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0218] It should be noted that in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0219] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to enable a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the intelligent prediction method for mango fruit drop hazards based on drone images in various embodiments of the present application.

[0220] In the description of the present application, "the first feature", "the second feature" may include one or more of such features.

[0221] In the description of the present application, the meaning of "a plurality of" is two or more.

[0222] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Those of ordinary skill in the art, under the inspiration of the present application and without departing from the purpose of the present application and the scope protected by the claims, can also make many forms, all of which fall within the protection scope of the present application.

[0223] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "schematic embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0224] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. An intelligent prediction method for mango fruit drop hazards based on drone images, characterized in that: include: Obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree, and determining height information of the target fruit tree; Determine a first flower region set of the target fruit tree in the canopy orthographic image, wherein the first flower region set includes all flower regions in the canopy orthographic image; Based on the first flower region set, predicting the fruit of the target fruit tree to obtain fruit prediction information of the target fruit tree; Predicting the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree; The step of predicting the fruit of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree includes: Determining a mean area value and a mean diameter value of the flower regions in the first flower region set; Re-determine whether each flower region in the first flower region set is a flower according to the numerical relationship between the region area of ​​each flower region and the area mean, and the numerical relationship between the region diameter and the diameter mean, and determine a second flower region set in the first flower region set; determining the fruit prediction information according to the second flower region set; The step of determining the fruit prediction information according to the second flower region set includes: Determining the amount of lost flowers corresponding to the second flower area set according to the flower loss coefficient of the target fruit tree; Obtaining the fruit prediction information based on the flower-fruit conversion rate of the target fruit tree and the amount of lost flowers; The predicting the fruit drop risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree comprises: According to the fruit prediction information y f , construct fruit risk items Based on the fruit risk item and the height information, obtaining the fruit falling risk; Fruit drop risk calculation: T H is the trunk height of the target fruit tree, C H is the canopy height of the target fruit tree.

2. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 1, characterized in that: The step of determining the height information of the target fruit tree comprises: Obtaining the trunk height of the target fruit tree; Binarization processing is performed on the canopy orthographic image; Perform pixel traversal on the binarized canopy orthographic image to obtain the canopy height of the target fruit tree; The height information is determined according to the canopy height and the trunk height.

3. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 1 or 2, characterized in that: Determining the first flower region set of the target fruit tree in the canopy orthographic image comprises: The flower regions in the canopy orthographic image are segmented by a deep semantic segmentation network to obtain the first flower region set, and the deep semantic segmentation network is trained by a canopy sample set.

4. The method for intelligent prediction of mango fruit falling hazards based on drone images according to claim 1 or 2, characterized in that: The step of obtaining a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree comprises: Acquire multiple fruit tree images of the target fruit tree; Based on the fruit tree image, the target fruit tree is three-dimensionally reconstructed to obtain the three-dimensional model of the fruit tree, so as to determine the fruit tree orthographic image of the three-dimensional model of the fruit tree; The orthographic image of the fruit tree is segmented to obtain the orthographic image of the canopy.

5. A device for intelligent prediction of mango fruit falling hazards based on drone images according to any one of claims 1 to 4, characterized in that: include: An acquisition module, used to acquire a canopy orthographic image of a three-dimensional fruit tree model of a target fruit tree, and determine height information of the target fruit tree; A first processing module, configured to determine a first flower region set of the target fruit tree in the canopy orthographic image; A second processing module is used to predict the fruit of the target fruit tree based on the first flower region set to obtain fruit prediction information of the target fruit tree; The third processing module is used to predict the fruit falling risk of the target fruit tree based on the fruit prediction information and the height information of the target fruit tree.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for intelligently predicting mango fruit falling hazards based on drone images as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for intelligently predicting mango fruit falling hazards based on drone images as described in any one of claims 1 to 4 is implemented.

Images