A method, system, and equipment for high-accuracy fruit yield estimation

Abstract

This invention discloses a high-accuracy fruit yield estimation method, system, and device, comprising: S1, acquiring distant images and close-up images of the same area within the fruit planting region, and augmenting the distant and close-up images to increase the quantity and diversity of image data; S2, inputting the distant and close-up images into a pre-established yield estimation network model to learn the information correlation between the distant and close-up images, thereby obtaining an estimated fruit yield value; the yield estimation model uses an improved YOLOv5 detection model; S3, inputting the estimated fruit yield value into a pre-established expert system for correction, thereby obtaining an accurate yield estimation result. This invention improves the accuracy of yield prediction by combining distant and close-up yield estimation, considering the relationship between local and global yields, and learning the correlation between them.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method, system, and device for estimating fruit yield with high accuracy. Background Technology

[0002] China is a major fruit producer, and in many regions, the fruit industry has become a pillar industry. Fruit yield estimation is a crucial task in orchard management. Accurate yield estimation helps managers improve fruit quality and better determine the intensity of fruit thinning and the scale of harvesting labor, while also adjusting production management and sales strategies. Current fruit estimation methods simply determine the total number of fruits in a fruit-growing area by identifying fruits in each of multiple images. Furthermore, the fruit identification models lack consideration for multi-scale fusion features of the target fruit, which can lead to biases in the identified fruit count and affect yield estimation. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a highly accurate method and system for estimating fruit yield. By learning the correlation between near and far-view image information, a model is developed to predict the total fruit yield, thereby improving the accuracy of yield prediction.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] This invention provides a highly accurate method for estimating fruit yield, comprising the following steps:

[0006] S1. Collect distant images of the fruit planting area and close-up images of the same area, and augment the distant and close-up images with image data to increase the quantity and diversity of image data;

[0007] S2. Input the distant and close-up images into a pre-established yield estimation network model to learn the information correlation between the distant and close-up images and obtain the estimated yield of the fruit. The yield estimation model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse, and then the head network is used for detection, thereby improving the network performance. The head network is used to predict the type and location of the target, using the features fused by the neck network to make the prediction. The specific improvements are as follows:

[0008] In the backbone network, the lightweight feature extraction network GhostNet is used to replace the original convolutional network;

[0009] An attention mechanism module, ECA, is introduced after the concat operation in the neck network.

[0010] A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure;

[0011] Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function;

[0012] S3. Input the estimated yield of the fruit into a pre-established expert system for correction to obtain an accurate yield estimate.

[0013] As a preferred technical solution, the image data augmentation of the distant and close-up images includes one or more of the following processes: image rotation, image movement, image scaling, image enhancement, noise injection, and contrast transformation.

[0014] As a preferred technical solution, step S2 involves having the yield estimation network model learn the positional correlation between the distant and close-up images, as well as the impact of the number of fruits in the close-up image on the yield of the distant fruits. Specifically:

[0015] S21. The close-up image is detected using the improved YOLOv5 detection model, and the number of predicted boxes output is taken as the number of fruits n in the close-up image, and encoded as a vector n′.

[0016] S22. Pass the distant image and the close-up image through the feature extraction network to generate the feature vector of the distant image and the feature vector of the close-up image respectively; perform a Concat operation with the vector n′ generated by the number of fruits n in the close-up image and the feature vector of the distant image and the feature vector of the close-up image respectively to obtain the feature vector of the distant image and the feature vector of the close-up image with a value of 1*1*2C.

[0017] S23. Input the 1*1*2C feature vector of the distant image and the 1*1*2C feature vector of the near image into the attention mechanism module to learn the information correlation between the distant and near images. The attention mechanism module outputs a vector of size 1*1*C as the information correlation vector, which is then input into an MLP network that includes C input neurons, multiple hidden layers and output neurons. The MLP network learns the relationship between the information correlation vector and the final fruit yield through the hidden layer neurons, and finally outputs an estimated yield value.

[0018] As a preferred technical solution, replacing the original convolutional network with the lightweight feature extraction network GhostNet in the backbone network involves using GhostNet to acquire a large number of feature maps in the backbone network. This not only preserves global features but also reduces the overall network computational load, achieving network lightweighting while maintaining accuracy. The GhostNet convolution calculation formula is as follows:

[0019] Y′=X*f′

[0020] Where X is the input feature map, f′ is the convolutional layer, and Y′ is the set of feature maps; simultaneously, in order to generate n feature maps, a series of linear transformation operations are applied to feature map Y′ to generate n Ghost feature maps, as shown in the formula:

[0021]

[0022] Where, y′ i (i = 1, ..., m) are m feature maps in Y′, Φ i,j It generates Ghost feature map y i,j The linear transformation operation is defined as follows: i represents the i-th feature map in the Y′ feature map set, and j represents the j-th linear operation in the s redundant operations. In addition, the last operation of the linear transformation operation is the identity mapping, which is used to preserve the information of the original feature map.

[0023] As a preferred technical solution, an attention mechanism module (ECA) is introduced after the concat operation in the neck network; the specific process is as follows:

[0024] The input feature map is transformed from a matrix of [h, w, c] into a vector of [1, 1, c] using global average pooling, where h is the width, w is the length, and c is the number of channels. An adaptive one-dimensional convolution kernel size is calculated based on the number of channels in the input feature map. This kernel size is then used in a one-dimensional convolution to obtain the weights for each channel of the feature map. Finally, the normalized weights are multiplied channel-by-channel by the original input feature map to generate a weighted feature map. The calculation formula for the attention mechanism module is as follows:

[0025] ω=σ(C1D K (y))

[0026] Where σ represents the Sigmoid activation function, C1D K This indicates a 1D convolution operation with a kernel size of k, and y represents the aggregated feature.

[0027] As a preferred technical solution, the structure of introducing BiFPN on the basis of the original multi-scale fusion detection module FPN+PAN structure is to add the feature maps of different scales in the backbone network and the feature maps after the Concat operation in the neck network, and then perform convolution operation, which increases the information content of image features.

[0028] As a preferred technical solution, in step S3, the expert system's knowledge base stores the acquired domain knowledge and experience in an appropriate manner, including planting records, weather information, and historical harvest data. Based on the estimated yield value obtained from the yield estimation network model and the image information, the expert system uses the knowledge from the knowledge base to deduce a yield result according to a certain reasoning strategy. This yield result is compared with the yield result obtained from the yield estimation network model using a loss function, thereby updating the parameters of the yield estimation network model to more accurately predict the yield.

[0029] As a preferred technical solution, the method also includes evaluating the prediction results based on the predicted fruit location map and the actual fruit location map, with the specific calculation formula as follows:

[0030]

[0031]

[0032]

[0033] Where TP is the number of fruits detected, FP is the number of fruits that are not fruits but are detected as fruits, FN is the number of fruits that are not detected, P is precision, R is recall, and F1 score is the harmonic mean between precision and recall.

[0034] In another aspect, the present invention provides a high-accuracy fruit yield estimation system, which is applied to the aforementioned high-accuracy fruit yield estimation method, including an image data acquisition module, a yield estimation module, and a yield estimation correction module.

[0035] The image data acquisition module is used to acquire distant images of the fruit planting area and close-up images of the same area, and to augment the distant and close-up images to increase the number and diversity of images.

[0036] The yield estimation module is used to input the distant and close-up images into a pre-established yield estimation network model, learn the information correlation between the distant and close-up images, and obtain the estimated yield value of the fruit. The yield estimation model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse, and then handing them over to the head network for detection, thereby improving the network performance. The head network is used to predict the type and location of the target, using the features fused by the neck network to make the prediction. The specific improvements are as follows:

[0037] In the backbone network, the lightweight feature extraction network GhostNet is used to replace the original convolutional network;

[0038] An attention mechanism module, ECA, is introduced after the concat operation in the neck network.

[0039] A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure;

[0040] Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function;

[0041] The yield estimation correction module is used to input the estimated yield of the fruit into a pre-established expert system for correction, so as to obtain an accurate yield estimation result.

[0042] In another aspect, the present invention provides an electronic device, characterized in that the electronic device comprises:

[0043] At least one processor; and,

[0044] A memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the highly accurate fruit yield estimation method.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] 1. The fruit yield estimation method of the present invention takes into account the relationship between local and global yields by combining near-field and far-field yield estimation, and improves the accuracy of yield prediction by learning the correlation between them.

[0048] 2. The fruit yield estimation method of this invention improves YOLOv5 to suit the characteristics of the fruit detection task. It introduces GhostNet to lightweight the backbone network, achieving network lightweighting while maintaining accuracy. An ECA attention module is introduced to perform feature enhancement after the feature map concat operation, improving the algorithm's detection accuracy without significantly increasing model complexity. In the neck network, the idea of ​​BiFPN is used for feature map skip-connection fusion, increasing the positional information of the feature map after the concat operation and improving the accuracy of small target detection.

[0049] 3. This invention introduces a fruit yield estimation method that combines expert systems with detection algorithms, which saves a lot of manpower costs and helps to monitor fruit growth and predict yield, effectively improving the automation and intelligence level of the fruit industry. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating a highly accurate fruit yield estimation method according to an embodiment of the present invention;

[0052] Figure 2 This is a structural diagram of the improved YOLOv5 network model proposed in this embodiment of the invention;

[0053] Figure 3 This is a diagram of a fruit yield estimation model based on the near-far view combined strategy proposed in an embodiment of the present invention.

[0054] Figure 4 This is an overall schematic diagram of a highly accurate fruit yield estimation method according to an embodiment of the present invention;

[0055] Figure 5 This is a block diagram of a high-accuracy fruit yield estimation system according to an embodiment of the present invention;

[0056] Figure 6 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0058] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0059] Please see Figure 1 , Figure 4 One embodiment of this application provides a method for estimating fruit yield with high accuracy, comprising the following steps:

[0060] S1. Collect distant images of the fruit planting area and close-up images of the same area, and manually estimate the fruit yield of the distant image as the label of the distant image. At the same time, augment the image data of the distant and close-up images to increase the quantity and diversity of image data.

[0061] Furthermore, image data augmentation of the distant and near images includes one or more processing methods such as image rotation, image movement, image scaling, image enhancement, noise injection, and contrast transformation to increase the quantity and diversity of image data and improve the scale of the image dataset.

[0062] Furthermore, the collected raw image data includes fruit images from different time periods, under different lighting conditions, and from different angles. The image types are consistent with the actual situation of fruit formation in the natural environment, including front lighting, back lighting, overlapping, and occlusion.

[0063] The image acquisition device can be a range-measuring image acquisition device, that is, an image acquisition device that can acquire depth information of the image; the image acquisition device can also be a mobile phone, camera, or LiDAR, etc., and the camera can be a binocular camera or other range-measuring camera. In addition, the image acquisition device can be mounted on different devices, such as on the robotic arm of a work vehicle or on a drone.

[0064] S2, please refer to Figure 3The distant and close-up images are input into a pre-established yield estimation network model to learn the information correlation between the distant and close-up images and obtain the estimated yield of the fruit.

[0065] Please see Figure 2 The yield estimation model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network extracts image feature information. The neck network fuses the features extracted by the backbone network, making the features learned by the neck network more diverse and better integrating the features provided by the backbone network. These features are then passed to the subsequent head network for detection, thereby improving the network's performance. The head network predicts the type and location of the target, using the features fused by the neck network to make the prediction. The specific improvements are as follows:

[0066] In the backbone network, the lightweight feature extraction network GhostNet is used to replace the original convolutional network;

[0067] An attention mechanism module, ECA, is introduced after the concat operation in the neck network.

[0068] A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure;

[0069] Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function.

[0070] Furthermore, step S2 involves teaching the yield estimation network model the positional correlation between the foreground and background images, as well as the impact of the number of fruits in the background image on the yield of the fruits in the background. Specifically, the yield estimation network model can infer the yield of the fruits in the background region from a local fruit region in the background image.

[0071] S21. The close-up image is detected using the improved YOLOv5 detection model. The number of predicted boxes output is taken as the number of fruits n in the close-up image and encoded as a 1*1*C vector n′.

[0072] S22. Pass the distant image and the close-up image through a feature extraction network to generate a 1*1*C feature vector for the distant image and a 1*1*C feature vector for the close-up image. Perform a Concat operation with the vector n′ generated from the number of fruits n in the close-up image and the feature vectors of the distant image and the close-up image respectively to obtain a 1*1*2C feature vector for the distant image and a 1*1*2C feature vector for the close-up image.

[0073] S23. Input the 1*1*2C feature vector of the distant image and the 1*1*2C feature vector of the near image into the attention mechanism module to learn the information correlation between the distant and near images. The attention mechanism module outputs a vector of size 1*1*C as the information correlation vector, which is then input into an MLP network that includes C input neurons, 2 hidden layers and 1 output neuron. The MLP network learns the relationship between the information correlation vector and the final fruit yield through the hidden layer neurons, and finally outputs an estimated yield value.

[0074] Furthermore, in step S2, the lightweight feature extraction network GhostNet replaces the original convolutional network in the backbone network. Specifically, GhostNet is used in the backbone network to acquire a large number of feature maps. Based on a set of inherent feature maps, a series of low-cost linear transformations are applied to generate many important feature maps that fully reveal inherent feature information. The original convolution is divided into two steps. The first step is still convolution, but the number of output channels is reduced. The second step uses the feature maps generated in the first step to generate the remaining feature maps through linear transformations. Finally, the output channels of the two steps are combined for output. In this improvement, the standard convolution in the YOLOv5 feature extraction network is replaced with the lightweight GhostNet network. A large number of feature maps can be acquired with a small amount of computation, preserving global features and reducing the overall network computation. GhostNet can make the network lightweight while maintaining accuracy. The calculation formula for GhostNet convolution is as follows:

[0075] Y′=X*f′

[0076] Y′∈R h′w′m ,X∈R hwc ,f′∈R ckkm

[0077] Where X is the input feature map (size parameters are h: width, w: length, c: number of channels), f′ is the convolutional layer (size parameters are c: number of input channels, k: kernel size, m: number of output channels), and Y′ is the feature map set (size parameters are h′, w′, m).

[0078] Meanwhile, in order to generate n feature maps, a series of linear transformation operations are performed on feature map Y′ to generate n Ghost feature maps, as shown in the formula:

[0079]

[0080] Where, y′ i (i = 1, ..., m) are m feature maps in Y′, Φ i,jIt generates Ghost feature map y i,j The linear transformation operation is defined as follows: 'i' represents the i-th feature map in the Y′ feature map set, and 'j' represents the j-th linear operation among 's' redundant operations. Furthermore, the last operation of the linear transformation is an identity mapping, used to preserve information from the original feature map.

[0081] In step S2, the introduction of the attention mechanism module ECA after the concat operation of the neck network specifically involves:

[0082] The input feature map is transformed from a matrix of [h, w, c] into a vector of [1, 1, c] using global average pooling, where h is the width, w is the length, and c is the number of channels. An adaptive one-dimensional convolution kernel size is calculated based on the number of channels in the input feature map. This kernel size is then used in a one-dimensional convolution to obtain the weights for each channel of the feature map. Finally, the normalized weights are multiplied channel-by-channel by the original input feature map to generate a weighted feature map. The calculation formula for the attention mechanism module is as follows:

[0083] ω=σ(C1D K (y)),

[0084] Where σ represents the Sigmoid activation function, C1D K This indicates a 1D convolution operation with a kernel size of k, and y represents the aggregated feature.

[0085] Specifically, to avoid interference from non-detection objects in the detection of target fruits, and to address the issue of decreased detection accuracy due to occlusion and overlapping targets caused by modifications to the backbone network, an efficient channel attention mechanism (ECA) is introduced at a key location in the neck network to further enhance the discriminative ability of the feature representation. In this embodiment, an ECA attention mechanism module is added after each concat operation in the model's neck network, such as... Figure 2 As shown.

[0086] Among them, the channel attention mechanism ECA focuses on the correlation between channels in a deep neural network and is an effective method to improve network performance. Common channel attention mechanisms include SENet and ECANet. The ECA attention mechanism is an improvement on SE and is a local cross-channel interaction strategy that does not reduce dimensionality. It is also a method for adaptively selecting the size of the one-dimensional convolutional kernel. ECA uses the sigmoid function to generate the weight ratio of each feature channel and combines the original H×W×C input features with the channel weights. This assigns large weights to important features in the original features for focused attention, while assigning small weights to ineffective features for autonomous suppression, ultimately obtaining features with channel attention.

[0087] In step S2, the introduction of BiFPN structure on the basis of the original multi-scale fusion detection module FPN+PAN structure involves adding the feature maps of three different scales (20X20, 40X40, 80X80) in the backbone network and the feature map after the concat operation in the neck network, followed by convolution. This ensures that more positional and semantic information is fused without increasing the network complexity too much, thereby increasing the information content of image features. This makes the network more accurate in regressing the target boundary and improves the recognition accuracy of occluded objects.

[0088] In step S2, replacing the Silu activation function of the CBS module in the neck network with the Hardwish activation function specifically involves:

[0089] HardSwish smoothly performs a non-linear difference between ReLU and linear functions; HardSwish has similar properties to Swish because they are both unbounded. Compared to Swish, HardSwish has advantages such as better numerical stability and faster computation speed.

[0090] S3. Input the estimated yield of the fruit into a pre-established planting expert system for correction to obtain an accurate yield estimate.

[0091] Furthermore, the expert system's knowledge base stores domain knowledge and experience acquired from experts in an appropriate manner, including planting records, weather information, and historical harvest data. Based on the estimated yield obtained from the yield estimation network model and image information, the expert system uses the knowledge base to deduce a yield result according to a certain reasoning strategy. This yield result is compared with the yield result obtained from the yield estimation network model using a loss function, thereby updating the parameters of the yield estimation network model to more accurately predict the yield.

[0092] Finally, the trained yield estimation network model is uploaded to the cloud server, and the collected Wogan images are used as input to the yield estimation network model on the cloud server. The network model identifies Wogan fruits in the images, and the yield of Wogan fruits is estimated based on the identification results of Wogan fruits in the images.

[0093] The present invention also includes: evaluating the prediction results based on the predicted fruit location map and the actual fruit location map, wherein the specific calculation formula is as follows:

[0094]

[0095]

[0096]

[0097] Where TP is the number of fruits detected, FP is the number of fruits that are not fruits but are detected as fruits, FN is the number of fruits that are not detected, P is precision, R is recall, and F1 score is the harmonic mean between precision and recall.

[0098] This invention designs a fruit yield estimation network model that combines foreground and background views, enabling it to estimate the global fruit yield based on local information from a background image. An improved YOLOv5 detection model is used to detect the number of fruits in local areas and encode them as yield vectors. An attention mechanism is employed to calculate the positional and yield correlations between the background and background image vectors, which integrate the yield vectors, and an MLP structure is used to predict the final yield value. Simultaneously, an expert system is designed to correct the output of the yield estimation network model. An expert knowledge base is manually constructed by collecting planting records and historical harvest data. The inference engine, based on the output of the estimation network model and the knowledge base, estimates the expected fruit yield for the sampled image scene according to a specific reasoning strategy. The fruit yield derived by the expert system and the fruit yield derived by the estimation network model are compared using a loss function to update the parameters of the estimation network model. This serves as a correction during the learning process of the estimation network model, enabling it to calculate more accurate results.

[0099] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0100] Based on the same idea as the high-accuracy fruit yield estimation method in the above embodiments, the present invention also provides a high-accuracy fruit yield estimation system, which can be used to execute the above-described high-accuracy fruit yield estimation method. For ease of explanation, the structural schematic diagram of an embodiment of the high-accuracy fruit yield estimation system only shows the parts relevant to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0101] Please see Figure 5 In another embodiment of this application, a high-accuracy fruit yield estimation system 100 is provided, which includes an image data acquisition module 101, a yield estimation module 102, and a yield estimation correction module 103.

[0102] The image data acquisition module 101 is used to acquire distant images of the fruit planting area and close-up images of the same area, and to expand the image data of the distant and close-up images to increase the number and diversity of images.

[0103] The yield estimation module 102 is used to input the distant and close-up images into a pre-established yield estimation network model, learn the information correlation between the distant and close-up images, and obtain the estimated yield value of the fruit. The yield estimation model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse, and then hand them over to the subsequent head network for detection, thereby improving the network performance. The head network is used to predict the type and location of the target, and makes a prediction using the features fused by the neck network. Specifically, the improvement is that the lightweight feature extraction network GhostNet is used to replace the original convolutional network in the backbone network.

[0104] An attention mechanism module, ECA, is introduced after the concat operation in the neck network.

[0105] A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure;

[0106] Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function;

[0107] The yield estimation correction module 103 is used to input the estimated yield value of the fruit into a pre-established planting expert system for correction, so as to obtain an accurate yield estimation result.

[0108] It should be noted that the high-accuracy fruit yield estimation system of the present invention corresponds one-to-one with the high-accuracy fruit yield estimation method of the present invention. The technical features and beneficial effects described in the embodiments of the high-accuracy fruit yield estimation method described above are applicable to the embodiments of the high-accuracy fruit yield estimation system. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0109] Furthermore, in the above embodiment of a high-accuracy fruit yield estimation system, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the high-accuracy fruit yield estimation system can be divided into different program modules to complete all or part of the functions described above.

[0110] Please see Figure 6 In one embodiment, an electronic device is provided for implementing a high-accuracy fruit yield estimation method. The electronic device 200 may include a first processor 201, a first memory 202, and a bus. It may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a high-accuracy fruit yield estimation program 203.

[0111] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of a high-accuracy fruit yield estimation program 203, but also to temporarily store data that has been output or will be output.

[0112] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0113] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0114] The deep learning-based citrus fruit picking and positioning program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0115] S1. Collect distant images of the fruit planting area and close-up images of the same area, and augment the distant and close-up images with image data to increase the quantity and diversity of image data;

[0116] S2. The distant and close-up images are input into a pre-established yield estimation network model to learn the information correlation between the distant and close-up images and obtain the estimated yield of the fruit. The yield estimation model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse, and then handing them over to the head network for detection, thereby improving the network performance. The head network is used to predict the type and location of the target, using the features fused by the neck network to make the prediction. The specific improvement is that the lightweight feature extraction network GhostNet is used to replace the original convolutional network in the backbone network.

[0117] An attention mechanism module, ECA, is introduced after the concat operation in the neck network.

[0118] A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure;

[0119] Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function;

[0120] S3. Input the estimated yield of the fruit into a pre-established planting expert system for correction to obtain an accurate yield estimate.

[0121] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0122] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A highly accurate method for estimating fruit yield, characterized in that, Includes the following steps: S1. Collect distant images of the fruit planting area and close-up images of the same area, and augment the distant and close-up images with image data to increase the quantity and diversity of image data; S2. The distant and close-up images are input into a pre-established yield estimation network model to learn the information correlation between the distant and close-up images and obtain the estimated yield of the fruit. The yield estimation network model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse, and then handing them over to the head network for detection, thereby improving the network performance. The head network is used to predict the type and location of the target, making a prediction using the features fused by the neck network. The specific improvements are as follows: In the backbone network, the lightweight feature extraction network GhostNet is used to replace the original convolutional network; An attention mechanism module, ECA, is introduced after the concat operation in the neck network. A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure; Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function; Step S2 involves having the yield estimation network model learn the positional correlation between the distant and close-up images, as well as the impact of the number of fruits in the close-up image on the yield of the distant fruits. Specifically: S21. The close-up image is detected using the improved YOLOv5 detection model, and the number of predicted boxes output is taken as the number of fruits n in the close-up image, and encoded as a vector n′. S22. Pass the distant image and the close-up image through the feature extraction network to generate the feature vector of the distant image and the feature vector of the close-up image respectively; perform a Concat operation with the vector n′ generated by the number of fruits n in the close-up image and the feature vector of the distant image and the feature vector of the close-up image respectively to obtain the feature vector of the distant image and the feature vector of the close-up image with a value of 1*1*2C. S23. Input the 1*1*2C feature vector of the distant image and the 1*1*2C feature vector of the near image into the attention mechanism module to learn the information correlation between the distant and near images. The attention mechanism module outputs a vector of size 1*1*C as the information correlation vector, which is then input into an MLP network that includes C input neurons, multiple hidden layers and output neurons. The MLP network learns the relationship between the information correlation vector and the final fruit yield through the hidden layer neurons, and finally outputs an estimated yield value. S3. Input the estimated yield of the fruit into a pre-established expert system for correction to obtain an accurate yield estimate.

2. The method for estimating fruit yield with high accuracy according to claim 1, characterized in that, The image data augmentation of the distant and close-up images includes one or more of the following processes: image rotation, image movement, image scaling, image enhancement, noise injection, and contrast transformation.

3. The method for estimating fruit yield with high accuracy according to claim 1, characterized in that, The replacement of the original convolutional network with the lightweight feature extraction network GhostNet in the backbone network involves using GhostNet to acquire a large number of feature maps in the backbone network. This not only preserves global features but also reduces the overall network computational cost, achieving a lightweight network while maintaining accuracy. The GhostNet convolution calculation formula is as follows: Y′=X*f′ Where X is the input feature map, f′ is the convolutional layer, and Y′ is the set of feature maps; simultaneously, in order to generate n feature maps, a series of linear transformation operations are applied to feature map Y′ to generate n Ghost feature maps, as shown in the formula: Where, y′ i There are m feature maps in Y′, where i = 1, ..., m, Φ i,j It generates Ghost feature map y i,j The linear transformation operation is defined as follows: i represents the i-th feature map in the Y′ feature map set, and j represents the j-th linear operation in the s redundant operations. In addition, the last operation of the linear transformation operation is the identity mapping, which is used to preserve the information of the original feature map.

4. The method for estimating fruit yield with high accuracy according to claim 1, characterized in that, The process of introducing an attention mechanism module (ECA) after the concat operation in the neck network is as follows: The input feature map is transformed from a matrix of [h, w, c] into a vector of [1, 1, c] using global average pooling, where h is the width, w is the length, and c is the number of channels. An adaptive one-dimensional convolution kernel size is calculated based on the number of channels in the input feature map. This kernel size is then used in a one-dimensional convolution to obtain the weights for each channel of the feature map. Finally, the normalized weights are multiplied channel-by-channel by the original input feature map to generate a weighted feature map. The calculation formula for the attention mechanism module is as follows: ω=σ(C1D K (y)) Where σ represents the Sigmoid activation function, C1D K This indicates a 1D convolution operation with a kernel size of k, and y represents the aggregated feature.

5. The method for estimating fruit yield with high accuracy according to claim 1, characterized in that, The structure that introduces BiFPN on the basis of the original multi-scale fusion detection module FPN+PAN structure adds the feature maps of different scales in the backbone network and the feature maps after the concat operation in the neck network, and then performs a convolution operation, which increases the amount of information in the image features.

6. The method for estimating fruit yield with high accuracy according to claim 1, characterized in that, In step S3, the expert system's knowledge base stores the acquired domain knowledge and experience in an appropriate manner, including planting records, weather information, and historical harvest data. Based on the estimated yield value obtained from the yield estimation network model and the image information, the expert system uses the knowledge base to deduce a yield result according to a certain reasoning strategy. This yield result is compared with the yield result obtained from the yield estimation network model using a loss function, thereby updating the parameters of the yield estimation network model to more accurately predict the yield.

7. The method for estimating fruit yield with high accuracy according to claim 1, characterized in that, It also includes evaluating the prediction results based on the predicted fruit location map and the actual fruit location map, with the specific calculation formula as follows: Where TP is the number of fruits detected, FP is the number of fruits that are not fruits but are detected as fruits, FN is the number of fruits that are not detected, P is precision, R is recall, and F1 score is the harmonic mean between precision and recall.

8. A high-accuracy fruit yield estimation system, characterized in that, A high-accuracy fruit yield estimation method applicable to any one of claims 1-7 includes an image data acquisition module, a yield estimation module, and a yield estimation correction module; The image data acquisition module is used to acquire distant images of the fruit planting area and close-up images of the same area, and to expand the image data of the distant and close-up images to increase the number and diversity of images. The yield estimation module is used to input the distant and close-up images into a pre-established yield estimation network model, learn the information correlation between the distant and close-up images, and obtain the estimated yield value of the fruit. The yield estimation network model uses an improved YOLOv5 detection model, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse, and then hand them over to the head network for detection, thereby improving the network performance. The head network is used to predict the type and location of the target, and makes predictions using the features fused by the neck network. The specific improvements are as follows: In the backbone network, the lightweight feature extraction network GhostNet is used to replace the original convolutional network; An attention mechanism module, ECA, is introduced after the concat operation in the neck network. A BiFPN structure is introduced based on the original multi-scale fusion detection module FPN+PAN structure; Replace the Silu activation function of the CBS module in the neck network with the Hardwish activation function; The yield estimation correction module is used to input the estimated yield of the fruit into a pre-established expert system for correction, so as to obtain an accurate yield estimation result.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform a high-accuracy fruit yield estimation method as described in any one of claims 1-7.

Images