Method, device and storage medium for predicting fruit number
By using a cascaded neural network model and a cross-loss function update mechanism, the problem of low accuracy in detecting the number of clustered fruits was solved, and efficient and accurate prediction of the number of fruits and the number of clusters was achieved.
Patent Information
- Application Number
- CN202311282658.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2026-08-25
- Estimated Expiration
- 2043-09-28
AI Technical Summary
Existing methods for detecting the number of fruits have low accuracy in predicting clustered fruits such as bananas, lychees, and grapes, and it is difficult to output both the number of fruits and the number of fruit clusters at the same time.
Two cascaded neural network models are used. First, a global detection model is used to obtain global fruit detection boxes, and then a local detection model is used to obtain local fruit detection boxes. The parameters of the two models are updated by using a cross loss function, and the correlation between global and local features is used to improve detection accuracy.
It improves the accuracy of fruit quantity prediction, and can output both fruit quantity and fruit cluster quantity simultaneously, thus enhancing the robustness and accuracy of the detection model.
Smart Images
Figure CN117351275B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of agricultural production technology, and in particular to a method, apparatus, equipment and storage medium for predicting the number of fruits. Background Technology
[0002] Video surveillance technology has been widely used in many fields, among which smart agriculture is an important application area. It can detect the types and quantities of crops based on the video information monitored, which puts forward higher accuracy requirements for image algorithms.
[0003] In actual crop testing, the fruits of many crops, such as bananas, lychees, and grapes, often naturally form clusters. Current detection and classification methods suffer from low accuracy in predicting the number of fruits. Summary of the Invention
[0004] This invention provides a method, apparatus, device, and storage medium for predicting fruit quantity, in order to solve the problem of low accuracy in predicting fruit quantity in the prior art.
[0005] In a first aspect, the present invention provides a method for predicting the number of fruits, the method comprising:
[0006] An input image containing clustered fruits is input into a global detection model, which outputs multiple global fruit detection boxes and the number of fruit clusters.
[0007] The multiple global fruit detection boxes are input into the local detection model, and the local detection model is used to output multiple local fruit detection boxes and the number of fruits.
[0008] A cross-loss function is determined based on the global fruit detection box and the plurality of local fruit detection boxes, and the parameters of the global detection model and the parameters of the local detection model are updated according to the cross-loss function.
[0009] Optionally, the plurality of global fruit detection boxes are input into the local detection model, including:
[0010] For each global fruit detection box corresponding to the input image, scaling is performed based on the size of the global fruit detection box and multiple preset sizes;
[0011] Determine at least one stitched image based on multiple scaled global fruit detection bounding boxes;
[0012] The at least one stitched image is input into the local detection model.
[0013] Optionally, a crossover loss function is determined based on the global fruit detection box and the plurality of local fruit detection boxes, including:
[0014] Cluster the multiple local fruit detection boxes corresponding to the input image to obtain clustered detection boxes;
[0015] Calculate the overlapping area between the clustered detection boxes and the global fruit detection boxes;
[0016] The crossover loss function is determined based on the overlapping region.
[0017] Optionally, determining the crossover loss function based on the overlapping region includes:
[0018] The first probability is determined based on the number of pixels in the overlapping region and the total number of pixels in the input image;
[0019] A second probability is determined based on the number of pixels in the global fruit detection box and the total number of pixels in the input image; a third probability is determined based on the total number of pixels in the multiple local fruit detection boxes and the total number of pixels in the input image.
[0020] Mutual information is determined based on the first probability, the second probability, and the third probability;
[0021] The difference between the target loss function and the mutual information is determined as the cross loss function; when updating the parameters of the global detection model, the target loss function is the loss function determined based on the global fruit detection box and the actual global fruit detection box; when updating the parameters of the local detection model, the target loss function is the loss function determined based on each local fruit detection box and the corresponding actual local fruit detection box.
[0022] Optionally, the method further includes:
[0023] Multiple sample images are acquired. For each sample image, the RGB value of each pixel in the sample image is determined. The K-means method is used to cluster each pixel into K clusters based on the RGB values. The RGB value corresponding to the center point of the target cluster is determined. The target cluster is the cluster with the most pixels among the K clusters.
[0024] When the difference between the RGB value of the center point of the target cluster and the RGB value of the target primary color is less than a threshold, the sample image corresponding to the target cluster is determined to be the input image; the target primary color is the primary color of the fruit.
[0025] Secondly, the present invention provides a method for predicting the number of fruits, the method comprising:
[0026] For any current frame image, determine whether there is a significant target in the current frame image relative to a reference frame image; the reference frame image is the previous frame image of the current frame image.
[0027] If a salient target is present in the current frame image, the current frame image is input into the global detection model; the global detection model is used to output the global fruit detection box and the number of fruit clusters; the global detection model is determined by the method described in any of the first aspects;
[0028] The global fruit detection box is input into the local detection model; the local detection model is used to output multiple local fruit detection boxes and the number of fruits; the local detection model is determined by the method described in any one of the first aspects.
[0029] Optionally, determining whether the current frame image contains a significant target relative to the reference frame image includes:
[0030] Determine multiple pixels in the current frame image that have moved relative to the reference frame image, and determine the foreground target detection box corresponding to the reference frame image;
[0031] Multiple moving pixels are connected by a closing operation to obtain a connected region. The overlapping region between the connected region and the foreground target detection box is determined. The overlapping region is compared with a preset value to determine whether there is a significant target in the current frame image.
[0032] Thirdly, the present invention provides a fruit quantity prediction device, the device comprising:
[0033] The first input module is used to input an input image containing clustered fruits into a global detection model, wherein the global detection model is used to output multiple global fruit detection boxes and the number of fruit clusters; the global detection model is determined by the method described in any one of the first aspects;
[0034] The second input module is used to input the multiple global fruit detection boxes into the local detection model, and the local detection model is used to output multiple local fruit detection boxes and the number of fruits.
[0035] An update module is used to determine a cross-loss function based on the global fruit detection box and the plurality of local fruit detection boxes, and update the parameters of the global detection model and the parameters of the local detection model respectively based on the cross-loss function.
[0036] Fourthly, the present invention provides a fruit quantity prediction device, the device comprising:
[0037] The judgment module is used to determine, for any current frame image, whether there is a significant target relative to a reference frame image; the reference frame image is the previous frame image of the current frame image.
[0038] The third input module is used to input the current frame image into the global detection model if there is a significant target in the current frame image; the global detection model is used to output the global fruit detection box and the number of fruit clusters.
[0039] The fourth input module is used to input the global fruit detection box into the local detection model; the local detection model is used to output multiple local fruit detection boxes and the number of fruits; the local detection model is determined by the method described in any one of the first aspects.
[0040] Fifthly, the present invention provides an electronic device, comprising: at least one processor and a memory;
[0041] The memory stores computer-executed instructions;
[0042] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any of the first aspects.
[0043] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method of any one of the first or second aspects.
[0044] This invention provides a method, apparatus, device, and storage medium for predicting the number of fruits. The method involves inputting an input image containing clustered fruits into a global detection model, which outputs multiple global fruit detection boxes and the number of fruit clusters. These global fruit detection boxes are then input into a local detection model, which outputs multiple local fruit detection boxes and the number of fruits. A cross-loss function is determined based on the global and local fruit detection boxes. The parameters of both the global and local detection models are updated according to this cross-loss function. By cascading the global and local detection models and cross-validating the results of one model against the results of the other, the ability of both models to utilize local and global features is improved, thereby increasing the accuracy of fruit number prediction. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0046] Figure 1 This is an application scenario diagram of a fruit quantity prediction method provided in an embodiment of the present invention;
[0047] Figure 2 A flowchart illustrating a fruit quantity prediction method provided in an embodiment of the present invention;
[0048] Figure 3 This is a schematic diagram illustrating the scaling of a global fruit detection box according to an embodiment of the present invention;
[0049] Figure 4 This is a schematic diagram illustrating the training of two models according to an embodiment of the present invention;
[0050] Figure 5 This is an overall flowchart of a fruit quantity prediction method provided in an embodiment of the present invention;
[0051] Figure 6 A flowchart illustrating another fruit quantity prediction method provided in an embodiment of the present invention;
[0052] Figure 7 This is a schematic diagram of the structure of a fruit quantity prediction device provided in an embodiment of the present invention;
[0053] Figure 8 This is a schematic diagram of another fruit quantity prediction device provided in an embodiment of the present invention;
[0054] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention.
[0055] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0056] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.
[0057] When processing crop images to determine the number of fruits, the common method is to directly detect the number of fruits. However, for clustered fruits such as bananas, lychees, and grapes, which often form a single group, direct fruit count detection may result in low accuracy. Based on clustered fruits, the detected fruit count is related to the information in the global fruit detection bounding box. A higher accuracy in the obtained global fruit detection bounding box (the detection box for the entire cluster of fruits) will improve the accuracy of the detected fruit count. Furthermore, in some scenarios, there is a requirement to output both the number of fruits and the number of fruit clusters simultaneously.
[0058] In existing technologies, when it is necessary to output both the number of fruits and the number of fruit clusters simultaneously, two models can be used to output the number of fruit clusters and the number of fruits respectively for the input image. However, this method has low accuracy. If the same model is used to identify clustered fruits and single fruits as two different categories, a high false detection rate will occur between the classes due to their highly coupled common features, leading to erroneous output results. In addition, the dataset needs to be designed to address the class imbalance problem.
[0059] To address the issue of low accuracy in detecting the number of fruits in existing technologies, this application proposes using two cascaded neural network models. One detection model first acquires a global fruit detection box (corresponding to a cluster of fruits), and then the other detection model acquires a local fruit detection box (corresponding to a single fruit) based on the global fruit detection box. Since the global and local fruit detection boxes are acquired from the same image, the detection results are correlated. The output of one model can be used to update the other model, which helps to improve the accuracy of detecting the number of fruits.
[0060] Figure 1 This is an application scenario diagram of a fruit quantity prediction method provided in an embodiment of the present invention, such as... Figure 1 As shown, when an input image exists, it can first be input into the global detection model to obtain the global fruit detection box, which is the detection box containing the entire cluster of fruits. Then, the global fruit detection box is input into the local detection model to obtain the local fruit detection box and the number of fruits. The local fruit detection box is the detection box of a single fruit. The local detection model can be updated through the global fruit detection box, and the global fruit detection model can be updated through the local fruit detection box, thereby improving the accuracy of the final output number of fruits.
[0061] Figure 2 This is a flowchart illustrating a fruit quantity prediction method provided in an embodiment of the present invention. The method includes steps S201 to S203:
[0062] Step S201: Input the input image containing clustered fruits into the global detection model, which is used to output multiple global fruit detection boxes and the number of fruit clusters.
[0063] When predicting fruit quantity using a neural network model, the model can be trained first. To improve the accuracy of fruit quantity prediction, two neural network models can be set up: a global detection model and a local detection model. The global detection model detects the overall characteristics of the crop (clustered fruits), while the local detection model detects the local characteristics of the crop (individual fruits). A multi-task structured approach is used to design the global and local detection models, and the output of the global detection model is used as the input of the local detection model, allowing the method to simultaneously output the number of fruit clusters and the total number of fruits.
[0064] The neural network model can be a YOLOv5 model, or other object detection models. This model outputs object detection boxes, their corresponding parameter information (center point position, width, and height), and object detection confidence (the probability that the content within the box is the target object). When using a YOLOv5 model, the SPP layer in the neck network can be removed, further reducing model complexity. For example, for a global detection model, the training parameters are: batch-size = 32, epochs = 30, scale = 0.9, and mosaic = 1.
[0065] Based on the above principle, the input image can be first input into the global detection model, and the global detection model can output the global fruit detection box, which is the detection box containing the entire cluster of fruits.
[0066] When training the model, the input images can be multiple images containing clusters of fruit crawled from the web. For example, bananas, grapes, and tomatoes can be selected for fruit quantity prediction. For each crop, 1000 images can be selected as input data.
[0067] Step S202: Input the multiple global fruit detection boxes into the local detection model, and the local detection model is used to output multiple local fruit detection boxes and the number of fruits.
[0068] After obtaining the global fruit detection bounding boxes, these boxes can be input into the local detection model. The local detection model can then output local fruit detection bounding boxes and the number of fruits. The local detection model can be a YOLOv5 model, or other object detection models.
[0069] For example, for a local detection model, the training parameters are: batch-size=32, epochs=20, scale=0.5, mosaic=1.
[0070] Step S203: Determine the cross-loss function based on the global fruit detection box and the plurality of local fruit detection boxes, and update the parameters of the global detection model and the parameters of the local detection model respectively based on the cross-loss function.
[0071] By setting up two detection models, we can obtain a global fruit detection box and multiple local fruit detection boxes. When both models are well trained, the multiple local fruit detection boxes should have a small error with the global fruit detection box after aggregation. Therefore, we can perform cross-validation based on the obtained global fruit detection box and multiple local fruit detection boxes. Cross-validation means updating the local detection model based on the output of the global detection model, and updating the global detection model based on the output of the local detection model.
[0072] Optionally, a cross-loss function can be determined based on the global fruit detection box and the multiple local fruit detection boxes to update the parameters of the global detection model and the local detection model.
[0073] This invention provides a method for predicting the number of fruits. The method involves inputting an input image containing clustered fruits into a global detection model, which outputs multiple global fruit detection boxes and the number of fruit clusters. These global fruit detection boxes are then input into a local detection model, which outputs multiple local fruit detection boxes and the number of fruits. A cross-loss function is determined based on the global and local fruit detection boxes. The parameters of both the global and local detection models are updated according to this cross-loss function. By cascading the global and local detection models and cross-validating the results of one model against the results of the other, the method improves the ability of both models to utilize local and global features, thereby increasing the accuracy of fruit number prediction.
[0074] Optionally, the plurality of global fruit detection boxes are input into the local detection model, including:
[0075] For each global fruit detection box corresponding to the input image, scaling is performed based on the size of the global fruit detection box and multiple preset sizes;
[0076] Determine at least one stitched image based on multiple scaled global fruit detection bounding boxes;
[0077] The at least one stitched image is input into the local detection model.
[0078] After determining multiple global fruit detection boxes using the global detection model, these boxes can be input into the local detection model to improve the local model's effective utilization of global features. However, directly inputting the global fruit detection boxes into the local detection model is equivalent to dividing the input image into blocks, which increases the time complexity of image processing for the local detection model and consumes more time during model training.
[0079] To ensure the utilization of global features while improving the efficiency of local detection models in processing multiple global fruit detection boxes, multiple global fruit detection boxes can be adaptively scaled and rearranged, and the images of multiple global fruit detection boxes can be combined into one or more stitched images.
[0080] Optionally, the input image size for the global detection model can be 640*640, and the input image size for the local detection model can be 320*320. For a global fruit detection box, when scaling, the larger of the length and width of the global fruit detection box can be determined. The length and width of the input image are the same, and the preset size can be 1 / n of the input image size (320). For example, the preset size can be 1 / 2, 1 / 4, 1 / 8, or 1 / 16 of the input image length, meaning there are preset size 1, preset size 2, preset size 3, and preset size 4, etc., with the values decreasing sequentially. After obtaining the larger of the length and width of the global fruit detection box, a target preset size that is close to this larger value can be determined. The global detection box can then be enlarged or reduced to adjust its size to the target preset size.
[0081] After adjusting the size of each global fruit detection box corresponding to an input image to the target preset size, the multiple global fruit detection boxes with adjusted sizes can be stitched together to obtain one or more stitched images, which are then input into the local detection model.
[0082] Figure 3 This is a schematic diagram illustrating the scaling of the global fruit detection box according to an embodiment of the present invention, as shown below. Figure 3 As shown, after obtaining three global fruit detection boxes R1, R2, and R3 for an input image, the three global fruit detection boxes can be scaled to obtain R1', R2', and R3', and then R1', R2', and R3' can be rearranged.
[0083] By using the above stitching method, we can include as many global fruit detection boxes as possible in a single stitched image, reducing the impact of data segmentation on the time complexity of the local detection model and improving the efficiency of the local detection model in processing multiple global fruit detection boxes.
[0084] Optionally, a crossover loss function is determined based on the global fruit detection box and the plurality of local fruit detection boxes, including:
[0085] Cluster the multiple local fruit detection boxes corresponding to the input image to obtain clustered detection boxes;
[0086] Calculate the overlapping area between the clustered detection boxes and the global fruit detection boxes;
[0087] The crossover loss function is determined based on the overlapping region.
[0088] To better utilize global and local features and avoid isolating the relationship between global and local features in the two detection models, cross-validation can be used to optimize the two models.
[0089] For a given image, the global detection model can output a global fruit detection box for that image, while the local detection model can also output multiple local fruit detection boxes for that image. When the global and local detection models are trained, the multiple local fruit detection boxes should have a large overlap with the global fruit detection boxes after clustering. The global detection module and the local detection model can be updated based on this characteristic.
[0090] After acquiring multiple local fruit detection boxes and global fruit detection boxes, the multiple local fruit detection boxes can be clustered to obtain clustered detection boxes. Optionally, during clustering, for each local fruit detection box, the center point of each local fruit detection box is determined. If the distance between two center points is less than a standard preset value, the two local fruit detection boxes are considered adjacent. This process of continuously taking adjacent regions is repeated, and these adjacent regions are merged to obtain the clustered detection boxes.
[0091] After obtaining the clustered detection boxes, the overlapping region between the clustered detection boxes and the global fruit detection boxes can be calculated. Based on this overlapping region, the cross loss function can be determined, and the global detection model and the local detection model can be optimized based on the cross loss function.
[0092] Figure 4 This is a schematic diagram illustrating the training of two models according to an embodiment of the present invention, as shown below. Figure 4As shown, after training the data, the local detection model can be obtained. The local detection model can obtain multiple local fruit detection boxes as detection results. Based on the detection results, ROI clustering and intra-cluster ROI merging can generate global validation data. Based on the global validation data, the output results of the local detection model and the output results of the global detection model, the two models can be cross-validated to update the two models. Repeat the above operation until both models converge, which means that the training of the two models is complete.
[0093] By determining the cross-loss function based on the overlapping region between the clustered detection boxes and the global fruit detection boxes, the model's effective utilization of global and local features can be improved, thereby enhancing the model's optimization performance.
[0094] Optionally, determining the crossover loss function based on the overlapping region includes:
[0095] The first probability is determined based on the number of pixels in the overlapping region and the total number of pixels in the input image;
[0096] A second probability is determined based on the number of pixels in the global fruit detection box and the total number of pixels in the input image; a third probability is determined based on the total number of pixels in the multiple local fruit detection boxes and the total number of pixels in the input image.
[0097] Mutual information is determined based on the first probability, the second probability, and the third probability;
[0098] The difference between the target loss function and the mutual information is determined as the cross loss function; when updating the parameters of the global detection model, the target loss function is the loss function determined based on the global fruit detection box and the actual global fruit detection box; when updating the parameters of the local detection model, the target loss function is the loss function determined based on each local fruit detection box and the corresponding actual local fruit detection box.
[0099] When determining the cross-loss function, it can be calculated based on mutual information.
[0100] Optionally, when training the global detection model, the corresponding target loss function can be calculated first. The difference between the target loss function and the mutual information is the cross-loss function corresponding to the global detection model. The target loss function can be determined by the global fruit detection boxes output by the global detection model and the labeled global fruit detection boxes.
[0101] Optionally, when training the local detection model, the corresponding target loss function can be calculated first. The difference between the target loss function and the mutual information is the cross-loss function corresponding to the local detection model. This target loss function can be determined by the multiple local fruit detection boxes output by the local detection model and the labeled local fruit detection boxes.
[0102] After obtaining the global fruit detection box, multiple local fruit detection boxes, and the overlapping area between the clustered detection boxes and the global fruit detection box for an input image, the mutual information can be calculated using the following formula.
[0103] Where I(X,Y) represents mutual information, p(x,y) represents the ratio of the pixels corresponding to the overlapping region of the clustered detection box and the global fruit detection box to the number of pixels in the image, which is the first probability, p(x) represents the second probability, p(y) represents the third probability, and N represents the number of input images. For each input image, the first product of the second probability and the third probability is calculated, the quotient of the first probability and the first product is calculated, the logarithm of the quotient is taken, the second product of the first probability and the logarithm is calculated, and the sum of the second product results for all input images is calculated. The summation result is used as the mutual information.
[0104]
[0105] Mutual information is calculated by finding the overlapping regions between the clustered detection boxes and the global fruit detection boxes. This can measure the correlation between the detection results of the global detection model and the detection results of the local detection model. The greater the mutual information, the better the two models are trained. Therefore, calculating the cross loss function based on mutual information can improve the optimization effect of the model.
[0106] Optionally, the method further includes:
[0107] Multiple sample images are acquired. For each sample image, the RGB value of each pixel in the sample image is determined. The K-means method is used to cluster each pixel into K clusters based on the RGB values. The RGB value corresponding to the center point of the target cluster is determined. The target cluster is the cluster with the most pixels among the K clusters.
[0108] When the difference between the RGB value of the center point of the target cluster and the RGB value of the target primary color is less than a threshold, the sample image corresponding to the target cluster is determined to be the input image; the target primary color is the primary color of the fruit.
[0109] When determining the input images, the crawled images can be filtered to obtain higher-quality images. Specifically, color clustering can be used to filter the images. Optionally, the RGB values of each pixel in the sample image can be determined first, and then the pixels in a sample image can be clustered using the K-means method to obtain K clusters. Clustering can group pixels with similar colors into one class. The target cluster can be determined first; the target cluster is the cluster with the largest number of pixels among the K clusters. For example, if there are 5 clusters, and the number of pixels in each of the 5 clusters is determined, then the target cluster is the cluster with the largest number of pixels among the 5 clusters.
[0110] For a target cluster, its center point can be determined, along with its RGB value. The difference between the center point's RGB value and the target dominant color's RGB value indicates the relationship between pixels within the cluster. The target dominant color is the dominant color of the fruit. When the difference between the center point's RGB value and the target dominant color's RGB value is small, it means that the colors of all pixels in the cluster are close to the fruit's color, indicating good image quality. Conversely, when the difference is large, it means that the colors of all pixels in the cluster differ significantly from the fruit's color, or that some pixels in the cluster have colors that differ significantly from the fruit's color, indicating poor image quality.
[0111] By clustering colors, higher quality sample images can be obtained, enabling global and local detection models to better distinguish fruits. The clustering operation can also effectively avoid high coupling between fruits and non-fruit parts at the data level.
[0112] Figure 5 This is an overall flowchart of a fruit quantity prediction method provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the solid line section represents the steps for inference based on the trained model. Before inference, the global detection model and the local detection model need to be trained based on the training data. During the training process, cross-validation training is performed using the output results of the global detection model and the output results of the local detection model, and the model is updated. The above operations are repeated until the model converges or reaches the predetermined number of training iterations, which indicates that the two models have been trained.
[0113] This application achieves multi-dimensional yield prediction by setting up two cascaded models to monitor clustered fruits and individual fruits respectively. Furthermore, based on the two cascaded models, cross-validation can be used to train both models, improving the accuracy of model training. In addition, by scaling and rearranging the global fruit detection boxes output by the global detection model, the efficiency of the local detection model in data processing can be improved.
[0114] Figure 6 This is a flowchart illustrating another fruit quantity prediction method provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the method includes:
[0115] Step S601: For any current frame image, determine whether there is a significant target in the current frame image relative to the reference frame image; the reference frame image is the previous frame image of the current frame image.
[0116] Step S602: If there is a significant target in the current frame image, the current frame image is input into the global detection model; the global detection model is used to output the global fruit detection box and the number of fruit clusters.
[0117] Step S603: Input the global fruit detection box into the local detection model; the local detection model is used to output multiple local fruit detection boxes and the number of fruits.
[0118] Once the global detection model and the local detection model have been trained, the two trained models can be used to predict the number of fruits.
[0119] The input data for this application can be surveillance video, which is usually from key monitoring areas where there is a need to monitor crops, such as greenhouses or farmland, in order to predict the number of fruits in the monitored areas.
[0120] Since the input is surveillance video, it can be processed to obtain multiple frames. For each frame, a judgment can be made to determine whether there is a salient target compared to the previous frame. If a salient target is found, it indicates that the current frame is different from the previous frame, and the number of fruits needs to be predicted. For example, when a user picks a bunch of fruits, it means that the image after picking the fruits has a salient target compared to the original image.
[0121] Optionally, the surveillance video can be processed at regular intervals, such as processing the acquired surveillance video daily or at preset intervals to obtain multiple frames of images.
[0122] Once a salient target is identified in the current frame, the frame can be input into a global detection model. This model outputs global fruit detection boxes and the number of fruit clusters. For example, if three fruit clusters exist in a single frame, three global fruit detection boxes and a count of 3 will be output. The global detection model can then monitor and label clusters of fruit.
[0123] After obtaining the global fruit detection bounding box, it can be input into the local detection model to further detect the number of fruits in the frame image. For example, when there are 4 fruits in a fruit cluster, 4 local fruit detection boxes can be used to label the 4 fruits respectively, and the number of fruits, 4, can be output.
[0124] This invention provides a method for predicting the number of fruits. For any given current frame image, it determines whether a salient target exists relative to a reference frame image (the previous frame). If a salient target exists in the current frame image, it is input into a global detection model, which outputs a global fruit detection box and the number of fruit clusters. The global fruit detection box is then input into a local detection model, which outputs multiple local fruit detection boxes and the number of fruits. By determining whether a salient target exists in the current frame image, the input image can be filtered, reducing data processing volume. Simultaneously, the global and local detection models can accurately predict the number of fruit clusters and the total number of fruits.
[0125] Optionally, determining whether the current frame image contains a significant target relative to the reference frame image includes:
[0126] Determine multiple pixels in the current frame image that have moved relative to the reference frame image, and determine the foreground target detection box corresponding to the reference frame image;
[0127] Multiple moving pixels are connected by a closing operation to obtain a connected region. The overlapping region between the connected region and the foreground target detection box is determined. The overlapping region is compared with a preset value to determine whether there is a significant target in the current frame image.
[0128] When determining whether a salient target exists in the current frame relative to a reference frame, we can first identify multiple pixels in the current frame whose positions have changed (i.e., pixels that are moving). The greater the overlap between these moving pixels and the foreground target detection box in the reference frame, the more salient a target is present in the current frame. Specifically, this can be determined using a region of interest overlap detection method. We can calculate the overlapping area between the moving pixels and the foreground target detection box; if the overlapping area is greater than a preset value, it indicates the presence of a salient target in the current frame.
[0129] The process of determining multiple pixels whose positions have changed in the current frame image is as follows:
[0130] First, for each pixel in the reference frame image, a background sample set is established, with a size of n for each background sample set. The pixel values of n neighboring points are selected as the sample values in the background sample set for that point.
[0131] Second, foreground detection is performed based on the background sample set. Each pixel in the current frame image is compared with the corresponding pixel value in the background sample set. For example, if the background sample set contains 20 pixel values, the distance between the current pixel and each pixel in the background sample set is compared. If the distance is less than a first threshold, the number of approximate sample points is increased by 1. When the number of approximate sample points is greater than a second threshold, the pixel is considered a background pixel; otherwise, it is determined to be a foreground pixel, which is the pixel whose position has changed. For example, the first threshold can be 20, and the second threshold can be 10. After determining the background pixels, the background pixels can be set to 0, and the foreground pixels can be set to 255, thus obtaining a segmented binary image containing the moving target.
[0132] Third, when a sudden and drastic change in lighting occurs in an image frame, or when the background suddenly becomes the foreground, some pixels are easily identified as foreground targets. The regions formed by these pixels are called ghosting regions. Therefore, ghosting regions can be processed to eliminate them, that is, removing ghosting pixels from the pixels whose positions have changed. When a region is a ghosting region, since the target is no longer in that position, the magnitude of the average gradient in that region is very small. Therefore, the average gradient of a region is calculated, and if the magnitude of the average gradient is less than a third threshold, the region is deleted. For example, the third threshold can be set to 30. Furthermore, there may be pedestrians in the region. Pedestrians are not the targets we want to track or detect, so the regions containing pedestrians can be deleted. Further, there may be very small contours in the image, which may be noise and also need to be removed.
[0133] Fourth, the pixel values in the background sample set can be updated. For example, after determining that a certain pixel in the current frame is the background, the background sample set can be updated. The update method can be to randomly select a sample in the sample set for updating.
[0134] Fifth, after acquiring multiple pixels in the current frame image that indicate a change in position, these pixels can be compared with the foreground target detection box in the reference frame image to determine whether there is a significant target in the current frame image.
[0135] Optionally, a closing operation can be used to connect multiple pixels in the current frame image to eliminate small holes between pixels representing foreground targets. This essentially reclassifies some pixels that actually belong to the foreground but were not initially identified as background pixels as foreground pixels. After determining the number of pixels in the first connected region and the number of pixels in the foreground target detection box determined based on the reference image, the overlapping region can be determined. The number of pixels in the overlapping region is calculated and compared with a preset value. If the number is greater than the preset value, it indicates that the foreground target has moved, meaning there is a significant target in the current frame image.
[0136] By calculating the number of overlapping pixels between the first pixel of the connected region and the second pixel of the foreground target detection box in the reference frame image, the overlap of the foreground target can be detected, thereby determining whether the foreground target has moved, which can improve the accuracy of detecting the presence of a significant target in the current frame image.
[0137] Figure 7 This is a schematic diagram of a fruit quantity prediction device provided in an embodiment of the present invention. The device 70 includes:
[0138] The first input module 701 is used to input an input image containing clustered fruits into a global detection model, wherein the global detection model is used to output multiple global fruit detection boxes and the number of fruit clusters;
[0139] The second input module 702 is used to input the multiple global fruit detection boxes into the local detection model, and the local detection model is used to output multiple local fruit detection boxes and the number of fruits.
[0140] The update module 703 is used to determine a cross-loss function based on the global fruit detection box and the plurality of local fruit detection boxes, and update the parameters of the global detection model and the parameters of the local detection model respectively based on the cross-loss function.
[0141] Optionally, the second input module 702 is specifically used for:
[0142] For each global fruit detection box corresponding to the input image, scaling is performed based on the size of the global fruit detection box and multiple preset sizes;
[0143] Determine at least one stitched image based on multiple scaled global fruit detection bounding boxes;
[0144] The at least one stitched image is input into the local detection model.
[0145] Optionally, when determining the crossover loss function based on the global fruit detection box and the plurality of local fruit detection boxes, the update module 703 is specifically used for:
[0146] Cluster the multiple local fruit detection boxes corresponding to the input image to obtain clustered detection boxes;
[0147] Calculate the overlapping area between the clustered detection boxes and the global fruit detection boxes;
[0148] The crossover loss function is determined based on the overlapping region.
[0149] Optionally, when determining the crossover loss function based on the overlapping region, the update module 703 is specifically used for:
[0150] The first probability is determined based on the number of pixels in the overlapping region and the total number of pixels in the input image;
[0151] A second probability is determined based on the number of pixels in the global fruit detection box and the total number of pixels in the input image; a third probability is determined based on the total number of pixels in the multiple local fruit detection boxes and the total number of pixels in the input image.
[0152] Mutual information is determined based on the first probability, the second probability, and the third probability;
[0153] The difference between the target loss function and the mutual information is determined as the cross loss function; when updating the parameters of the global detection model, the target loss function is the loss function determined based on the global fruit detection box and the actual global fruit detection box; when updating the parameters of the local detection model, the target loss function is the loss function determined based on each local fruit detection box and the corresponding actual local fruit detection box.
[0154] Optionally, the device further includes: a processing module, used for:
[0155] Multiple sample images are acquired. For each sample image, the RGB value of each pixel in the sample image is determined. The K-means method is used to cluster each pixel into K clusters based on the RGB values. The RGB value corresponding to the center point of the target cluster is determined. The target cluster is the cluster with the most pixels among the K clusters.
[0156] When the difference between the RGB value of the center point of the target cluster and the RGB value of the target primary color is less than a threshold, the sample image corresponding to the target cluster is determined to be the input image; the target primary color is the primary color of the fruit.
[0157] The fruit quantity prediction device provided in this embodiment of the invention can achieve the above-mentioned... Figure 2 The fruit quantity prediction method shown in the embodiment has a similar implementation principle and technical effect, and will not be described again here.
[0158] Figure 8 This is a schematic diagram of another fruit quantity prediction device provided in an embodiment of the present invention. The device 80 includes:
[0159] The judgment module 801 is used to determine, for any current frame image, whether there is a significant target relative to a reference frame image; the reference frame image is the previous frame image of the current frame image.
[0160] The third input module 802 is used to input the current frame image into the global detection model if there is a significant target in the current frame image; the global detection model is used to output the global fruit detection box and the number of fruit clusters.
[0161] The fourth input module 803 is used to input the global fruit detection box into the local detection model; the local detection model is used to output multiple local fruit detection boxes and the number of fruits.
[0162] Optionally, the determination module 801 is specifically used for:
[0163] Determine multiple pixels in the current frame image that have moved relative to the reference frame image, and determine the foreground target detection box corresponding to the reference frame image;
[0164] Multiple moving pixels are connected by a closing operation to obtain a connected region. The overlapping region between the connected region and the foreground target detection box is determined. The overlapping region is compared with a preset value to determine whether there is a significant target in the current frame image.
[0165] The fruit quantity prediction device provided in this embodiment of the invention can achieve the above-mentioned... Figure 6 The fruit quantity prediction method shown in the embodiment has a similar implementation principle and technical effect, and will not be described again here.
[0166] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Figure 9 As shown, the electronic device provided in this embodiment includes at least one processor 901 and a memory 902. The processor 901 and the memory 902 are connected via a bus 903.
[0167] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to execute the method in the above method embodiment.
[0168] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0169] In the above Figure 9 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0170] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.
[0171] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0172] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the above embodiments.
[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the above method embodiments.
[0174] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0175] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0176] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0177] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0179] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for predicting fruit quantity, characterized in that, The method includes: An input image containing clustered fruits is input into a global detection model, which outputs multiple global fruit detection boxes and the number of fruit clusters. The multiple global fruit detection boxes are input into the local detection model, and the local detection model is used to output multiple local fruit detection boxes and the number of fruits. A cross-loss function is determined based on the global fruit detection box and the plurality of local fruit detection boxes, and the parameters of the global detection model and the parameters of the local detection model are updated respectively based on the cross-loss function; The step of determining the cross-loss function based on the global fruit detection box and the plurality of local fruit detection boxes includes: Cluster the multiple local fruit detection boxes corresponding to the input image to obtain clustered detection boxes; Calculate the overlapping area between the clustered detection boxes and the global fruit detection boxes; The first probability is determined based on the number of pixels in the overlapping region and the total number of pixels in the input image; A second probability is determined based on the number of pixels in the global fruit detection box and the total number of pixels in the input image; a third probability is determined based on the total number of pixels in the multiple local fruit detection boxes and the total number of pixels in the input image. Mutual information is determined based on the first probability, the second probability, and the third probability; The difference between the target loss function and the mutual information is determined as the cross loss function; when updating the parameters of the global detection model, the target loss function is the loss function determined based on the global fruit detection box and the actual global fruit detection box; when updating the parameters of the local detection model, the target loss function is the loss function determined based on each local fruit detection box and the corresponding actual local fruit detection box.
2. The method according to claim 1, characterized in that, Inputting the multiple global fruit detection boxes into the local detection model includes: For each global fruit detection box corresponding to the input image, scaling is performed based on the size of the global fruit detection box and multiple preset sizes; Determine at least one stitched image based on multiple scaled global fruit detection boxes; The at least one stitched image is input into the local detection model.
3. The method according to claim 1 or 2, characterized in that, The method further includes: Multiple sample images are acquired. For each sample image, the RGB value of each pixel in the sample image is determined. The K-means method is used to cluster each pixel into K clusters based on the RGB values. The RGB value corresponding to the center point of the target cluster is determined. The target cluster is the cluster with the most pixels among the K clusters. When the difference between the RGB value of the center point of the target cluster and the RGB value of the target primary color is less than a threshold, the sample image corresponding to the target cluster is determined to be the input image; the target primary color is the primary color of the fruit.
4. A method for predicting fruit quantity, characterized in that, The method includes: For any current frame image, determine whether there is a significant target in the current frame image relative to a reference frame image; the reference frame image is the previous frame image of the current frame image. If a significant target exists in the current frame image, the current frame image is input into the global detection model; the global detection model is used to output the global fruit detection box and the number of fruit clusters; the global detection model is determined by the method described in any one of claims 1-3; The global fruit detection box is input into the local detection model; the local detection model is used to output multiple local fruit detection boxes and the number of fruits; the local detection model is determined by the method described in any one of claims 1-3.
5. The method according to claim 4, characterized in that, Determining whether the current frame image contains a salient target relative to the reference frame image includes: Determine multiple pixels in the current frame image that have moved relative to the reference frame image, and determine the foreground target detection box corresponding to the reference frame image; Multiple moving pixels are connected by a closing operation to obtain a connected region. The overlapping region between the connected region and the foreground target detection box is determined. The overlapping region is compared with a preset value to determine whether there is a significant target in the current frame image.
6. A fruit quantity prediction device, characterized in that, The device includes: The first input module is used to input an input image containing clustered fruits into a global detection model, and the global detection model is used to output multiple global fruit detection boxes and the number of fruit clusters. The second input module is used to input the multiple global fruit detection boxes into the local detection model, and the local detection model is used to output multiple local fruit detection boxes and the number of fruits. An update module is used to determine a cross-loss function based on the global fruit detection box and the plurality of local fruit detection boxes, and update the parameters of the global detection model and the parameters of the local detection model respectively based on the cross-loss function; The update module is specifically used to cluster multiple local fruit detection boxes corresponding to the input image to obtain clustered detection boxes; calculate the overlapping region between the clustered detection boxes and the global fruit detection box; determine a first probability based on the number of pixels in the overlapping region and the total number of pixels in the input image; determine a second probability based on the number of pixels in the global fruit detection box and the total number of pixels in the input image; determine a third probability based on the total number of pixels in the multiple local fruit detection boxes and the total number of pixels in the input image; determine mutual information based on the first probability, the second probability, and the third probability; and determine the cross-loss function as the difference between the target loss function and the mutual information. When updating the parameters of the global detection model, the target loss function is a loss function determined based on the global fruit detection box and the actual global fruit detection box; when updating the parameters of the local detection model, the target loss function is a loss function determined based on each local fruit detection box and the corresponding actual local fruit detection box.
7. A fruit quantity prediction device, characterized in that, The device includes: The judgment module is used to determine, for any current frame image, whether there is a significant target relative to a reference frame image; the reference frame image is the previous frame image of the current frame image. The third input module is used to input the current frame image into the global detection model if there is a significant target in the current frame image; the global detection model is used to output the global fruit detection box and the number of fruit clusters; the global detection model is determined by the fruit number prediction method according to any one of claims 1-3; The fourth input module is used to input the global fruit detection box into the local detection model; the local detection model is used to output multiple local fruit detection boxes and the number of fruits; the local detection model is determined by the fruit number prediction method according to any one of claims 1-3.
8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 5.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.