Method and system for counting fruit yield in orchard

Abstract

The application relates to an orchard fruit yield statistical method and system. Fruit orchard videos are frame by frame transmitted into a YOLOV7 detection network to obtain detection box information of fruits in each frame of picture; when the confidence of the detection box is greater than a first confidence threshold, the detection box is tracked; when the confidence of the detection box is greater than a second confidence threshold, the detection box is determined as a high-score detection box; when the confidence of the detection box is less than or equal to the second confidence threshold, the detection box is determined as a low-score detection box; the high-score detection box and the low-score detection box are respectively input into a high-score matching unit and a low-score matching unit, and are matched with trajectories in a trajectory saving module to obtain matching results; according to the matching results, an independent ID is given to fruits on each trajectory; the number of distributed IDs is counted to obtain the yield of the fruits. The application can realize high-precision fruit counting, effectively reduce false detection and reduce repeated counting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of fruit yield statistics, in particular to a method and system for counting fruit yield in orchards. BACKGROUND

[0002] With the rapid development of computer vision and smart agriculture, fruit detection technology has become a research hotspot. This technology has important practical significance for realizing fruit yield statistics, automatic fruit picking and orchard automation management. Fruit yield statistics plays a crucial role in fruit harvesting operation planning and marketing strategy. Accurate statistics of orchard yield before harvesting helps managers allocate labor reasonably according to the fruit distribution in the orchard, and in the future, it helps to deploy picking robot work strategies to efficiently complete the picking task and realize intelligent production in orchards.

[0003] In recent years, deep learning technology has gradually replaced traditional computer vision technology and become the mainstream method for fruit counting tasks. Many scholars have conducted extensive research. There are two key problems in vision-based fruit counting: fruit detection accuracy and repeated counting of the same fruit. SUMMARY

[0004] The purpose of the present application is to provide a method and system for counting fruit yield in orchards, which can realize high-precision fruit counting, effectively reduce false detection and reduce repeated counting.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] A method for counting fruit yield in orchards includes:

[0007] Collecting fruit videos in orchards;

[0008] Transmitting the fruit videos in orchards into YOLOV7 detection network frame by frame to obtain detection box information of fruits in each frame of picture, wherein the detection box information includes the confidence of the detection box;

[0009] Judging whether the confidence of the detection box is greater than a first confidence threshold;

[0010] If the confidence of the detection box is greater than the first confidence threshold, tracking the detection box;

[0011] If the confidence of the detection box is less than or equal to the first confidence threshold, returning to "collecting fruit videos in orchards";

[0012] Judging whether the confidence of the detection box is greater than a second confidence threshold;

[0013] If the confidence of the detection box is greater than the second confidence threshold, determining that the detection box is a high-score detection box;

[0014] if the confidence of the detection box is less than or equal to a second confidence threshold, determining that the detection box is a low-score detection box;

[0015] inputting the high-score detection box into a high-score matching unit and matching the high-score detection box with a track in a track storage module to obtain a high-score matching result;

[0016] inputting the low-score detection box into a low-score matching unit and matching the low-score detection box with a track in the track storage module to obtain a low-score matching result;

[0017] assigning an independent ID to a fruit on each track according to the high-score matching result and the low-score matching result;

[0018] counting the number of the assigned IDs to obtain a yield of the fruits.

[0019] Optionally, the detection box information comprises position information (x, y, w, h), a confidence score (Confidence score), wherein x and y are center coordinates of the detection box, w and h are width and height values of the detection box, and score is a confidence of the target box.

[0020] Optionally, the assigning of the independent ID to the fruit on each track according to the high-score matching result and the low-score matching result specifically comprises:

[0021] determining whether a track without a matched detection box exceeds a set number of frames;

[0022] if yes, deleting the track of the detection box and regarding the target fruit as disappearing in the video;

[0023] if no, keeping the track of the detection box;

[0024] determining whether there is a detection box without a matched track;

[0025] if yes, creating a new track and assigning an independent ID to a fruit on the track;

[0026] if no, directly entering the step of counting the number of the assigned IDs to obtain the yield of the fruits.

[0027] Optionally, the inputting of the high-score detection box into the high-score matching unit and the matching of the high-score detection box with the track in the track storage module to obtain the high-score matching result specifically comprises:

[0028] inputting the high-score detection box into the high-score matching unit, and predicting a possible position of a detection box in a next frame for each frame of picture by Kalman filtering;

[0029] The detection frame of the current frame is matched with the detection frame predicted by Kalman filtering in the last frame through a Hungarian matching algorithm to obtain a high-score matching result.

[0030] An orchard fruit yield statistics system includes:

[0031] A video acquisition module is configured to acquire an orchard fruit video.

[0032] A detection frame information determination module is configured to input the orchard fruit video frame by frame into a YOLOV7 detection network to obtain detection frame information of fruits in each frame of picture, the detection frame information including a confidence of the detection frame.

[0033] A first determination module is configured to determine whether the confidence of the detection frame is greater than a first confidence threshold.

[0034] A first determination result module is configured to track the detection frame when the confidence of the detection frame is greater than the first confidence threshold.

[0035] When the confidence of the detection frame is less than or equal to the first confidence threshold, the method returns to "acquiring the orchard fruit video".

[0036] A second determination module is configured to determine whether the confidence of the detection frame is greater than a second confidence threshold.

[0037] A high-score detection frame determination module is configured to determine that the detection frame is a high-score detection frame when the confidence of the detection frame is greater than the second confidence threshold.

[0038] A low-score detection frame determination module is configured to determine that the detection frame is a low-score detection frame when the confidence of the detection frame is less than or equal to the second confidence threshold.

[0039] A high-score matching module is configured to input the high-score detection frame into a high-score matching unit and match the high-score detection frame with a track in a track storage module to obtain a high-score matching result.

[0040] A low-score matching module is configured to input the low-score detection frame into a low-score matching unit and match the low-score detection frame with a track in the track storage module to obtain a low-score matching result.

[0041] An ID assignment module is configured to assign an independent ID to each fruit on a track according to the high-score matching result and the low-score matching result.

[0042] A fruit yield determination module is configured to count the number of assigned IDs to obtain the yield of the fruits.

[0043] Optionally, the detection frame information comprises position information (x, y, w, h), a confidence score (Confidence score), wherein x and y are center coordinates of the detection frame, w and h are width and height values of the detection frame, and the score is a confidence of the target frame.

[0044] Optionally, the ID assigning module specifically comprises:

[0045] The first judging unit is configured to judge whether the track without matching the detection frame exceeds a set frame number.

[0046] The track deleting unit is configured to delete the track of the detection frame when the track without matching the detection frame exceeds the set frame number, and regard the target fruit as disappearing in the video.

[0047] The track retaining unit is configured to retain the track of the detection frame when the track without matching the detection frame does not exceed the set frame number.

[0048] The second judging unit is configured to judge whether there is a detection frame without matching the track.

[0049] The track creating unit is configured to create a new track when there is a detection frame without matching the track, and assign an independent ID to the fruit on the track.

[0050] When there is no detection frame without matching the track, directly enter the fruit yield determining module.

[0051] Optionally, the high-score matching module specifically comprises:

[0052] The Kalman filtering unit is configured to input the high-score detection frame into the high-score matching unit, and predict a possible position of the detection frame in the next frame for each frame of picture through Kalman filtering.

[0053] The matching unit is configured to match the detection frame of the current frame with the detection frame predicted by the Kalman filtering in the last frame through the Hungarian matching algorithm, and obtain a high-score matching result.

[0054] According to the embodiments of the present application, the following technical effects are provided:

[0055] The application provides an orchard fruit yield statistical method, which comprises collecting orchard fruit video; inputting the orchard fruit video into a YOLOV7 detection network frame by frame to obtain detection box information of fruits in each frame of picture, the detection box information comprising a detection box; judging whether the confidence of the detection box is greater than a first confidence threshold; if the confidence of the detection box is greater than the first confidence threshold, tracking the detection box; if the confidence of the detection box is less than or equal to the first confidence threshold, returning to the step of collecting orchard fruit video; judging whether the confidence of the detection box is greater than a second confidence threshold; if the confidence of the detection box is greater than the second confidence threshold, determining that the detection box is a high-score detection box; if the confidence of the detection box is less than or equal to the second confidence threshold, determining that the detection box is a low-score detection box; inputting the high-score detection box into a high-score matching unit and matching the high-score detection box with a track in a track saving module to obtain a high-score matching result; inputting the low-score detection box into a low-score matching unit and matching the low-score detection box with the track in the track saving module to obtain a low-score matching result; assigning an independent ID to each fruit on a track according to the high-score matching result and the low-score matching result; and counting the number of the assigned IDs to obtain the yield of the fruits. The improved yolov7 network combined with the tracking algorithm can realize high-precision fruit counting, effectively reduce false detection and reduce repeated counting. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only show some of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0057] Figure 1 The orchard fruit yield statistical method flowchart of the present application;

[0058] Figure 2 The orchard fruit yield statistical system structure diagram of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only show some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of the present application.

[0060] The purpose of the present application is to provide an orchard fruit yield statistical method and system, which can realize high-precision fruit counting, effectively reduce false detection and reduce repeated counting.

[0061] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0062] Figure 1 The flow chart of the orchard fruit yield statistical method of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the orchard fruit yield statistical method comprises the following steps:

[0063] Step 101: Collecting an orchard fruit video.

[0064] Step 102: Inputting the orchard fruit video frame by frame into a YOLOV7 detection network to obtain detection box information of fruits in each frame of picture, wherein the detection box information comprises a confidence of the detection box.

[0065] Specifically, the detection box information comprises position information (x, y, w, h), a confidence score (Confidence score), wherein x and y are center coordinates of the detection box, w and h are width and height values of the detection box, and score is the confidence of the target box.

[0066] YOLO algorithm is the most typical representative of one-stage target detection algorithm, which is based on deep neural network for object recognition and positioning, and has very fast running speed and can be used in real-time system. YOLOV7 is the most advanced algorithm in the current YOLO series, which surpasses the previous YOLO series in accuracy and speed.

[0067] Step 103: Judging whether the confidence of the detection box is greater than a first confidence threshold.

[0068] Step 104: If the confidence of the detection box is greater than the first confidence threshold, tracking the detection box.

[0069] If the confidence of the detection box is less than or equal to the first confidence threshold, returning to "collecting an orchard fruit video".

[0070] Step 105: Judging whether the confidence of the detection box is greater than a second confidence threshold.

[0071] Step 106: If the confidence of the detection box is greater than the second confidence threshold, determining that the detection box is a high-score detection box.

[0072] Step 107: If the confidence of the detection box is less than or equal to the second confidence threshold, determining that the detection box is a low-score detection box.

[0073] Step 108: Inputting the high-score detection box into a high-score matching unit and matching the high-score detection box with a track in a track saving module to obtain a high-score matching result, which specifically comprises:

[0074] The high-score detection frame is input into a high-score matching unit, and Kalman filtering is used to predict the possible position of the corresponding detection frame in the next frame for each detection frame in the picture frame;

[0075] The detection frame of the current frame is matched with the detection frame predicted by Kalman filtering of the previous frame through the Hungarian matching algorithm to obtain a high-score matching result.

[0076] The principle of the high-score matching unit and the low-score matching unit is the same, Kalman filtering is used to predict the possible position of the corresponding detection frame in the next frame for each detection frame in the picture frame, and the detection frame of the current frame is matched with the detection frame predicted by Kalman filtering of the previous frame through the Hungarian matching algorithm (to determine the positional relationship between the current detection frame and the predicted detection frame), which is also called IOU matching in the field.

[0077] Step 109: The low-score detection frame is input into a low-score matching unit and matched with the trajectory in the trajectory saving module to obtain a low-score matching result.

[0078] Step 110: An independent ID is assigned to the fruit on each trajectory according to the high-score matching result and the low-score matching result, specifically including:

[0079] It is determined whether the trajectory without matching the detection frame exceeds a set number of frames;

[0080] If yes, the trajectory of the detection frame is deleted, and the target fruit is regarded as disappearing in the video;

[0081] If no, the trajectory of the detection frame is retained.

[0082] It is determined whether there is a detection frame without matching the trajectory;

[0083] If yes, a new trajectory is created, and an independent ID is assigned to the fruit on the trajectory;

[0084] If no, the step of "counting the number of assigned IDs to obtain the yield of fruits" is directly entered.

[0085] After each matching, there will be a trajectory without matching the detection frame and a detection frame without matching the trajectory. When a trajectory exceeds 60 (a value that can be changed as appropriate) frames, the trajectory deletion module will delete the trajectory, i.e., the target fruit is regarded as disappearing in the video. The detection frame without matching the trajectory creates a new trajectory (its position is taken as the initial position of the trajectory) through the trajectory creation module, and when a trajectory exceeds 5 frames (as appropriate), the trajectory is sent to the trajectory saving module. The fruit on each trajectory is assigned an independent ID.

[0086] Step 111: The number of assigned IDs is counted to obtain the yield of fruits.

[0087] The detection algorithm used by the application is the most advanced yolov7 detection algorithm, which has obvious improvement in detection accuracy and detection speed compared with the algorithms used in the field.

[0088] The tracking algorithm used by the application is a more robust tracking algorithm, which almost uses every detection box, unlike previous methods that only retain high-score detection boxes. Therefore, the tracking algorithm used by the application is much higher in counting accuracy than previous algorithms, and since low-score boxes and high-score boxes are reasonably utilized, the obtained tracking trajectory is more coherent, and the ID switching phenomenon (i.e. repeated counting) is greatly reduced.

[0089] Figure 2 The orchard fruit yield statistical system structure diagram of the application. As shown in Figure 2 The orchard fruit yield statistical system includes:

[0090] The video acquisition module 201 is used for acquiring orchard fruit video;

[0091] The detection box information determination module 202 is used for transmitting the orchard fruit video frame by frame into the YOLOV7 detection network to obtain the detection box information of the fruit in each frame picture, and the detection box information includes the confidence of the detection box.

[0092] The first judgment module 203 is used for judging whether the confidence of the detection box is greater than the first confidence threshold;

[0093] The first judgment result module 204 is used for tracking the detection box when the confidence of the detection box is greater than the first confidence threshold;

[0094] When the confidence of the detection box is less than or equal to the first confidence threshold, return to "acquire orchard fruit video";

[0095] The second judgment module 205 is used for judging whether the confidence of the detection box is greater than the second confidence threshold;

[0096] The high-score detection box determination module 206 is used for determining that the detection box is a high-score detection box when the confidence of the detection box is greater than the second confidence threshold;

[0097] The low-score detection box determination module 207 is used for determining that the detection box is a low-score detection box when the confidence of the detection box is less than or equal to the second confidence threshold;

[0098] The high-score matching module 208 is used for inputting the high-score detection box into the high-score matching unit and matching with the track in the track saving module to obtain a high-score matching result;

[0099] The low-score matching module 209 is configured to input the low-score detection frame into a low-score matching unit, match the low-score detection frame with the trajectories in the trajectory storage module, and obtain a low-score matching result.

[0100] The ID assigning module 210 is configured to assign an independent ID to the fruit on each trajectory according to the high-score matching result and the low-score matching result.

[0101] The fruit yield determining module 211 is configured to count the number of the assigned IDs, and determine the yield of the fruit.

[0102] The detection frame information includes position information (x, y, w, h), a confidence score (Confidence score), wherein x and y are the center coordinates of the detection frame, w and h are the width and height values of the detection frame, and score is the confidence of the target frame.

[0103] The ID assigning module 210 specifically includes:

[0104] The first judging unit is configured to judge whether the trajectory without matching the detection frame exceeds a set frame number.

[0105] The trajectory deleting unit is configured to delete the trajectory of the detection frame when the trajectory without matching the detection frame exceeds the set frame number, and regard the target fruit as disappearing in the video.

[0106] The trajectory retaining unit is configured to retain the trajectory of the detection frame when the trajectory without matching the detection frame does not exceed the set frame number.

[0107] The second judging unit is configured to judge whether there is a detection frame without matching the trajectory.

[0108] The trajectory creating unit is configured to create a new trajectory and assign an independent ID to the fruit on the trajectory when there is a detection frame without matching the trajectory.

[0109] When there is no detection frame without matching the trajectory, the fruit yield determining module is directly entered.

[0110] The high-score matching module 208 specifically includes:

[0111] The Kalman filtering unit is configured to input the high-score detection frame into a high-score matching unit, and predict the possible position of the detection frame in the next frame by Kalman filtering for the detection frame in each frame of picture.

[0112] The matching unit is configured to match the detection frame of the current frame with the detection frame predicted by Kalman filtering in the last frame through the Hungarian matching algorithm, and obtain a high-score matching result.

[0113] The various embodiments described in this specification are presented for the purpose of illustrating the principles of the application and its best mode of operation. Each of the embodiments described in this specification has different advantages and disadvantages, and the embodiments disclosed in this specification are not intended to limit the scope of the application to the embodiments disclosed. Embodiments of the system disclosed in this specification are intended to be limited only by the appended claims.

[0114] The principles and implementations of the present application are described in this specification with specific examples. The above description of the embodiments is only to help understand the method of the present application and its core idea. For those skilled in the art, the specific implementation and application scope of the present application can be changed according to the idea of the present application. In summary, the content of this specification should not be understood as a limitation of the present application.

Claims

1. A method for statistically analyzing fruit yield in an orchard, characterized in that, include: Videos of fruit harvesting in orchards; The video of the orchard fruit is fed frame by frame into the YOLOv7 detection network to obtain the detection box information of the fruit in each frame. The detection box information includes the confidence level of the detection box. Determine whether the confidence level of the detection box is greater than the first confidence threshold; If the confidence level of the detection box is greater than the first confidence threshold, the detection box is tracked. If the confidence level of the detection box is less than or equal to the first confidence threshold, return "Collect video of orchard fruit"; Determine whether the confidence level of the detection box is greater than the second confidence threshold; If the confidence level of the detection box is greater than the second confidence threshold, the detection box is determined to be a high-scoring detection box; If the confidence level of the detection box is less than or equal to the second confidence threshold, the detection box is determined to be a low-scoring detection box; The high-resolution detection box is input into the high-resolution matching unit and matched with the trajectory in the trajectory storage module to obtain the high-resolution matching result; The low-scoring detection box is input into the low-scoring matching unit and matched with the trajectory in the trajectory storage module to obtain the low-scoring matching result; Based on the high-score matching results and the low-score matching results, each fruit on the trajectory is assigned a unique ID; Count the total number of IDs to obtain the fruit yield; Assigning a unique ID to each fruit on a trajectory based on the high-score matching result and the low-score matching result specifically includes: Determine if the number of frames for which no detection box was found exceeds a set limit; If so, delete the trajectory of the detection box, and consider the fruit to have disappeared from the video; If not, retain the trajectory of the detection box; Determine if there are any detection boxes that do not match the trajectory; If so, create a new trajectory and assign a unique ID to the fruit on the trajectory; If not, proceed directly to the step of "counting the number of shared IDs and obtaining the fruit yield".

2. The method for statistically analyzing orchard fruit yield according to claim 1, characterized in that, The detection box information includes location information (x, y, w, h) and a confidence score, where x and y are the center coordinates of the detection box, w and h are the width and height values ​​of the detection box, and the score is the confidence score of the target box.

3. The method for statistically analyzing orchard fruit yield according to claim 1, characterized in that, The step of inputting the high-resolution detection box into the high-resolution matching unit and matching it with the trajectory in the trajectory storage module to obtain the high-resolution matching result specifically includes: The high-resolution detection box is input into the high-resolution matching unit, and Kalman filtering is used to predict the possible location of the corresponding detection box in the next frame for each frame of the image. The detection bounding boxes in the current frame are matched with the detection bounding boxes predicted by the Kalman filter in the previous frame using the Hungarian matching algorithm to obtain high-scoring matching results.

4. A fruit yield statistics system for orchards, characterized in that, include: The video capture module is used to capture videos of fruit in the orchard. The detection box information determination module is used to input the orchard fruit video frame by frame into the YOLOv7 detection network to obtain the detection box information of the fruit in each frame image. The detection box information includes the confidence of the detection box. The first judgment module is used to determine whether the confidence level of the detection box is greater than the first confidence level threshold. The first judgment result module is used to track the detection box when the confidence of the detection box is greater than the first confidence threshold. When the confidence level of the detection box is less than or equal to the first confidence threshold, return "Collect video of orchard fruit"; The second judgment module is used to determine whether the confidence level of the detection box is greater than the second confidence level threshold. The high-resolution detection box determination module is used to determine that the detection box is a high-resolution detection box when the confidence level of the detection box is greater than a second confidence threshold. The low-scoring detection box determination module is used to determine that the detection box is a low-scoring detection box when the confidence level of the detection box is less than or equal to a second confidence threshold. The high-scoring matching module is used to input the high-scoring detection box into the high-scoring matching unit and match it with the trajectory in the trajectory storage module to obtain the high-scoring matching result; The low-score matching module is used to input the low-score detection box into the low-score matching unit and match it with the trajectory in the trajectory storage module to obtain the low-score matching result. The ID assignment module is used to assign a unique ID to each fruit on each trajectory based on the high-score matching result and the low-score matching result; The fruit yield determination module is used to count the number of shared IDs and obtain the fruit yield. The ID assignment module specifically includes: The first judgment unit is used to determine whether the trajectory for which no detection box is matched exceeds the set number of frames; The trajectory deletion unit is used to delete the trajectory of the detection box when the number of frames without matching the detection box exceeds a set number, thus treating the fruit as having disappeared from the video. The trajectory retention unit is used to retain the trajectory of the detection box when the number of unmatched trajectories does not exceed a set number of frames. The second judgment unit is used to determine whether there are detection boxes that do not match the trajectory; The trajectory creation unit is used to create a new trajectory when there is a detection box that does not match the trajectory, and assign a unique ID to the fruit on the trajectory; If no unmatched detection box is found, proceed directly to the "Fruit Yield Determination Module".

5. The orchard fruit yield statistics system according to claim 4, characterized in that, The detection box information includes location information (x, y, w, h) and a confidence score, where x and y are the center coordinates of the detection box, w and h are the width and height values ​​of the detection box, and the score is the confidence score of the target box.

6. The orchard fruit yield statistics system according to claim 4, characterized in that, The high-resolution matching module specifically includes: The Kalman filter unit is used to input the high-resolution detection box into the high-resolution matching unit and predict the possible location of the corresponding detection box in the next frame for each frame image through Kalman filtering. The matching unit is used to match the detection boxes of the current frame with the detection boxes predicted by the Kalman filter in the previous frame using the Hungarian matching algorithm to obtain a high-scoring matching result.

Images