A method and system for remote monitoring of real-time yield measurement in orchards

The method and system address real-time and remote orchard yield monitoring by using a multi-target tracking network and improved YOLOv5s network to automatically detect and track fruits, ensuring accurate and efficient yield data transmission in large-scale orchards.

CN114863323BActive Publication Date: 2024-12-06AGRI INFORMATION INST OF CHINESE ACAD OF AGRI SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210388291.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-13
Publication Date
2024-12-06
Estimated Expiration
2042-04-13

AI Technical Summary

Technical Problem

Existing orchard yield measurement technologies face challenges in achieving real-time, accurate, and remote monitoring of fruit yield due to issues with repeated fruit counting, high computational requirements, and lack of a closed-loop system for mobile platforms, making them unsuitable for edge computing and large-scale orchards.

Method used

A method and system utilizing a multi-target tracking network trained with annotated image sequences, combined with a Kalman filter and improved YOLOv5s network, to automatically detect and track fruits in orchard videos, transmitting results via wireless bridges for remote monitoring.

Benefits of technology

Enables real-time, accurate fruit counting and remote transmission of yield data, reducing repeated counting errors and improving tracking performance, suitable for large-scale orchards with efficient data transmission up to several kilometers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114863323B_ABST
    Figure CN114863323B_ABST
Patent Text Reader

Abstract

The present invention proposes a remote monitoring method and system for real-time yield measurement in an orchard, comprising: obtaining a video of fruit trees in an orchard and extracting image frames at intervals to form a sequence image set, using the sequence image set and its annotation information as training samples and labels, training a multi-target tracking network, and obtaining a fruit quantity measurement model; an unmanned vehicle collects orchard videos in the orchard and transmits them to an edge computing terminal in the unmanned vehicle, and automatically detects and tracks fruit targets in the orchard video through a fruit quantity measurement model configured in the edge computing terminal, assigns an ID to the fruit in each frame of the video, and generates an image with a fruit detection frame and a specified ID; and is used to count the total amount of fruit in the orchard as the execution result of the current yield measurement task according to the ID of each fruit. The present invention realizes the automatic measurement of the total amount of fruit under the condition of continuous movement of the orchard mobile platform.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of orchard production, and in particular to a method and system for remote monitoring of real-time production measurement in an orchard. Background Art

[0002] Smart agriculture is becoming increasingly popular in modern agriculture, and intelligent monitoring of fruit trees provides an effective means for precise planting in orchards. Fruit yield is an important indicator of orchard cultivation management, which helps growers arrange fruit harvesting, storage and sales more reasonably. The traditional method of measuring yield mainly relies on manual labor, that is, randomly selecting a fixed percentage of fruit trees in the orchard (such as 5% or 10%) to calculate the number of fruits, thereby inferring the total yield of the entire orchard. However, this manual estimation method is not only time-consuming and labor-intensive, but also may cause counting errors due to human brain fatigue. Therefore, using more advanced machine vision technology to achieve automatic orchard yield measurement has become a more ideal solution.

[0003] In the automatic measurement of orchard yield, the rapid and accurate measurement of the number of fruits is extremely critical. Most existing technologies calculate the number of fruits in a single fruit tree or a local area of ​​a fruit tree by detecting fruit targets in a single image. Since this type of method is prone to the problem of repeated fruit counting, it is difficult to obtain the true fruit yield of the entire orchard; other methods attempt to achieve fruit yield measurement by three-dimensional reconstruction of each fruit tree in the orchard. However, this type of yield measurement method based on three-dimensional reconstruction of fruit trees is not only limited by the effect of three-dimensional reconstruction, but also the large graphics computing load during three-dimensional reconstruction makes it difficult to perform real-time orchard yield measurement tasks.

[0004] The target tracking-based method can effectively solve the problem of identifying the same fruit in video images and avoid the yield measurement error caused by repeated counting of fruits. However, the current general target tracking models are usually large in size and have high requirements for hardware computing resources, and are not easy to be deployed to edge computing terminals for actual orchard scenarios. In addition, there are currently few reports on solutions suitable for remote real-time monitoring of orchard yield measurement. Overall, the existing orchard yield measurement solutions have not formed a closed-loop system from mobile continuous yield measurement to remote transmission and display of yield measurement results. Therefore, how to achieve real-time and accurate measurement of the number of fruits in the orchard and long-distance wireless transmission of yield measurement results during the continuous movement of the orchard mobile platform is a key technical problem that needs to be solved in the current automatic yield measurement in orchards. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a remote monitoring method for real-time yield measurement in an orchard, which is characterized by comprising:

[0006] Step 1, obtain a video of fruit trees in an orchard, extract image frames from the video at intervals to form a sequence image set, annotate each image in the sequence image set, and the annotation information includes the fruit position coordinates in the image and the same fruit targets in adjacent images. Use the sequence image set and its annotation information as training samples and labels to train a multi-target tracking network to obtain a fruit quantity estimation model;

[0007] Step 2: The automatically navigated unmanned vehicle collects orchard videos in the orchard and transmits them to the edge computing terminal in the unmanned vehicle. The fruit quantity estimation model configured in the edge computing terminal automatically detects and tracks the fruit targets in the orchard video, assigns an ID to the fruit in each frame of the orchard video, and generates an image with a fruit detection frame and a specified ID.

[0008] Step 3: According to the ID of each fruit, the total amount of fruit in the orchard is counted as the execution result of the current yield measurement task.

[0009] The orchard real-time yield measurement and remote monitoring method is characterized in that the process of the fruit quantity measurement model tracking the fruit in the video in step 2 includes:

[0010] Let V be the input orchard video, Det be the fruit detector, KF be the Kalman filter, T be the trajectory set of the orchard video, each trajectory of which includes the detection box information and identity information of the fruit target; set three thresholds T high , T low and e. Among them, T high and T low is the confidence threshold of target detection, and e is the confidence threshold of target tracking;

[0011] Step 21, the fruit detector Det adds a detection frame and confidence to the current frame of the orchard video, and the detection confidence is higher than the threshold T high The detection box is classified as D high , D high Contains above threshold T high The position of the detection frame and its identity features; the detection confidence is higher than the threshold T low The detection box is classified as D low , D low Contains above threshold T low The location of the detection box and its identity features;

[0012] Step 22: For the trajectories in the trajectory set T, use the Kalman filter KF to predict the coordinates of the detection box in each trajectory in the current frame;

[0013] Step 23: D high It is associated with all the trajectories in the trajectory set T, and D is obtained by using the intersection-over-union ratio and identity features between the detection boxes. highThe similarity between the detection box and the detection box predicted by the filter is matched using the Hungarian algorithm. The unmatched detection boxes are stored in D remain Stack, unmatched traces, stored in T remain Stack;

[0014] Step 24: The low confidence detection box D low With trajectory T remain Make the association and get D low The similarity between the detection box and the detection box predicted by the filter. The detection box with a similarity less than the threshold is not matched. The unmatched trajectory is stored in T re-remain , delete the unmatched D low Detection frame;

[0015] Step 25: For T re-remain The trajectory in the target is considered to be temporarily lost, but T re-remain The trajectories in will still be retained in the trajectory set T. If T re-remain If the late matching of the middle track is successful or the existence time exceeds the threshold, then the T re-remain and T, otherwise continue to save it in the trajectory set T;

[0016] Step 26: For D remain If the detection confidence of the detection box in is higher than e and survives for more than two frames, it is initialized as a new trajectory and stored in the trajectory set T;

[0017] Step 27: For each frame of the video, output a trajectory set T of all detected and tracked fruit bounding boxes and corresponding IDs in the current frame.

[0018] The orchard real-time yield measurement and remote monitoring method is characterized in that step 3 also includes transmitting the execution result and the image containing the fruit detection frame and the corresponding ID to the remote monitoring module to visualize the yield measurement result image.

[0019] The orchard real-time yield measurement and remote monitoring method is characterized in that the construction process of the fruit detection network in the fruit quantity measurement model includes:

[0020] The path aggregation network in the Neck structure in the YOLOv5s network is removed, and the "CSP2_1+CBL+upsampling" operation is added to the last layer of Neck, and the output of CSP1_1 in the YOLOv5s network backbone network is concatenated. The output is directly connected to the detection branch and identity embedding branch in the FairMOT model.

[0021] Among them, the detection branch contains the candidate target center point heat map, the target bounding box size, and the offset of the target center relative to the actual position of the original image, and the identity embedding branch contains the identity features of the candidate targets in the detection results.

[0022] The present invention also proposes a remote monitoring system for real-time yield measurement in an orchard, which is characterized by comprising:

[0023] A training module is used to obtain a video of fruit trees in an orchard, extract image frames of the video at intervals to form a sequence image set, annotate each image in the sequence image set, and the annotation information includes the position coordinates of the fruit in the image and the same fruit target in adjacent images. The sequence image set and its annotation information are used as training samples and labels to train a multi-target tracking network to obtain a fruit quantity estimation model;

[0024] The acquisition module is used for the unmanned vehicle with automatic navigation to collect orchard videos in the orchard and transmit them to the edge computing terminal in the unmanned vehicle. The fruit quantity estimation model configured in the edge computing terminal is used to automatically detect and track the fruit targets in the orchard video, assign an ID to the fruit in each frame of the orchard video, and generate an image with a fruit detection frame and a specified ID.

[0025] The statistical module is used to count the total amount of fruits in the orchard according to the ID of each fruit as the execution result of the current yield measurement task.

[0026] The orchard real-time yield measurement remote monitoring system is characterized in that the process of the fruit quantity measurement model in the acquisition module tracking the fruit in the video includes:

[0027] Let V be the input orchard video, Det be the fruit detector, KF be the Kalman filter, T be the trajectory set of the orchard video, each trajectory of which includes the detection box information and identity information of the fruit target; set three thresholds T high , T low and e. Among them, T high and T low is the confidence threshold of target detection, and e is the confidence threshold of target tracking;

[0028] The fruit detector Det adds a detection box and confidence to the current frame of the orchard video. The detection confidence is higher than the threshold T high The detection box is classified as D high , D high Contains above threshold T high The position of the detection frame and its identity features; the detection confidence is higher than the threshold T low The detection box is classified as D low , D low Contains above threshold T low The location of the detection box and its identity features;

[0029] For the trajectories in the trajectory set T, the Kalman filter KF is used to predict the coordinates of the detection box in each trajectory in the current frame;

[0030] D high It is associated with all the trajectories in the trajectory set T, and D is obtained by using the intersection-over-union ratio and identity features between the detection boxes. high The similarity between the detection box and the detection box predicted by the filter is matched using the Hungarian algorithm. The unmatched detection boxes are stored in D remain Stack, unmatched traces, stored in T remain Stack;

[0031] The low confidence detection box D low With trajectory T remain Make the association and get D low The similarity between the detection box and the detection box predicted by the filter. The detection box with a similarity less than the threshold is not matched. The unmatched trajectory is stored in T re-remain , delete the unmatched D low Detection frame;

[0032] For T re-remain The trajectory in the target is considered to be temporarily lost, but T re-remain The trajectories in will still be retained in the trajectory set T. If T re-remain If the late matching of the middle track is successful or the existence time exceeds the threshold, then the T re-remain and T, otherwise continue to save it in the trajectory set T;

[0033] For D remain If the detection confidence of the detection box in is higher than e and survives for more than two frames, it is initialized as a new trajectory and stored in the trajectory set T;

[0034] For each frame of the video, output the trajectory set T of all detected and tracked fruit bounding boxes and corresponding IDs in the current frame.

[0035] The orchard real-time yield measurement remote monitoring system is characterized in that the statistical module also includes transmitting the execution result and the image containing the fruit detection frame and the corresponding ID to the remote monitoring module to visualize the yield measurement result image.

[0036] The orchard real-time yield measurement and remote monitoring system is characterized in that the construction process of the fruit detection network in the fruit quantity measurement model includes:

[0037] Remove the path aggregation network (PAN) in the Neck structure in the YOLOv5s network, add a set of "CSP2_1+CBL+upsampling" operations to the last layer of Neck, and concatenate it with the output of CSP1_1 in the YOLOv5s network backbone network (Backbone), and directly connect its output to the detection branch (Detection branch) and identity embedding branch (Identity embedding branch) in the FairMOT model;

[0038] Among them, the detection branch contains the candidate target center point heat map, the target bounding box size, and the offset of the target center relative to the actual position of the original image, and the identity embedding branch contains the identity features of the candidate targets in the detection results.

[0039] The present invention also proposes a storage medium for storing a program for executing any one of the orchard real-time yield measurement and remote monitoring methods.

[0040] The present invention also proposes a client, which is used for any one of the above-mentioned orchard real-time yield measurement and remote monitoring systems.

[0041] It can be seen from the above scheme that the advantages of the present invention are:

[0042] 1. It realizes the automatic calculation of the total amount of fruit and the remote transmission of the result data under the condition of continuous movement of the mobile platform in the orchard, which can solve the problem of remote monitoring in yield measurement while maintaining real-time yield measurement performance.

[0043] 2. Improve the Neck structure of the YOLOv5s network and use the improved YOLOv5s network as the feature extraction network of the fruit quantity measurement model, which has greater advantages in both reasoning performance and model size. At the same time, in the target tracking stage, the high-score detection frame and the low-score detection frame are processed separately to improve the fruit tracking performance of the model and solve the problem of repeated fruit counting caused by fruit tracking failure in yield measurement.

[0044] 3. Combining the Imagezmq-based image data receiving and sending technology with the point-to-point wireless bridge remote transmission technology, the yield measurement results can be transmitted in real time from the orchard to the remote monitoring module several kilometers away, allowing users to grasp the orchard yield measurement information in a timely manner. It is highly efficient and suitable for unmanned automatic yield measurement tasks in large-scale orchards. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a schematic diagram of the composition structure of the orchard real-time yield measurement and remote monitoring system according to an embodiment of the present invention;

[0046] Figure 2 1. It is a schematic diagram of the workflow of measuring yield by the real-time yield measuring module of an orchard according to an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the improved YOLOv5s network structure in the orchard real-time yield measurement module of the embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of a process of generating an image data packet and starting data transmission in a fruit quantity estimation model according to an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of a flow chart of remote monitoring implemented by a computer terminal in a remote monitoring module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention aims to provide a method and system for remote monitoring of orchard yield in real time to solve the problems raised in the above background technology. The present invention provides the following solutions:

[0051] Provided is an orchard real-time yield measurement and remote monitoring system, which includes an orchard real-time yield measurement module and a remote monitoring module. The orchard real-time yield measurement module and the remote monitoring module communicate over long distances via point-to-point wireless bridge technology. The orchard real-time yield measurement module is used for real-time yield measurement and result transmission of fruits at the orchard site. The module mainly includes components such as a monocular camera, an edge computing terminal, a vehicle-mounted wireless AP (Access Point), a high-gain dual-polarization omnidirectional antenna, and an automatic navigation unmanned vehicle. Among them, the monocular camera, the edge computing terminal, the vehicle-mounted wireless AP, and the high-gain dual-polarization omnidirectional antenna are all mounted on the automatic navigation unmanned vehicle. The remote monitoring module is used to receive the yield measurement result data and display it in real time. The module mainly includes components such as a wireless bridge and a computer terminal.

[0052] The orchard real-time yield measurement module is deployed in the orchard scenario. The main steps for implementing yield measurement include:

[0053] First, when the autonomous navigation vehicle is moving, a monocular camera is used to obtain orchard video in real time and transmit it to the edge computing terminal;

[0054] Second, the edge computing terminal is equipped with a trained fruit quantity estimation model. When the model detects that a video is being transmitted, the real-time yield estimation process is started.

[0055] Third, the fruit quantity estimation model reads and processes the video frame by frame, including automatic detection and tracking of fruit targets in consecutive video frames, and assigning a specified ID to the fruits detected in the video frames;

[0056] Fourth, for the processed image, the fruit quantity estimation model fuses the fruit detection frame information and ID information with the original video frame to generate an image with a fruit detection frame and a specified ID, thus forming a yield measurement result image;

[0057] Fifth, use the program module based on Simplejpeg and Imagezmq to encode and compress the measurement result image, generate image data packets and start the data transmission process;

[0058] Sixth, the edge computing terminal and the vehicle-mounted wireless AP are connected to their respective RJ45 ports through an unshielded twisted pair cable. The image data packet is transmitted from the RJ45 port of the edge computing terminal to the vehicle-mounted wireless AP, and the image data packet is sent out in the form of broadcast through the vehicle-mounted wireless AP;

[0059] Seventh, execute steps 2 to 6 repeatedly, and regard the maximum value of the monitored fruit ID as the total amount of fruit in the orchard, that is, the result of this yield measurement task.

[0060] In one embodiment, the specific steps of building the fruit quantity estimation model are as follows:

[0061] S1, collect videos of fruit trees in the orchard and create a dataset for model training;

[0062] S2, optimizing and improving the network structure of the FairMOT model to obtain an improved FairMOT model;

[0063] S3. Use the prepared data set to train the improved FairMOT model to obtain a fruit quantity estimation model.

[0064] Based on the above technical solution, the specific steps of step S2 are as follows:

[0065] S2.1. For the YOLOv5s network, retain its backbone network (Backbone) and improve the Neck structure of YOLOv5s. Use the improved YOLOv5s network as the feature extraction network of FairMOT to improve the model reasoning speed and reduce the model size while maintaining high reasoning accuracy.

[0066] S2.2. Improve the target tracking algorithm of FairMOT and add a special operation to the original target tracking framework to process the high-score frame and the low-score frame after target detection separately. In particular, for the low-confidence bounding box that is difficult to process, the similarity between the low-score detection frame and the tracking trajectory is used to find possible fruit targets from the low-score frame, so as to further restore the fruit target trajectory and improve the fruit tracking effect.

[0067] The remote monitoring module is deployed in the monitoring center scenario. It uses a wireless bridge to receive the yield measurement result data sent by the orchard real-time yield measurement module and transmits it to the computer terminal through the RJ45 port. The computer terminal is deployed with a yield measurement result monitoring program module. The main steps of implementing remote monitoring include:

[0068] First, use the Imagezmq-based program module to connect to the IP address of the orchard real-time yield measurement module and listen to the port number used for data transmission;

[0069] Second, when image data is detected, the image data packet is received and decoded to obtain the measurement result image;

[0070] Third, visualize the yield measurement result images;

[0071] Fourth, loop the second to third steps and write the received images into the specified video file for subsequent viewing.

[0072] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0073] See also Figure 1 As shown, an orchard real-time production measurement and remote monitoring system provided by an embodiment of the present invention includes an orchard real-time production measurement module and a remote monitoring module. The orchard real-time production measurement module and the remote monitoring module communicate over long distances through point-to-point wireless bridge technology, and the effective communication transmission distance is within the range of 3 to 5 kilometers. The orchard real-time production measurement module is used for real-time production measurement and result transmission at the orchard site. The module mainly includes components such as a monocular camera, an edge computing terminal (such as Jetson AGXXavier), a vehicle-mounted wireless AP (operating in the 5GHz frequency band), a high-gain dual-polarized omnidirectional antenna, and an automatic navigation unmanned vehicle. Among them, the monocular camera, the edge computing terminal, the vehicle-mounted wireless AP, and the high-gain dual-polarized omnidirectional antenna are all mounted on the automatic navigation unmanned vehicle. The remote monitoring module is used to receive the production measurement result data and display it in real time. The module mainly includes components such as a wireless bridge (operating in the 5GHz frequency band) and a computer terminal (which can be a desktop computer or a laptop).

[0074] See also Figure 2 As shown in the figure, the orchard real-time yield measurement module is deployed in the orchard scenario. The main steps for implementing yield measurement include:

[0075] Step A: When the automatic navigation unmanned vehicle is moving, a monocular camera is used to obtain the orchard video in real time and transmit it to the edge computing terminal;

[0076] Step B: The edge computing terminal is configured with a trained fruit quantity estimation model. When the model detects that a video is being input, the real-time yield estimation process is started;

[0077] Step C: The fruit quantity estimation model reads and processes the video frame by frame, including automatic detection and tracking of fruit targets in consecutive video frames, and assigning a designated ID to the fruit detected in the video frame;

[0078] Step D: for the processed image, the fruit quantity estimation model fuses the fruit detection frame information and ID information with the original video frame to generate an image with a fruit detection frame and a detection frame specified ID, thereby forming a yield measurement result image;

[0079] Step E: Encode and compress the measurement result image using a program module based on Simplejpeg and Imagezmq, generate an image data packet and start the data sending process;

[0080] Step F: The edge computing terminal and the vehicle-mounted wireless AP are connected to their respective RJ45 ports through an unshielded twisted pair cable. The image data packet is transmitted from the RJ45 port of the edge computing terminal to the vehicle-mounted wireless AP, and the image data packet is sent out in the form of broadcast through the vehicle-mounted wireless AP.

[0081] Step G, looping through steps B to F, and taking the maximum value of the monitored fruit ID as the total amount of fruit in the orchard, that is, the result of this yield measurement task.

[0082] On the basis of the above technical solution, the specific steps of building the fruit quantity estimation model in step B are as follows:

[0083] Step B1: Collect videos of fruit trees in the orchard to create a data set for model training;

[0084] Step B2, optimizing and improving the network structure of the FairMOT model to obtain an improved FairMOT model;

[0085] Step B3: Use the prepared data set to train the improved FairMOT model to obtain a fruit quantity estimation model.

[0086] Based on the above technical solution, the specific steps of step B1 are as follows:

[0087] Step B1.1, at the orchard site, move along the orchard work path to shoot videos of fruit trees in the orchard;

[0088] Step B1.2: for the video of the fruit trees in the garden, extract image frames at a certain frame rate interval to form a sequence image data set;

[0089] Step B1.3, labeling each image in the sequence image dataset, wherein the labeling information includes the position coordinate information of the fruit in the image and the same fruit target in adjacent images;

[0090] Step B1.4, generate annotation files for model training, each annotation file corresponds to an image, and the annotation file contains the location information and identity information of the fruit in the image, in the format of "class id, x_center / img_width, y_center / img_height, w / img_width, h / img_height";

[0091] Step B1.5: Convert the annotation file to the "COCO" format for subsequent model training.

[0092] Based on the above technical solution, the specific steps of step B2 are as follows:

[0093] Step B2.1: For the target detection network, such as the YOLOv5s network, retain its backbone network (Backbone), and improve the Neck structure of the YOLOv5s network. The improved YOLOv5s network is used as the feature extraction network of FairMOT to improve the model reasoning speed and reduce the model size while maintaining a high reasoning accuracy.

[0094] Step B2.2, improve the target tracking algorithm of FairMOT, add a special operation in the original target tracking framework, separate the high-score frame and the low-score frame after target detection, especially for the low-confidence bounding box that is difficult to handle, use the similarity between the low-score detection frame and the tracking trajectory to find possible fruit targets from the low-score frame, so as to further restore the real fruit target trajectory and improve the tracking effect.

[0095] See also Figure 3 As shown, the specific steps of improving the YOLOv5s network in step B2.1 are as follows: remove the PAN network in the Neck structure of the YOLOv5s network, add a set of "CSP2_1+CBL+upsampling" operations in the last layer of Neck, and perform a Concat operation with the output of CSP1_1 in Backbone, and directly connect its output to the detection branch (Detection branch) and identity embedding branch (Identity embedding branch) in the FairMOT model. Among them, the detection branch contains the candidate target center point heatmap (Keypoint heatmap), the target bounding box size (Object size), and the offset (Local offset) of the target center relative to the actual position of the original image. The identity embedding branch contains the identity feature (ReID) of the candidate target of the detection result, which represents the appearance feature of the fruit in the detection frame and is used to judge whether the target frame in different image frames is the same fruit.

[0096] The specific steps of fruit target tracking in step B2.2 are as follows:

[0097] Step B2.2.1, let V be the input orchard video, Det be the fruit detector, and KF be the Kalman filter. T is the set of video trajectories, each of which includes the detection box information and identity information of the fruit target. At the same time, set three thresholds T high , T low and e. Among them, T high and T low is the confidence threshold of target detection, and e is the confidence threshold of target tracking.

[0098] Step B2.2.2: For all detection boxes and confidences generated by the detector Det, the detection confidence is higher than the threshold T high The classification is D high , D high Contains above threshold T high The position of the detection frame and its identity features; the detection confidence is higher than the threshold T low The classification is D low , including those above the threshold T low The location information of the detection box and its ReID feature information.

[0099] Step B2.2.3: For the trajectories in the trajectory set T, use the Kalman filter KF to predict their coordinates in the current frame.

[0100] Step B2.2.4: D high The first association is performed with all the trajectories in the trajectory set T, and D is obtained by using the intersection over union (IoU) and identity features between the detection boxes. high The similarity between the detection box and the detection box predicted by the filter is matched using the Hungarian algorithm. The detection boxes that are not matched successfully are stored in D remain Stack, unmatched traces, stored in T remain Stack.

[0101] Step B2.2.5: Move the low confidence detection box D low With trajectory T remain The second association is performed using the same matching method as the first one. The unmatched trajectories are stored in T re-remain At the same time, the low-confidence detection boxes that have not been successfully matched are directly deleted and not stored in D remain Stack. That is, the trajectory that failed to match successfully for the second time will be stored in the stack T re-remain .

[0102] Step B2.2.6: For T re-remainThe trajectory in the target is considered to be temporarily lost, but T re-remain The trajectories in will still be retained in the trajectory set T. If T re-remain If the late matching of the middle track is successful or exists for more than a certain time (such as 30 frames), then re-remain and T, otherwise continue to save it in T.

[0103] Step B2.2.7: For D remain If the detection confidence of the detection box in is higher than e and survives for more than two frames, it is initialized as a new trajectory and stored in the trajectory set T.

[0104] Step B2.2.8: For each frame of the video, output T all detected and tracked fruit bounding boxes and corresponding IDs for the current frame.

[0105] See also Figure 4 As shown, based on the above technical solution, the specific steps of using the Simplejpeg library and the Imagezmq library in step E to implement the sending of the measurement result image data packet are:

[0106] Step E1, initialize the ImageSender object in the Imagezmq library;

[0107] Step E2: Set the connect_to variable of the ImageSender object to the IP address of the Jetson AGX Xavier edge computing terminal and the port number used for remote communication, and set the REQ_REP variable to False so that there is no need to wait for any confirmation information from the remote end each time a message is sent;

[0108] Step E3, judging whether the current measurement result image frame is empty, if not, calling the encode_jpeg function in the Simplejpeg library to encode the current image frame, setting the color space colorspace variable to BGR, and setting the compression quality quality variable to a value less than 100, to obtain the encoded measurement result image data packet;

[0109] Step E4, using the send_jpg function of the ImageSender object to start sending (publishing) the current measurement result image data packet;

[0110] Step E5, looping through steps E3 to E4, and continuously sending the image data of the production measurement results.

[0111] The remote monitoring module is deployed in the monitoring center scenario, and uses a wireless bridge to receive the yield measurement result data sent by the orchard real-time yield measurement module, and transmits it to the computer terminal through the RJ45 port. The computer terminal is deployed with a yield measurement result monitoring program module, see Figure 5 As shown, the specific steps for implementing remote monitoring are as follows:

[0112] Step S1, initialize the ImageHub object in the Imagezmq class;

[0113] Step S2: Set the open_port variable of the ImageHub object to the IP address and port number of the edge computing terminal in the orchard real-time production measurement module, and set the REQ_REP variable to False, so that there is no need to wait for any confirmation information from the remote end each time a message is sent;

[0114] Step S3, calling the recv_jpg function of the ImageHub object, and if data is detected to be incoming, receiving the measurement result image data packet sent by the specified IP address;

[0115] Step S4, calling the decode_jpeg function in the Simplejpeg library, setting the colorspace variable to RGB, decompressing the received measurement result image data packet, and obtaining the decompressed measurement result image;

[0116] Step S5, using the imshow function in the OpenCV library to visualize the measurement result graph;

[0117] Step S6, looping step S3 to step S5, at the same time, using the imwrite function in the OpenCV library to write the received image frame into the specified video file, the video file name is set to the current system time, the format is "%y%m%d%H%M%S", and the local storage of the measurement results is completed.

[0118] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0119] The present invention also proposes a remote monitoring system for real-time yield measurement in an orchard, which is characterized by comprising:

[0120] A training module is used to obtain a video of fruit trees in an orchard, extract image frames of the video at intervals to form a sequence image set, annotate each image in the sequence image set, and the annotation information includes the position coordinates of the fruit in the image and the same fruit target in adjacent images. The sequence image set and its annotation information are used as training samples and labels to train a multi-target tracking network to obtain a fruit quantity estimation model;

[0121] The acquisition module is used for the unmanned vehicle with automatic navigation to collect orchard videos in the orchard and transmit them to the edge computing terminal in the unmanned vehicle. The fruit quantity estimation model configured in the edge computing terminal is used to automatically detect and track the fruit targets in the orchard video, assign an ID to the fruit in each frame of the orchard video, and generate an image with a fruit detection frame and a specified ID.

[0122] The statistical module is used to count the total amount of fruits in the orchard according to the ID of each fruit as the execution result of the current yield measurement task.

[0123] The orchard real-time yield measurement remote monitoring system is characterized in that the process of the fruit quantity measurement model in the acquisition module tracking the fruit in the video includes:

[0124] Let V be the input orchard video, Det be the fruit detector, KF be the Kalman filter, T be the trajectory set of the orchard video, each trajectory of which includes the detection box information and identity information of the fruit target; set three thresholds T high , T low and e. Among them, T high and T low is the confidence threshold of target detection, and e is the confidence threshold of target tracking;

[0125] The fruit detector Det adds a detection box and confidence to the current frame of the orchard video. The detection confidence is higher than the threshold T high The detection box is classified as D high , D high Contains above threshold T high The position of the detection frame and its identity features; the detection confidence is higher than the threshold T low The detection box is classified as D low , D low Contains above threshold T low The location of the detection box and its identity features;

[0126] For the trajectories in the trajectory set T, the Kalman filter KF is used to predict the coordinates of the detection box in each trajectory in the current frame;

[0127] D high It is associated with all the trajectories in the trajectory set T, and D is obtained by using the intersection-over-union ratio and identity features between the detection boxes. high The similarity between the detection box and the detection box predicted by the filter is matched using the Hungarian algorithm. The unmatched detection boxes are stored in D remain Stack, unmatched traces, stored in T remain Stack;

[0128] The low confidence detection box D low With trajectory T remainMake the association and get D low The similarity between the detection box and the detection box predicted by the filter. The detection box with a similarity less than the threshold is not matched. The unmatched trajectory is stored in T re-remain , delete the unmatched D low Detection frame;

[0129] For T re-remain The trajectory in the target is considered to be temporarily lost, but T re-remain The trajectories in will still be retained in the trajectory set T. If T re-remain If the late matching of the middle track is successful or the existence time exceeds the threshold, then the T re-remain and T, otherwise continue to save it in the trajectory set T;

[0130] For D remain If the detection confidence of the detection box in is higher than e and survives for more than two frames, it is initialized as a new trajectory and stored in the trajectory set T;

[0131] For each frame of the video, output the trajectory set T of all detected and tracked fruit bounding boxes and corresponding IDs in the current frame.

[0132] The orchard real-time yield measurement remote monitoring system is characterized in that the statistical module also includes transmitting the execution result and the image containing the fruit detection frame and the corresponding ID to the remote monitoring module to visualize the yield measurement result image.

[0133] The orchard real-time yield measurement and remote monitoring system is characterized in that the construction process of the fruit detection network in the fruit quantity measurement model includes:

[0134] Remove the path aggregation network (PAN) in the Neck structure in the YOLOv5s network, add a set of "CSP2_1+CBL+upsampling" operations to the last layer of Neck, and concatenate it with the output of CSP1_1 in the YOLOv5s network backbone network (Backbone), and directly connect its output to the detection branch (Detection branch) and identity embedding branch (Identity embedding branch) in the FairMOT model;

[0135] Among them, the detection branch contains the candidate target center point heat map, the target bounding box size, and the offset of the target center relative to the actual position of the original image, and the identity embedding branch contains the identity features of the candidate targets in the detection results.

[0136] The present invention also proposes a storage medium for storing a program for executing any one of the orchard real-time yield measurement and remote monitoring methods.

[0137] The present invention also proposes a client, which is used for any one of the above-mentioned orchard real-time yield measurement and remote monitoring systems.

Claims

1. A method for remote monitoring of orchard yield in real time, characterized in that: include: Step 1, obtain a video of fruit trees in an orchard, extract image frames from the video at intervals to form a sequence image set, annotate each image in the sequence image set, and the annotation information includes the fruit position coordinates in the image and the same fruit targets in adjacent images. Use the sequence image set and its annotation information as training samples and labels to train a multi-target tracking network to obtain a fruit quantity estimation model; Step 2: The automatically navigated unmanned vehicle collects orchard videos in the orchard and transmits them to the edge computing terminal in the unmanned vehicle. The fruit quantity estimation model configured in the edge computing terminal automatically detects and tracks the fruit targets in the orchard video, assigns an ID to the fruit in each frame of the orchard video, and generates an image with a fruit detection frame and a specified ID. Step 3: According to the ID of each fruit, the total amount of fruits in the orchard is counted as the execution result of the current yield measurement task; The process of the fruit quantity estimation model tracking the fruits in the video in step 2 includes: Let V be the input orchard video, Det be the fruit detector, KF be the Kalman filter, T be the trajectory set of the orchard video, each trajectory of which includes the detection box information and identity information of the fruit target; set three thresholds T high , T low and e; where T high and T low is the confidence threshold of target detection, and e is the confidence threshold of target tracking; Step 21, the fruit detector Det adds a detection frame and confidence to the current frame of the orchard video, and the detection confidence is higher than the threshold T high The detection box is classified as D high , D high Contains above threshold T high The position of the detection frame and its identity features; the detection confidence is higher than the threshold T low The detection box is classified as D low , D low Contains above threshold T low The location of the detection box and its identity features; Step 22: For the trajectories in the trajectory set T, use the Kalman filter KF to predict the coordinates of the detection box in each trajectory in the current frame; Step 23: D high It is associated with all the trajectories in the trajectory set T, and D is obtained by using the intersection-over-union ratio and identity features between the detection boxes. high The similarity between the detection box and the detection box predicted by the filter is matched using the Hungarian algorithm. The unmatched detection boxes are stored in D remain Stack, unmatched traces, stored in T remain Stack; Step 24: The low confidence detection box D low With trajectory T remain Make the association and get D low The similarity between the detection box and the detection box predicted by the filter. The detection box with a similarity less than the threshold is not matched. The unmatched trajectory is stored in T re-remain , delete the D that did not match successfully low Detection frame; Step 25: For T re-remain The trajectory in the target is considered to be temporarily lost, but T re-remain The trajectories in will still be retained in the trajectory set T; if T re-remain If the late matching of the middle track is successful or the existence time exceeds the threshold, then the T re-remain and T, otherwise continue to save it in the trajectory set T; Step 26: For D remain If the detection confidence of the detection box in is higher than e and survives for more than two frames, it is initialized as a new trajectory and stored in the trajectory set T; Step 27: For each frame of the video, output a trajectory set T of all detected and tracked fruit bounding boxes and corresponding IDs in the current frame.

2. The method for remote monitoring of orchard yield in real time according to claim 1, characterized in that: The step 3 also includes transmitting the execution result and the image including the fruit detection frame and the corresponding ID to the remote monitoring module so as to visualize the yield measurement result image.

3. The method for real-time yield measurement and remote monitoring of an orchard as claimed in claim 1, characterized in that: The construction process of the fruit detection network in the fruit quantity estimation model includes: The path aggregation network in the Neck structure in the YOLOv5s network is removed, and the "CSP2_1+CBL+upsampling" operation is added to the last layer of Neck, and the output of CSP1_1 in the YOLOv5s network backbone network is concatenated. The output is directly connected to the detection branch and identity embedding branch in the FairMOT model. Among them, the detection branch contains the candidate target center point heat map, the target bounding box size, and the offset of the target center relative to the actual position of the original image, and the identity embedding branch contains the identity features of the candidate targets in the detection results.

4. A remote monitoring system for real-time yield measurement in an orchard, characterized in that: include: A training module is used to obtain a video of fruit trees in an orchard, extract image frames from the video at intervals to form a sequence image set, annotate each image in the sequence image set, and the annotation information includes the fruit position coordinates in the image and the same fruit targets in adjacent images. The sequence image set and its annotation information are used as training samples and labels to train a multi-target tracking network to obtain a fruit quantity estimation model; The acquisition module is used for the unmanned vehicle with automatic navigation to collect orchard videos in the orchard and transmit them to the edge computing terminal in the unmanned vehicle. The fruit quantity estimation model configured in the edge computing terminal is used to detect and track the fruit targets in the orchard video, and an ID is assigned to the fruit in each frame of the orchard video, thereby generating an image with a fruit detection frame and a specified ID. The statistical module is used to count the total amount of fruits in the orchard according to the ID of each fruit as the execution result of the current yield measurement task; The process of the fruit quantity estimation model in the acquisition module tracking the fruits in the video includes: Let V be the input orchard video, Det be the fruit detector, KF be the Kalman filter, T be the trajectory set of the orchard video, each trajectory of which includes the detection box information and identity information of the fruit target; set three thresholds T high , T low and e; where T high and T low is the confidence threshold of target detection, and e is the confidence threshold of target tracking; The fruit detector Det adds a detection box and confidence to the current frame of the orchard video. The detection confidence is higher than the threshold T high The detection box is classified as D high , D high Contains above threshold T high The position of the detection frame and its identity features; the detection confidence is higher than the threshold T low The detection box is classified as D low , D low Contains above threshold T low The location of the detection box and its identity features; For the trajectories in the trajectory set T, the Kalman filter KF is used to predict the coordinates of the detection box in each trajectory in the current frame; D high It is associated with all the trajectories in the trajectory set T, and D is obtained by using the intersection-over-union ratio and identity features between the detection boxes. high The similarity between the detection box and the detection box predicted by the filter is matched using the Hungarian algorithm. The unmatched detection boxes are stored in D remain Stack, unmatched traces, stored in T remain Stack; The low confidence detection box D low With trajectory T remain Make the association and get D low The similarity between the detection box and the detection box predicted by the filter. The detection box with a similarity less than the threshold is not matched. The unmatched trajectory is stored in T re-remain , delete the D that did not match successfully low Detection frame; For T re-remain The trajectory in the target is considered to be temporarily lost, but T re-remain The trajectories in will still be retained in the trajectory set T; if T re-remain If the late matching of the middle track is successful or the existence time exceeds the threshold, then the T re-remain and T, otherwise continue to save it in the trajectory set T; For D remain If the detection confidence of the detection box in is higher than e and survives for more than two frames, it is initialized as a new trajectory and stored in the trajectory set T; For each frame of the video, output the trajectory set T of all detected and tracked fruit bounding boxes and corresponding IDs in the current frame.

5. The orchard real-time yield measurement remote monitoring system according to claim 4, characterized in that: The statistical module also includes transmitting the execution result and the image including the fruit detection frame and the corresponding ID to the remote monitoring module to visualize the yield measurement result image.

6. The orchard real-time yield measurement remote monitoring system according to claim 4, characterized in that: The construction process of the fruit detection network in the fruit quantity estimation model includes: The path aggregation network in the Neck structure in the YOLOv5s network is removed, and a set of "CSP2_1+CBL+upsampling" operations are added to the last layer of Neck, and the output of CSP1_1 in the YOLOv5s network backbone network is spliced. The output is directly connected to the detection branch and identity embedding branch in the FairMOT model; Among them, the detection branch contains the candidate target center point heat map, the target bounding box size, and the offset of the target center relative to the actual position of the original image, and the identity embedding branch contains the identity features of the candidate targets in the detection results.

7. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the orchard real-time yield measurement and remote monitoring method as claimed in any one of claims 1 to 3.

8. A client, used in the orchard real-time yield measurement and remote monitoring system as described in any one of claims 4 to 6.