A method for automatic counting of fruits based on three-dimensional point cloud

Abstract

The application discloses a kind of fruit automatic counting methods based on three-dimensional point cloud, it is related to fruit fruit yield estimation technical field.The method first utilizes LIDAR camera to capture the point cloud sequence data of orchard scene and its corresponding camera motion trajectory information, and the three-dimensional scene is overflown to obtain fruit occupancy map FOM;Then construct the efficient object detection module based on point cloud sequence data, use the fast target suggestion and state initial estimation network guided by space-time attention, based on early integration stage FOM guide RPN and intermediate integration stage Fusion R-CNN fusion framework carry out fruit detection;Finally, based on AB3DMOT algorithm, construct the fruit tracking module based on point cloud sequence data, carry out space-time memory oriented motion prediction and fruit occupancy map update.The application is by the cumulative memory information of object state change with time back to detection module, significantly improve the precision and tracking efficiency when 3D target is detected by target detector, realizes the object continuous positioning, tracking and identification in three-dimensional space.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fruit yield estimation, and particularly relates to a fruit automatic counting method based on three-dimensional point cloud. BACKGROUND

[0002] At present, the mainstream fruit counting method mainly relies on shooting videos in orchards and processing with subsequent computer vision algorithms. The specific process is as follows: first, high-definition video data is collected in the target orchard and transmitted to the ground station or cloud platform for storage and processing; second, advanced computer vision algorithms are applied to in-depth analysis of video frames to realize accurate detection, tracking and counting of fruits. This method not only provides strong support for fruit and vegetable yield prediction, but also achieves remarkable results. However, in practical application, this method still faces many challenges. The current mainstream method is mainly based on two-dimensional video data, using various segmentation or detection network architectures (such as VCC, ResNet, YOLO, etc.) for fruit detection and counting. However, fruit tracking and counting are still challenging, mainly due to repeated calculation of fruits and calculation of fruits outside the region of interest (such as fallen fruits or fruits on the back row of trees). In particular, repeated counting may be caused by various situations, such as repeatedly observing the same fruit in consecutive images or repeatedly counting the same fruit from different angles.

[0003] With the continuous progress of information collection technology, depth information is introduced into the target recognition field, and the emergence of three-dimensional point cloud technology provides a new perspective for fruit detection. Three-dimensional information can accurately reflect the spatial position of the object, and through point cloud stitching technology, the limitations of camera field of view can be compensated, and the complete three-dimensional information of the target can be obtained. However, in the three-dimensional point cloud scene of the orchard, fruits are often blocked or truncated by other objects, affecting the accuracy of target detection and tracking. The three-dimensional object detector in the existing detection-based tracking system usually re-detects objects in each new video frame, failing to fully utilize the previous detection results. This leads to missed detection of objects in cases of severe occlusion and truncation. Although the subsequent tracking module can associate the detected frames and integrate temporal information to some extent to improve the results, the detection module still works independently frame by frame, failing to fully utilize the memory information of objects in the scene. This working method not only is inefficient, but also contradicts the way humans construct spatial memory. Therefore, constructing a system that can continuously detect and track objects in a three-dimensional scene under occlusion and truncation, like a human, is one of the key scientific problems that the present application needs to solve. SUMMARY

[0004] The application aims to provide a fruit automatic counting method based on three-dimensional point cloud, which realizes the continuous positioning, tracking and identification of objects in three-dimensional space by optimizing algorithm design, fully utilizing three-dimensional point cloud information and combining object memory information, and solves the problems of repeated calculation of fruits and calculation of fruits outside the region of interest.

[0005] To achieve the above-mentioned purpose, the application provides a fruit automatic counting method based on three-dimensional point cloud, and the specific steps are as follows:

[0006] S1, using a LIDAR camera to capture point cloud sequence data of an orchard scene and corresponding camera motion trajectory information, and obtaining a fruit occupancy map FOM by overlooking the three-dimensional scene;

[0007] S2, constructing an efficient object detection module based on point cloud sequence data, using a fast target proposal and state initial estimation network guided by space-time attention, screening out high-probability regions containing fruits and unknown regions by using the fruit occupancy map FOM, and dividing the detection module into an early integration stage and an intermediate integration stage, guiding RPN based on the fruit occupancy map FOM of the early integration stage and performing fruit detection by using a Fusion R-CNN fusion framework of the intermediate integration stage;

[0008] S3, constructing a fruit tracking module based on point cloud sequence data based on a three-dimensional multi-target tracking algorithm AB3DMOT, and performing space-time memory guided motion prediction and fruit occupancy map FOM updating.

[0009] Preferably, in S2, the early integration stage guides RPN by using the fruit occupancy map FOM, and the specific steps are as follows:

[0010] S211, according to the fruit occupancy map FOM obtained in the last frame, the fruit occupancy map FOM is divided into a high occupancy area, a low occupancy area and an unobserved area by selecting a threshold value sigma, and the area without fruit is discarded according to the fruit occupancy map FOM after time fusion;

[0011] S212, after obtaining the fruit occupancy segmentation point cloud by using the fruit occupancy map, a 3D target detection algorithm PointRCNN is used as a basic detection architecture, a bin-based 3D box generation module is used as RPN, the search space is discretized into a series of unit cells bins, and 3D detection is performed on the dense fruit area.

[0012] Preferably, in S2, the intermediate integration stage is based on a Fusion R-CNN fusion framework, and the point cloud of the region of interest RoI is formed by fusing the target proposal box b prop of the current frame and the future state prediction box b pred of the last frame, and inputting the point cloud to a detection frame regression network R-CNN for detection frame prediction.

[0013] Preferably, in S3, the space-time memory oriented motion prediction selects a three-dimensional multi-target tracking algorithm AB3DMOT, inputs the detection frame prediction result in S2 into a Kalman filter, uses 3D IoU association and a Hungarian algorithm to quickly establish the correspondence relationship between the current frame and the historical trajectory, and simultaneously follows the setting of the Kalman filter in AB3DMOT, at each time t, the preferred motion prediction adopts a constant speed model independent of the camera self-motion to describe the interframe displacement between fruits, and generates a fused target state as the final future frame state prediction.

[0014] Preferably, in S3, the fruit occupancy map FOM updates the space-time memory information provided by the FOM constructed based on the previous frame, and updates the fruit occupancy map in real time by integrating the point cloud sequence data received in the current frame and the future frame fruit state prediction map generated by the motion prediction module.

[0015] Therefore, the present application proposes a fruit automatic counting method based on three-dimensional point clouds, which has the following beneficial effects:

[0016] (1) The fruit automatic counting method based on three-dimensional point clouds of the present application innovatively uses the concept of space-time memory to solve the problems of occlusion and continuous tracking in real scenes, effectively reduces the waste of computing resources in invalid areas, and significantly improves the detection speed and accuracy; at the same time, the continuous observation data is comprehensively utilized, the prediction of the future state of the object is dynamically optimized, and it is integrated into the whole detection process, thereby enhancing the robustness and accuracy of the detection system.

[0017] (2) The fruit automatic counting method based on three-dimensional point clouds of the present application uses 3D IoU association and a Hungarian algorithm to quickly establish the correspondence relationship between the current frame and the historical trajectory, and simultaneously follows the setting of the Kalman filter in AB3DMOT, thereby simplifying the operation while maintaining high accuracy, and realizing accurate prediction of the motion state of the fruit at a lower computational cost.

[0018] The technical solutions of the present application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is the overall system flowchart of the fruit automatic counting method based on three-dimensional point clouds of the present application;

[0020] Figure 2 is a bin-based positioning schematic diagram in the PointRCNN framework of the fruit automatic counting method based on three-dimensional point clouds of the present application;

[0021] Figure 3is a visual description of a fruit automatic counting method based on three-dimensional point cloud Fusion R-CNN. DETAILED DESCRIPTION

[0022] In order to make the technical solutions, advantages and purposes of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below. The described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0023] Unless otherwise defined, the technical terms or scientific terms used in the present application shall have the usual meanings understood by those of ordinary skill in the art to which the present application belongs.

[0024] As shown in Figure 1 The present application provides a fruit automatic counting method based on three-dimensional point cloud. In accordance with the conventional practice in the field of SLAM, the module responsible for predicting the current frame is regarded as the front end, and the module responsible for integrating the time series observation data and predicting the future fruit state is regarded as the back end. The specific steps are as follows:

[0025] 1. Capture the point cloud sequence data of the orchard scene and its corresponding camera motion trajectory information using the LIDAR camera, and obtain the fruit occupancy map FOM by looking down on the three-dimensional scene.

[0026] Fruit Occupancy Map (FOM) is a 2D grid map obtained by looking down on a 3D scene, each grid cell corresponds to an area in the real world defined by the grid size, and stores the probability of the area being occupied by the fruit, i.e. the probability of the area existing fruit. The entire 3D space can be divided into high occupancy probability area (high probability of existing fruit), low occupancy probability area (low probability of existing fruit) and unobserved area (area with insufficient observation to determine whether there is fruit). When generating fruit bounding boxes, only high occupancy probability area and unobserved area will be processed by the network, reducing the number of point clouds the network needs to process, improving efficiency, and preventing the fruit bounding box network from generating false fruit bounding boxes in areas with low probability of existing fruit. Find the corresponding FOM grid point from the point cloud, and complete it by projecting the point cloud along the Z axis and then discretizing it. Conversely, you can find the point cloud of the corresponding fruit in space from the FOM grid point. In the FOM update process, after completing the detection of each frame and obtaining the future state estimation of the fruit in the next frame through motion prediction, the 3D detection box of each fruit in the next frame will be mapped to the three-dimensional scene, and the point cloud will be divided into fruit point cloud and background point cloud. For the area where the fruit point cloud is located, the probability of the fruit appearing in the next frame is high, so the fruit occupancy probability of the corresponding FOM grid point will be increased, otherwise it is considered that the probability of the fruit appearing in the next frame is low, and the fruit occupancy probability of the corresponding FOM grid point will be reduced.

[0027] 2. Construct an efficient object detection module based on point cloud sequence data, guide the RPN through the early integration stage fruit occupancy map FOM, and perform fruit detection through the intermediate integration stage Fusion R-CNN.

[0028] 2.1. In the early integration stage of the front end, guide the RPN through the FOM, perform 3D detection of the fruit dense area, which specifically includes:

[0029] 2.1.1. After inputting the point cloud data, in order to ensure that detection is only performed in previously unobserved areas and continuous detection of observed fruits, integration must be performed in the initial stage of the detection process. According to the fruit occupancy map obtained in the last frame, by selecting a threshold σ, the input point cloud can be divided into three parts according to the occupancy score corresponding to the grid point position (x, y) in the world coordinate system:

[0030]

[0031] Where M t is the fruit occupancy; M t greater than the threshold σ is the high occupancy area; M t less than the threshold -σ is the low occupancy area; M t greater than or equal to the threshold -σ and less than or equal to the threshold σ is the unobserved area.

[0032] By discarding the regions without target through the fused FOM according to time, there is no need to detect in the region where it has been determined that there is no fruit, and the computing resources are saved at the backbone network level. At the same time, the number of anchor points and the number of generated proposal boxes are greatly reduced, which means that the computing time can be saved at the RPN and 3D non-maximum suppression (NMS) stages, and no input frame is skipped. Compared with some video fruit detection methods that skip input frames in order to speed up detection, not skipping input frames improves the reliability of the system.

[0033] 2.1.2, after obtaining the fruit occupancy segmentation point cloud through the fruit occupancy map, 3D detection is performed on the fruit dense region. Due to the inherent characteristics of the fruit point cloud data: sparse point distribution and irregular shape caused by occlusion, it is extremely challenging to directly and accurately predict the three-dimensional detection box parameters (center position, size and orientation) of the fruit; this application selects PointRCNN as the basic detection architecture, and regards the "bin-based 3D Box Generation" module in it as RPN. By discretizing the search space into a series of bins, the model performs classification and regression within a limited, discretized range, which simplifies the calculation process while reducing unnecessary background noise. In the LiDAR coordinate system, the three-dimensional bounding box of the fruit is represented as (x; y; z; h; w; l; θ), where (x; y; z) is the target center position, (h; w; l) is the size of the fruit, and θ is the direction angle of the fruit observed from the bird's eye view. In order to constrain the generated 3D detection box, a bin-based regression loss function is used to estimate the 3D detection box of the fruit.

[0034] As shown in Figure 2 To estimate the fruit center position, the area around each fruit point is divided into a series of discrete cells bins along the x and z axes. Specifically, a search range S is set for each fruit point on the X and Z axes, and each one-dimensional search range is equally divided into cells with a length of δ, so as to represent different fruit center positions (x; z) on the X-Z plane. The bin-based classification combined with the cross-entropy loss function is used to process the positioning of the X and Z axes respectively, which is more effective than directly using the smooth L1 loss function for regression, and can better estimate the fruit center on the X and Z axes. The positioning loss of the X and Z axes consists of two parts, one for bin classification along each X and Z axis, and the other for residual regression within the classified bin. For the center position y along the vertical y axis, the smooth L1 loss is directly used for regression because the y value of most fruits is within a very small range. Using the L1 loss is sufficient to obtain accurate y values. Therefore, the local target function can be expressed as:

[0035]

[0036]

[0037] where (x (p) , y (p) , z (p) ) is the coordinate of the fruit point of interest, (x p , y p , z p ) is the coordinate of the center of its corresponding fruit, and are the real-valued bin assignments along the x and z axes, and are the real-valued residuals for further localization refinement within the bin assignments, C is the bin length for normalization; the direction 2π is divided into n bins, and the bin classification targets are computed in the same way as the x or z predictions and the residual regression targets The fruit size (h; w; l) is directly regressed by computing the residuals of the average fruit size of each class in the whole training set.

[0038] In the inference stage, for the bin-based predicted parameters x, z, θ, the bin center with the highest prediction confidence is selected and added with the predicted residual to get the refined parameters; for other directly regressed parameters, including y, h, w and l, the predicted residuals are added to their initial values; for different training loss terms, the overall three-dimensional bounding box regression loss L reg can be expressed as:

[0039]

[0040] where N pos is the number of fruit points, and are the predicted bin assignments and residuals for fruit point p, and are the ground-truth targets, F cls is the cross-entropy classification loss, F res is the smooth L1 loss.

[0041] To eliminate redundant target proposal boxes, directional IoU is used for non-maximum suppression from the bird's eye view to generate a small number of high-quality target proposal boxes. Directional NMS with an IoU threshold of 0.8 is used, and only the top 100 target proposal boxes are retained for the intermediate stage subnetwork.

[0042] ​2.2. In the middle stage of the front-end integration stage, the current frame target proposal box is fused with the last frame target future state prediction box using Fusion R-CNN, and is input into the detection box regression network, so that the front-end detection box prediction is more accurate and stable.

[0043] In the task of 3D target detection, the target fruit may not be able to generate a fruit candidate box in the current frame due to occlusion, and thus the optimization of the fruit candidate box and the generation of the detection result cannot be performed, resulting in target missing detection. In the fruit candidate box optimization stage, the fruit distribution prior accumulated in time is further fused, fruit 3D prediction boxes are sampled from the fruit future state estimation, and the 3D prediction boxes are added to the fruit candidate box to fuse the time sequence information, so that the target fruit cannot generate a candidate box due to occlusion or truncation. After the fruit candidate box optimization network outputs the final fruit candidate box, the prediction boxes are also added to the fruit candidate box for 3D box non-maximum suppression, further ensuring that the occluded fruit will not be ignored by the network.

[0044] After generating multiple target proposal boxes in the early stage of the current frame, the current frame target proposal box b prop and the last frame target future state prediction box b pred The fusion process is similar in concept to the correction step in Kalman filtering, which first obtains b prop and b pred by 3D RoIPooling module cropping, and then combines to form a RoI point cloud which is input into the regression network. b pred participate in the three-dimensional NMS process at the same time, and combine the fused fruit score, wherein the fusion is realized by the moving average of the target fruit confidence score obtained by continuous observation, and the calculation formula is as follows:

[0045]

[0046] Among them, represents the score of the back-end detection box from the 1st frame, is the score of the 3D detection box detected from the tth frame, and a e [0, 1] is a decay parameter.

[0047] As shown in Figure 3 , the Fusion R-CNN generates a current frame observation value, denoted as b front , by combining the current frame target fruit state with the back-end target future state obtained by time fusion and prediction, and inputting into the detection box regression network (R-CNN), so that the front-end detection box prediction is more accurate and stable. The RoI point cloud fusion step before R-CNN ensures that even if the RPN cannot generate an accurate target proposal box, the point cloud information of the fruit of interest can be transmitted through the network; by adding b frontThe fruit state temporarily optimized from the previous frames is integrated into the non-maximum suppression process, which helps to enhance the robustness of the entire system. Even if the R-CNN cannot predict a detection box with high enough confidence due to severe truncation or occlusion, the NMS module can rely on score fusion to select the highest-scoring prediction from b front the previous frames.

[0048] 3. Based on the AB3DMOT algorithm, a fruit tracking module based on point cloud sequence data is constructed to perform spatio-temporal memory oriented motion prediction and fruit occupancy map (FOM) update.

[0049] 3.1. To achieve Kalman filtering and motion prediction, the current front-end prediction is input into the Kalman filter. The three-dimensional multi-target tracking algorithm AB3DMOT is selected, and the Kalman filter settings in AB3DMOT are followed. At each time t, the fruit motion at t+1 is predicted by the Kalman filter. The constant velocity model independent of camera self-motion is used to approximate the inter-frame displacement between fruits, and the fused target state is generated as the final future frame state prediction.

[0050] 3.2. The update of the FOM is based on the spatio-temporal memory information provided by the FOM constructed in the previous frame. By integrating the point cloud data received in the current frame and the future frame fruit state prediction map generated by the motion prediction module, the fruit occupancy map is updated in real time. Conceptually, the update process of the FOM is similar to the projection mapping operation for two-dimensional bird's eye view in dense three-dimensional reconstruction systems such as KinectFusion. The FOM construction focuses on integrating the occupancy state observation information of the target fruit in each frame, and can dynamically update the spatio-temporal occupancy distribution of each fruit in the scene. Specifically, the progressive construction of the fruit occupancy map M t can be achieved by the following mathematical expression:

[0051]

[0052] If the point p i is in any bounding box of the future state of the fruit, then Φ(p i ) = +γ, otherwise -1, is the fused target classification score rescaled by a factor λ. Let denote the transformation from camera frame to world frame, and Voxelize(·) represents the operation of converting the world frame point cloud to a two-dimensional grid. M0 is initialized as a 2D grid with all zeros.

[0053] Therefore, the application provides a fruit automatic counting method based on a three-dimensional point cloud, cumulative memory information of object state changing over time is fed back to a detection module, accuracy and tracking efficiency of a target detector detecting a 3D target are significantly improved, object continuous positioning, tracking and identification in a three-dimensional space are realized, and problems of repeated calculation of fruits and calculation of fruits outside a region of interest are solved.

[0054] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can still be modified or replaced by equivalents, and these modifications or replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A fruit automatic counting method based on three-dimensional point cloud, characterized in that: The specific steps are as follows: S1. Use the LIDAR camera to capture the point cloud sequence data of the orchard scene and its corresponding camera motion trajectory information, and obtain the fruit occupancy map (FOM) by looking down at the 3D scene; S2. Build an efficient object detection module based on point cloud sequence data. This module uses a fast object proposal and state initial estimation network guided by spatiotemporal attention, and uses the fruit occupancy map (FOM) to screen out high-probability areas containing fruit and unknown areas. The detection module is divided into an early integration stage and an intermediate integration stage. Fruit detection is performed based on the fruit occupancy map (FOM) in the early integration stage, guided by the RPN, and the Fusion R-CNN fusion framework in the intermediate integration stage. S3. Based on the 3D multi-target tracking algorithm AB3DMOT, a fruit tracking module based on point cloud sequence data is constructed to perform spatiotemporal memory-guided motion prediction and fruit occupancy map (FOM) updates. In S2, the early integration stage guides the RPN through the fruit occupancy map FOM, and the specific steps are as follows: S211, according to the fruit occupancy map FOM obtained in the previous frame, by selecting a threshold , which are divided into high-occupancy areas, low-occupancy areas and unobserved areas, and the areas without fruits are discarded according to the time-fused fruit occupancy map FOM; S212. After obtaining the fruit occupancy segmentation point cloud from the fruit occupancy map, the 3D object detection algorithm PointRCNN is used as the basic detection architecture, and the bin-based 3D Box Generation module is used as the RPN. The search space is discretized into a series of cell bins to perform 3D detection in the fruit-dense area. The intermediate integration stage is based on the Fusion R-CNN fusion framework, which proposes the target frame of the current frame. And the target future state prediction box of the previous frame The points are fused and combined to form the point cloud of the region of interest (RoI) and input into the detection box regression network R-CNN for detection box prediction. In S3, the spatiotemporal memory-guided motion prediction uses the 3D multi-target tracking algorithm AB3DMOT. The detection box prediction results in S2 are input into the Kalman filter. The 3D IoU association and Hungarian algorithm are used to quickly establish the correspondence between the current frame and the historical trajectory. At the same time, the settings of the Kalman filter in AB3DMOT are followed. , predicted by the Kalman filter Fruit Movement; The fruit occupancy map FOM is updated based on the spatiotemporal memory information provided by the FOM constructed in the previous frame. The fruit occupancy map is updated in real time by integrating the point cloud sequence data received in the current frame and the fruit state prediction map for the future frame just generated by the motion prediction module.

2. The automatic fruit counting method based on three-dimensional point cloud according to claim 1, characterized in that: Motion prediction uses a constant velocity model independent of camera self-motion to describe the inter-frame displacement of fruits, and produces a fused target state as the final future frame state prediction.

Images