Automatic fruit counting method based on three-dimensional point cloud

Abstract

The invention discloses an automatic fruit counting method based on a three-dimensional point cloud, and relates to the technical field of fruit yield estimation. The method comprises the following steps: firstly, capturing point cloud sequence data of an orchard scene and corresponding camera movement track information by using an LIDAR camera, and performing aerial view on the three-dimensional scene to obtain a fruit occupation map FOM; then, constructing an efficient object detection module based on point cloud sequence data, and performing fruit detection by using a space-time attention guided fast target suggestion and state initial estimation network, an FOM guided RPN based on an early integration stage and a Fusion R-CNN fusion framework in a middle integration stage; and finally, based on an AB3DMOT algorithm, constructing a fruit tracking module based on point cloud sequence data, and performing space-time memory guided motion prediction and fruit occupation map updating. According to the invention, the accumulated memory information of the object state changing with time is recharged to the detection module, so that the precision and tracking efficiency of the target detector in detecting the 3D target are remarkably improved, and continuous positioning, tracking and identification of the object in the three-dimensional space are realized.

Description

Technical Field

[0001] The invention relates to the technical field of fruit yield estimation, in particular to an automatic fruit counting method based on three-dimensional point cloud. Background Art

[0002] At present, the mainstream fruit counting method mainly relies on shooting videos in the orchard and processing them with the help of subsequent computer vision algorithms. The specific process is summarized as follows: First, high-definition video data is collected in the target orchard and transmitted to the ground station or cloud platform for storage and processing; second, advanced computer vision algorithms are used to deeply analyze the video frames to achieve accurate detection, tracking and counting of fruits. This method not only provides strong support for fruit and vegetable yield prediction, but also has achieved remarkable results. However, in practical applications, this method still faces many challenges. The current mainstream methods are mainly based on two-dimensional video data, using a variety of segmentation or detection network architectures (such as VCC, ResNet, YOLO, etc.) to detect and count fruits. However, fruit tracking and counting are still quite challenging, mainly due to repeated counting of fruits and counting fruits outside the area of ​​interest (such as fallen fruits or fruits on the back row of trees). In particular, repeated counting may come from a variety of situations, such as repeatedly observing the same fruit in consecutive images or repeatedly counting the same fruit from different angles.

[0003] With the continuous advancement of information acquisition technology, depth information has been introduced into the field of target recognition, and the emergence of three-dimensional point cloud technology has provided a new perspective for fruit detection. Three-dimensional information can accurately reflect the spatial position of an object. Through point cloud stitching technology, the limitations of the camera's field of view can be compensated to obtain complete three-dimensional information of the target. However, in the three-dimensional point cloud scene of the orchard, the fruit is often occluded or truncated by other objects, which affects the accuracy of target detection and tracking. The three-dimensional object detector in the existing detection-based tracking system usually re-detects the object in each new video frame, failing to make full use of the previous detection results. This leads to objects being easily missed in the case of severe occlusion and truncation. Although the subsequent tracking module can associate the detected frames and integrate temporal information to a certain extent to improve the results, the detection module still works independently frame by frame, failing to make full use of the memory information of the objects in the scene. This way of working is not only inefficient, but also contrary to the way people build spatial memory. Therefore, building a system that can continuously detect and track objects in a three-dimensional scene under occlusion and truncation like a human is one of the key scientific problems that this patent urgently needs to solve. Summary of the invention

[0004] The purpose of the present invention is to propose an automatic fruit counting method based on three-dimensional point cloud, which realizes the continuous positioning, tracking and identification of objects in three-dimensional space by optimizing algorithm design, making full use of three-dimensional point cloud information, and combining the memory information of objects, thereby solving the problems of repeated counting of fruits and counting fruits outside the area of ​​interest.

[0005] To achieve the above object, the present invention proposes a fruit automatic counting method based on three-dimensional point cloud, and the specific steps are as follows:

[0006] S1. Use LIDAR camera to capture the point cloud sequence data of the orchard scene and its corresponding camera motion trajectory information, and obtain the fruit occupancy map FOM by overlooking the three-dimensional scene;

[0007] S2. Construct an efficient object detection module based on point cloud sequence data. The detection module uses a fast target proposal and state initial estimation network guided by spatiotemporal attention, and uses the fruit occupancy map FOM to screen out high-probability areas containing fruits and unknown areas. The detection module is divided into an early integration stage and an intermediate integration stage. Fruit detection is performed based on the fruit occupancy map FOM-guided RPN in the early integration stage and the Fusion R-CNN fusion framework in the intermediate integration stage.

[0008] S3. Based on the three-dimensional multi-target tracking algorithm AB3DMOT, a fruit tracking module based on point cloud sequence data is constructed to perform spatiotemporal memory-guided motion prediction and fruit occupancy map FOM update.

[0009] Preferably, in S2, the early integration stage guides the RPN through the fruit occupancy map FOM, and the specific steps are as follows:

[0010] S211, according to the fruit occupation map FOM obtained in the previous frame, by selecting a threshold σ, it is divided into a high occupation area, a low occupation area and an unobserved area, and the area without fruit is discarded according to the time-fused fruit occupation map FOM;

[0011] S212. After obtaining the fruit occupancy segmentation point cloud through the fruit occupancy map, the 3D object detection algorithm PointRCNN is used as the basic detection architecture, and the bin-based 3D Box Generation module is used as RPN. The search space is discretized into a series of cell bins to perform 3D detection in the fruit-dense area.

[0012] Preferably, in S2, the intermediate integration stage is based on the Fusion R-CNN fusion framework to propose the current frame target frame b prop And the target future state prediction box b of the previous frame pred The points are fused and combined to form the point cloud of the region of interest (RoI) and input into the detection box regression network R-CNN for detection box prediction.

[0013] Preferably, in S3, the spatiotemporal memory-guided motion prediction uses the three-dimensional multi-target tracking algorithm AB3DMOT, inputs the detection box prediction result in S2 into the Kalman filter, and uses the 3D IoU association and Hungarian algorithm to quickly establish the correspondence between the current frame and the historical trajectory. At the same time, following the setting of the Kalman filter in AB3DMOT, at each time t, the Kalman filter predicts t+1. Preferably, the motion prediction uses a constant speed model independent of the camera's self-motion to describe the inter-frame displacement between fruits, and generates a fused target state as the final future frame state prediction.

[0014] Preferably, in S3, the fruit occupancy map FOM is updated based on the spatiotemporal memory information provided by the FOM constructed in the previous frame, and the fruit occupancy map is updated in real time by integrating the point cloud sequence data received in the current frame and the fruit state prediction map for the future frame just generated by the motion prediction module.

[0015] Therefore, the present invention proposes a fruit automatic counting method based on three-dimensional point cloud, and its beneficial effects are as follows:

[0016] (1) The present invention provides an automatic fruit counting method based on three-dimensional point cloud, which innovatively uses the concept of spatiotemporal memory to deal with the problems of occlusion and continuous tracking in real scenes, effectively reduces the waste of computing resources in invalid areas, and significantly improves the detection speed and accuracy; at the same time, it comprehensively utilizes continuous observation data, dynamically optimizes the prediction of the future state of the object, and integrates it into the entire detection process, thereby enhancing the robustness and accuracy of the detection system.

[0017] (2) The present invention provides an automatic fruit counting method based on three-dimensional point cloud, which uses 3D IoU association and Hungarian algorithm to quickly establish the correspondence between the current frame and the historical trajectory, and follows the setting of the Kalman filter in AB3DMOT, simplifies the calculation while maintaining high accuracy, and realizes accurate prediction of the fruit movement state at a low computational cost.

[0018] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an overall system flow chart of an automatic fruit counting method based on three-dimensional point cloud of the present invention;

[0020] Figure 2 It is a schematic diagram of bin-based positioning in a PointRCNN framework of a fruit automatic counting method based on three-dimensional point cloud of the present invention;

[0021] Figure 3It is a visualization illustration of Fusion R-CNN, an automatic fruit counting method based on three-dimensional point cloud of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention are clearly and completely described below. The described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.

[0023] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0024] like Figure 1 As shown, the present invention provides a fruit automatic counting method based on three-dimensional point cloud, following the conventional practice in the field of SLAM, the module responsible for current frame prediction is regarded as the front end, and the module responsible for integrating time series observation data and predicting future fruit states is regarded as the back end, and the specific steps are as follows:

[0025] 1. Use the LIDAR camera to capture the point cloud sequence data of the orchard scene and its corresponding camera motion trajectory information, and get the fruit occupancy map FOM by overlooking the three-dimensional scene.

[0026] The Fruit Occupancy Map (FOM) is a 2D grid map obtained by looking down on the 3D scene. Each grid unit corresponds to an area in the real world framework defined by the grid size, and stores the probability of fruit occupancy to be detected in the area, that is, the probability of fruit in the area. The entire 3D space can be divided into high occupancy probability areas (high probability of fruit), low occupancy probability areas (low probability of fruit), and unobserved areas (areas where insufficient observations make it impossible to determine whether there is fruit). When generating fruit candidate boxes, only high occupancy probability areas and unobserved areas will be processed by the network, reducing the number of point clouds that the network needs to process, improving efficiency, and preventing the fruit candidate box network from generating incorrect fruit candidate boxes at locations with low probability of fruit. Find the corresponding FOM grid points from the point cloud, and complete it by projecting the point cloud along the Z axis and then discretizing it. Conversely, the point cloud corresponding to the fruit in space can be found from the FOM grid points. In the FOM update process, after completing the detection of each frame and obtaining the future state estimate of the fruit in the next frame through motion prediction, the 3D detection frame of each fruit in the next frame will be mapped to the 3D scene, and the point cloud will be divided into fruit point cloud and background point cloud. For the area where the fruit point cloud is located, the probability of fruit appearing in the next frame is high, so the probability of fruit occupancy at the corresponding FOM grid point will be increased. Otherwise, it is considered that the probability of fruit appearing in the next frame in this area is low, and the probability of fruit occupancy at the corresponding FOM grid point will be reduced.

[0027] 2. Build an efficient object detection module based on point cloud sequence data, and perform fruit detection through the fruit occupancy map FOM in the early integration stage to guide RPN and the Fusion R-CNN in the middle integration stage.

[0028] 2.1. In the early integration stage of the front end, the RPN is guided by FOM to perform 3D detection of fruit-dense areas, including:

[0029] 2.1.1. After inputting the point cloud data, in order to ensure that only previously unobserved areas are detected and that observed fruits are continuously detected, integration operations must be performed at the very beginning of the detection process. Based on the fruit occupancy map obtained in the previous frame, by selecting a threshold σ, the input point cloud can be divided into three parts according to the occupancy score corresponding to the grid point position (x, y) in the world coordinate system:

[0030]

[0031] Among them, M t Occupied by fruits; M t Greater than the threshold σ is a high occupancy area; M t Less than the threshold -σ is a low occupancy area; M t The area that is greater than or equal to the threshold -σ and less than or equal to the threshold σ is the unobserved area.

[0032] By discarding areas without objects based on the temporally fused FOM, there is no need to detect in areas where it is determined that there is no fruit, saving computing resources at the backbone network level. At the same time, the number of anchor points and the number of proposal boxes generated are also greatly reduced, which means that computing time can be saved in the RPN and 3D non-maximum suppression (NMS) stages, and no input frames will be skipped. Compared with some video fruit detection methods that skip input frames to speed up detection, not skipping input frames will improve the reliability of the system.

[0033] 2.1.2. After obtaining the fruit occupancy segmentation point cloud through the fruit occupancy map, 3D detection of the fruit-dense area is performed. Due to the inherent characteristics of the fruit point cloud data: the point distribution is sparse and the shape is irregular due to occlusion, it is extremely challenging to directly and accurately predict the 3D detection box parameters (center position, size and orientation) of the fruit. This application selects PointRCNN as the basic detection architecture, and regards the "bin-based 3D Box Generation" module as RPN. By discretizing the search space into a series of bins, the model is classified and regressed within a limited, discrete range, simplifying the calculation process while reducing the influence of unnecessary background noise. In the LiDAR coordinate system, the 3D bounding box of the fruit is represented as (x; y; z; nine; w; l; θ), where (x; y; z) is the target center position, (nine; w; l) is the fruit size, and θ is the fruit direction angle observed from a bird's-eye view; in order to constrain the generated 3D detection box, a bin-based regression loss function is proposed to be used to estimate the 3D detection box of the fruit.

[0034] like Figure 2 As shown in the figure, in order to estimate the center position of the fruit, the area around each fruit point is divided into a series of discrete cell bins along the x-axis and z-axis. Specifically, a search range S is set for the current fruit point on the x-axis and z-axis respectively, and each one-dimensional search range is equally divided into cells of length δ to represent the center positions (x; z) of different fruits on the XZ plane. Using cell-based classification combined with the cross entropy loss function to process the positioning of the x-axis and z-axis respectively is more effective than directly using the smooth L1 loss function for regression, and can better classify and estimate the fruit center on the x-axis and z-axis. The positioning loss of the x-axis and z-axis consists of two terms, one for bin classification along each x-axis and z-axis, and the other for residual regression within the classification bin. For the center position y along the vertical y-axis, the smooth L1 loss is directly used for regression, because the y values ​​of most fruits are in a small range. Using L1 loss is sufficient to obtain accurate y values. Therefore, the localization objective function can be expressed as:

[0035]

[0036]

[0037] Among them, (x (p) ,y (p) , z (p) ) is the coordinate of the fruit point of interest, (x p ,y p , z p ) is the center coordinate of the corresponding fruit, and are the true value bin assignments along the x and z axes, and is the true value residual further localized and refined within the assigned bin, C is the bin length used for normalization; the direction 2π is divided into n bins, and the bin classification target is calculated in the same way as the x or z prediction. and the residual regression target Fruit size (h; w; l) is calculated by calculating the residual of the mean fruit size of each class in the entire training set Direct return.

[0038] In the inference phase, for the bin-based prediction parameters x, z, θ, the bin center with the highest prediction confidence is selected and the prediction residual is added to obtain the refined parameters; for other direct regression parameters, including y, h, w and l, the prediction residual is added to their initial values; for different training loss terms, the overall 3D detection box regression loss L reg It can be expressed as:

[0039]

[0040] Among them, N pos Count the fruits. and Assign the predicted bin value and residual to the fruit point p, and is the ground-truth target, F cls is the cross entropy classification loss, F res is the smoothed L1 loss.

[0041] In order to eliminate redundant object proposals, non-maximum suppression of directional IoU is used from a bird's-eye view to generate a small number of high-quality object proposals. Directed NMS with an IoU threshold of 0.8 is used, and only the top 100 object proposals are retained for refining the intermediate stage subnet.

[0042] 2.2. In the mid-term integration stage of the front end, FusionR-CNN is used to fuse the target proposal box of the current frame with the target future state prediction box of the previous frame, and input it into the detection box regression network to make the front-end detection box prediction more accurate and stable.

[0043] In the task of 3D target detection, the target fruit may not generate a fruit candidate frame in the current frame due to occlusion, and then the optimization of the fruit candidate frame and the generation of detection results cannot be performed, resulting in missed detection of the target. The present application further integrates the time-series accumulated fruit distribution prior in the fruit candidate frame optimization stage, samples the fruit 3D prediction frame from the future state estimation of the fruit, and fuses the time series information by adding the 3D prediction frame to the fruit candidate frame to prevent the target fruit from being blocked or truncated and unable to generate a candidate frame. After the fruit candidate frame optimization network outputs the final fruit candidate frame, these prediction frames will also be added to the fruit candidate frame to perform non-maximum suppression of the 3D frame, further ensuring that the occluded fruit will not be ignored by the network.

[0044] After generating multiple target proposal boxes in the early stage of the current frame, the current frame target proposal box b prop And the target future state prediction box b of the previous frame pred The fusion process is conceptually similar to the correction step in Kalman filtering. First, it is clipped by the 3D RoIPooling module to obtain b prop and b pred , and then combined to form a RoI point cloud and sent to the regression network. pred At the same time, it participates in the three-dimensional NMS process and combines the fused fruit scores. The fusion is achieved by the moving average of the confidence scores of the target fruits obtained through continuous observations. The calculation formula is as follows:

[0045]

[0046] in, Indicates the score of the backend detection box starting from the first frame. is the score of the 3D detection box detected starting from the tth frame, and α∈[0, 1] is the attenuation parameter.

[0047] like Figure 3 As shown, Fusion R-CNN generates the current frame observation value at the front end, denoted as b front By combining the current frame target fruit state with the back-end target future state obtained through time fusion and prediction, and inputting it into the detection box regression network (R-CNN), the front-end detection box prediction is made more accurate and stable. The RoI point cloud fusion step before R-CNN ensures that even if RPN cannot generate an accurate target proposal box, the point cloud information of the relevant fruit of interest can be transmitted through the network; by frontIntegrating into the non-maximum suppression process, the fruit state temporarily optimized from the previous frames helps to enhance the robustness of the entire system. Even if R-CNN cannot predict a sufficiently high confidence detection box due to severe truncation or occlusion, the NMS module can rely on score fusion to obtain the fruit state from b front Select the prediction with the highest score.

[0048] 3. Based on the AB3DMOT algorithm, a fruit tracking module based on point cloud sequence data is constructed to perform spatiotemporal memory-guided motion prediction and fruit occupancy map FOM update.

[0049] 3.1. To realize Kalman filtering and motion prediction, the current front-end prediction is input into the Kalman filter, and the three-dimensional multi-target tracking algorithm AB3DMOT is selected. Following the setting of the Kalman filter in AB3DMOT, at each time t, the Kalman filter predicts the movement of the fruit at t+1. The motion prediction intends to use a constant speed model independent of the camera's self-motion to approximately describe the inter-frame displacement between fruits, and generate a fused target state as the final future frame state prediction.

[0050] 3.2. The update of FOM is based on the spatiotemporal memory information provided by the FOM constructed in the previous frame. By integrating the point cloud data received in the current frame and the fruit state prediction map for the future frame just generated by the motion prediction module, the fruit occupancy map is updated in real time. At the conceptual level, the update process of FOM is similar to the projection mapping operation performed on the two-dimensional bird's-eye view in dense 3D reconstruction systems such as KinectFusion. FOM construction focuses on integrating the occupancy state observation information of the target fruit in each frame, and can dynamically update the spatiotemporal occupancy distribution of each fruit in the scene. Specifically, the fruit occupancy map M t The progressive construction of can be realized through the following mathematical expression:

[0051]

[0052] If point p i In any bounding box of the future state of the fruit, then Φ(p i )=+γ, otherwise -1, is the fused target classification score rescaled by the factor λ. represents the conversion from camera frame to world frame, Voxelize(·) represents the operation of converting the world frame point cloud into a two-dimensional grid, and M0 is initialized to a 2D grid with all zeros.

[0053] Therefore, the present invention provides an automatic fruit counting method based on three-dimensional point clouds. By feeding the accumulated memory information of the object state changing over time back to the detection module, the accuracy and tracking efficiency of the target detector when detecting 3D targets are significantly improved, and continuous positioning, tracking and identification of objects in three-dimensional space are achieved, solving the problems of repeated counting of fruits and counting of fruits outside the area of ​​interest.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A fruit automatic counting method based on three-dimensional point cloud, characterized in that: The specific steps are as follows: S1. Use LIDAR camera to capture the point cloud sequence data of the orchard scene and its corresponding camera motion trajectory information, and obtain the fruit occupancy map FOM by overlooking the three-dimensional scene; S2. Construct an efficient object detection module based on point cloud sequence data. This detection module uses a fast target proposal and state initial estimation network guided by spatiotemporal attention, and uses the fruit occupancy map FOM to screen out high-probability areas containing fruits and unknown areas. The detection module is divided into an early integration stage and an intermediate integration stage. Fruit detection is performed based on the fruit occupancy map FOM-guided RPN in the early integration stage and the FusionR-CNN fusion framework in the intermediate integration stage. S3. Based on the three-dimensional multi-target tracking algorithm AB3DMOT, a fruit tracking module based on point cloud sequence data is constructed to perform spatiotemporal memory-guided motion prediction and fruit occupancy map FOM update.

2. The automatic fruit counting method based on three-dimensional point cloud according to claim 1 is characterized in that: In S2, the early integration stage guides the RPN through the fruit occupancy map FOM. The specific steps are as follows: S211, according to the fruit occupation map FOM obtained in the previous frame, by selecting a threshold σ, it is divided into a high occupation area, a low occupation area and an unobserved area, and the area without fruit is discarded according to the time-fused fruit occupation map FOM; S212. After obtaining the fruit occupancy segmentation point cloud through the fruit occupancy map, the 3D object detection algorithm PointRCNN is used as the basic detection architecture, and the bin-based 3D Box Generation module is used as RPN. The search space is discretized into a series of cell bins to perform 3D detection in the fruit-dense area.

3. The automatic fruit counting method based on three-dimensional point cloud according to claim 1 is characterized in that: In S2, the intermediate integration stage is based on the Fusion R-CNN fusion framework to propose the current frame target box b prop And the target future state prediction box b of the previous frame pred The points are fused and combined to form the point cloud of the region of interest (RoI) and input into the detection box regression network R-CNN for detection box prediction.

4. The automatic fruit counting method based on three-dimensional point cloud according to claim 1 is characterized in that: In S3, the spatiotemporal memory-guided motion prediction uses the three-dimensional multi-target tracking algorithm AB3DMOT. The detection box prediction results in S2 are input into the Kalman filter. The 3D IoU association and Hungarian algorithm are used to quickly establish the correspondence between the current frame and the historical trajectory. At the same time, following the settings of the Kalman filter in AB3DMOT, at each time t, the Kalman filter predicts the fruit movement at t+1.

5. The method for automatic fruit counting based on three-dimensional point cloud according to claim 4, characterized in that: Motion prediction uses a constant velocity model independent of camera self-motion to describe the inter-frame displacement of fruits, producing a fused target state as the final future frame state prediction.

6. The method for automatic fruit counting based on three-dimensional point cloud according to claim 1, characterized in that: In S3, the fruit occupancy map FOM is updated based on the spatiotemporal memory information provided by the FOM constructed in the previous frame. The fruit occupancy map is updated in real time by integrating the point cloud sequence data received in the current frame and the fruit state prediction map for the future frame just generated by the motion prediction module.

Images