Multi-angle automatic detection and counting method for peanut plant fruits based on point cloud mapping

Abstract

The application provides a peanut plant fruit multi-angle automatic detection and statistics method based on point cloud mapping, relates to the technical field of point cloud mapping, and comprises the following steps: collecting an RGB-D image sequence by a depth camera to establish a point cloud dataset; matching fruit coordinates at adjacent time points and calculating a centroid position change and a volume change rate by using a Kalman filtering algorithm, determining a fruit growth trajectory, and repairing tracking interruption caused by occlusion; extracting morphological feature parameters to establish a growth curve, calculating a growth rate, and dividing growth stages. The application realizes accurate monitoring of peanut fruit growth, and improves the accuracy of peanut yield and quality prediction.

Description

Technical Field

[0001] This invention relates to point cloud mapping technology, and more particularly to a method for automatic multi-angle detection and statistical analysis of peanut plant fruits based on point cloud mapping. Background Technology

[0002] As an important grain and oil crop, monitoring the yield and quality of peanuts is of great significance for agricultural production and scientific research. Traditionally, the detection and statistics of peanut plants and fruits have mainly relied on manual observation and sampling surveys. This method is not only time-consuming and labor-intensive, but also makes it difficult to achieve continuous and dynamic monitoring of the growth status of peanut plants and fruits.

[0003] With the development of computer vision and deep learning technologies, image-based crop monitoring methods have been widely applied. Currently, in the field of peanut fruit detection and statistics, researchers have begun to use two-dimensional image processing technology to identify and count peanut fruits. Meanwhile, the application of three-dimensional point cloud technology in crop morphology analysis is also gradually increasing, providing new methods for precise fruit location and volume measurement.

[0004] Existing methods for detecting peanut fruits are mostly based on two-dimensional image analysis from a single angle, which makes it difficult to fully capture the three-dimensional structure of the peanut plant. They are also easily affected by leaf shading, resulting in a high rate of missed detection, especially for peanut fruits that grow below the soil surface, where the detection effect is even more limited.

[0005] Existing technologies lack a continuous tracking mechanism for the growth trajectory of peanut fruits, making it difficult to effectively address changes in fruit position and shading caused by plant growth and environmental changes. It is also difficult to achieve continuous monitoring of the same fruit throughout the entire growth cycle and cannot accurately record the complete development process of the fruit from flowering to maturity. Summary of the Invention

[0006] This invention provides a method for automatic multi-angle detection and statistical analysis of peanut plant fruits based on point cloud mapping, which can solve the problems in the prior art.

[0007] A first aspect of this invention provides a method for automatic multi-angle detection and statistical analysis of peanut plant fruits based on point cloud mapping, comprising: RGB-D image sequences of peanut plants were acquired using a depth camera to establish a point cloud dataset of peanut plants. After preprocessing the point cloud dataset, three-dimensional spatial coordinate information was extracted. Based on the three-dimensional spatial coordinate information, the Kalman filter algorithm is used to match the three-dimensional coordinates of the fruit at adjacent time points, calculate the change in the centroid position and the rate of change in volume of the matched fruit, determine the fruit growth trajectory based on the change in the centroid position and the rate of change in volume, and repair the fruit growth trajectory when tracking is interrupted due to occlusion. The growth trajectory of the fruit is analyzed to extract morphological characteristic parameters of the fruit during the growth process. A growth curve is established using the morphological characteristic parameters. The fruit growth rate is calculated and the growth stages are divided based on the growth curve. At the same time, abnormal growth states are identified and marked. A dynamic monitoring report is generated based on the fruit growth rate, the growth stages and the abnormal growth states.

[0008] RGB-D image sequences of peanut plants were acquired using a depth camera to establish a point cloud dataset of peanut plants. After preprocessing the point cloud dataset, three-dimensional spatial coordinate information was extracted, including: Obtain the intrinsic and extrinsic calibration parameters of the depth camera, and fix the depth camera above the peanut plant collection area; The exposure parameters of the depth camera are adjusted by an adaptive exposure time control strategy. The adaptive exposure time control strategy dynamically calculates the optimal exposure time based on the image brightness histogram distribution, and a clear RGB-D image is acquired according to the optimal exposure time. The original point cloud dataset of peanut plants was constructed based on the clear RGB-D image sequence. The original point cloud dataset is preprocessed by using a bilateral filtering algorithm to remove noise points, preserve plant edge features, extract plant point clouds, remove background point clouds, and register and fuse multiple frames of point cloud data. The three-dimensional spatial coordinate information of the main stem, branches and fruits of peanut plants was extracted based on the preprocessed point cloud dataset.

[0009] Based on the aforementioned three-dimensional spatial coordinate information, a Kalman filter algorithm is used to match the three-dimensional coordinates of the fruit at adjacent time points. The calculation of the change in the centroid position and the rate of change in volume of the matched fruit includes: The three-dimensional spatial coordinate information of peanut plant fruits at adjacent time points is obtained. Based on the three-dimensional spatial coordinate information, a fruit state vector is constructed. The state vector contains the position coordinates and motion velocity components of the fruit. The Kalman filter algorithm is used to predict the fruit state vector to obtain the predicted position of the fruit at the next time point. The matching probability is calculated using the predicted position and the actual collected three-dimensional coordinate information. The correspondence between fruits at adjacent time points is established based on the matching probability to determine the optimal matching fruit pair. The centroid coordinates are calculated for the optimal matching fruit pair, and the change in centroid position at adjacent time points is calculated based on the centroid coordinates. Obtain the volume data of the optimal matching fruit pair at adjacent time points, and calculate the volume change rate based on the volume data.

[0010] The fruit growth trajectory is determined based on the change in centroid position and the rate of volume change, and the fruit growth trajectory is repaired when tracking is interrupted due to occlusion, including: The direction and speed of the fruit's movement are calculated based on the change in the centroid position, and the growth rate of the fruit is calculated in combination with the volume change rate. Based on the direction of movement, the speed of movement, and the growth rate, a sequence of fruit growth states is constructed, and the sequence of fruit growth states is determined as the fruit growth trajectory. The continuity of the fruit growth trajectory is monitored, and when an interruption in tracking is detected, the sequence of the fruit growth state before the interruption is obtained. Based on the fruit growth state sequence before the interruption, trajectory repair is performed. The fruit movement trend and growth trend before the interruption are calculated, and multiple candidate search points are set in the occluded area. The candidate search points are scored according to the movement trend and the growth trend, and the candidate search point with the highest score is selected as the trajectory connection point. The continuity of the fruit growth trajectory is restored using the trajectory connection point to obtain the repaired fruit growth trajectory.

[0011] Growth analysis is performed on the fruits along their growth trajectory, and morphological characteristic parameters of the fruits during the growth process are extracted. A growth curve is then established using these morphological characteristic parameters, including: Three-dimensional point cloud data of fruit in the fruit growth trajectory is obtained, fruit morphological feature parameters are extracted based on the three-dimensional point cloud data, a local coordinate system is constructed by calculating the main axis direction of the fruit, and the fruit morphological feature parameters are calculated in the local coordinate system. Record the morphological characteristic parameters of fruits at different growth stages to establish a time-series characteristic dataset; perform piecewise fitting on the time-series characteristic dataset, determine growth nodes based on the volume growth rate and morphological change rate, and generate growth curves by fitting piecewise polynomial functions between adjacent growth nodes.

[0012] The time-series feature dataset is piecewise fitted, and growth nodes are determined based on the volume growth rate and morphological change rate. A piecewise polynomial function is used to fit growth curves between adjacent growth nodes, including: The volume growth rate and morphological change rate are calculated for the time-series feature dataset, and the abrupt change points of the volume growth rate and morphological change rate are identified as growth nodes; data segments between adjacent growth nodes are determined, and the start and end times of each data segment are recorded; curve fitting is performed on each data segment, and the curve fitting adopts a piecewise polynomial function. The piecewise polynomial functions obtained from fitting each data segment are connected to generate a complete fruit growth curve.

[0013] A second aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0014] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0015] The beneficial effects of this application are as follows: By acquiring RGB-D image sequences of peanut plants using a depth camera and establishing a point cloud dataset, a comprehensive acquisition of the three-dimensional spatial information of peanut fruits was achieved. Compared with traditional two-dimensional image detection methods, this method can more accurately capture the spatial distribution characteristics of the fruits.

[0016] The innovative method uses the Kalman filter algorithm to match the three-dimensional coordinates of the fruit at adjacent time points, and determines the fruit growth trajectory by the change in centroid position and the rate of change in volume, thus solving the detection difficulties caused by dense plant growth and mutual shading of fruits in the existing technology.

[0017] This invention can automatically repair the fruit growth trajectory when tracking is interrupted due to occlusion, effectively improving the continuity and integrity of peanut fruit detection and greatly reducing data loss problems.

[0018] By extracting morphological parameters during the fruit growth process and establishing growth curves, the growth rate of peanut fruits can be accurately calculated and the growth stages can be scientifically divided, providing precise growth monitoring data support for agricultural production. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the automatic multi-angle detection and statistical method for peanut plant fruits based on point cloud mapping, according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0022] Figure 1 This is a flowchart illustrating the automatic multi-angle detection and statistical method for peanut plant fruits based on point cloud mapping, as described in an embodiment of the present invention. Figure 1 As shown, the method includes: RGB-D image sequences of peanut plants were acquired using a depth camera to establish a point cloud dataset of peanut plants. After preprocessing the point cloud dataset, three-dimensional spatial coordinate information was extracted. Based on the three-dimensional spatial coordinate information, the Kalman filter algorithm is used to match the three-dimensional coordinates of the fruit at adjacent time points, calculate the change in the centroid position and the rate of change in volume of the matched fruit, determine the fruit growth trajectory based on the change in the centroid position and the rate of change in volume, and repair the fruit growth trajectory when tracking is interrupted due to occlusion. The growth trajectory of the fruit is analyzed to extract morphological characteristic parameters of the fruit during the growth process. A growth curve is established using the morphological characteristic parameters. The fruit growth rate is calculated and the growth stages are divided based on the growth curve. At the same time, abnormal growth states are identified and marked. A dynamic monitoring report is generated based on the fruit growth rate, the growth stages and the abnormal growth states.

[0023] In one optional implementation, RGB-D image sequences of peanut plants are acquired using a depth camera to establish a point cloud dataset of peanut plants. The point cloud dataset is then preprocessed to extract three-dimensional spatial coordinate information, including: Obtain the intrinsic and extrinsic calibration parameters of the depth camera, and fix the depth camera above the peanut plant collection area; The exposure parameters of the depth camera are adjusted by an adaptive exposure time control strategy. The adaptive exposure time control strategy dynamically calculates the optimal exposure time based on the image brightness histogram distribution, and a clear RGB-D image is acquired according to the optimal exposure time. The original point cloud dataset of peanut plants was constructed based on the clear RGB-D image sequence. The original point cloud dataset is preprocessed by using a bilateral filtering algorithm to remove noise points, preserve plant edge features, extract plant point clouds, remove background point clouds, and register and fuse multiple frames of point cloud data. The three-dimensional spatial coordinate information of the main stem, branches and fruits of peanut plants was extracted based on the preprocessed point cloud dataset.

[0024] The intrinsic and extrinsic calibration parameters of the depth camera were obtained. Camera calibration was performed using a standard checkerboard calibration board, acquiring multiple sets of images of the calibration board at different angles and positions. The intrinsic parameter matrix of the depth camera (including focal lengths fx and fy, and principal point coordinates cx and cy) and distortion coefficients (including radial distortion coefficients k1 and k2, and tangential distortion coefficients p1 and p2) were calculated using a calibration algorithm. Extrinsic parameter calibration primarily determined the camera's position and orientation in the actual acquisition environment, establishing the transformation relationship between the world coordinate system and the camera coordinate system. After calibration, the depth camera was fixed above the peanut plant acquisition area, with the camera's optical axis perpendicular to the ground, at a height of 1.5 meters, ensuring the entire plant was completely within the camera's field of view.

[0025] An adaptive exposure time control strategy is used to adjust the exposure parameters of the depth camera. A preview image is acquired, and its brightness histogram is calculated to obtain the average brightness value Lavg and standard deviation Lstd. Based on the target brightness value Ltarget (usually set to the median of the image grayscale value, 128), the brightness deviation ΔL = Ltarget - Lavg is calculated. When |ΔL| is greater than a preset threshold (e.g., 10), the exposure time is adjusted according to the formula Tnew = Tcurrent × (1 + k × ΔL), where k is an adjustment coefficient (ranging from 0.01 to 0.05), Tcurrent is the current exposure time, and Tnew is the adjusted exposure time. Furthermore, Lstd is used to determine if the image contrast is appropriate. When Lstd is lower than a threshold (e.g., 30), the exposure time is appropriately increased to improve contrast. After multiple iterative adjustments, the optimal exposure time is finally determined, resulting in a clear RGB-D image.

[0026] Based on optimized exposure parameters, a depth camera was used to capture images from multiple angles around the peanut plant, rotating 30 degrees at a time, resulting in a total of 12 RGB-D image sequences from different viewpoints. The RGB image resolution for each viewpoint was 1920×1080 pixels, and the depth map resolution was 640×480 pixels, with a depth accuracy of ±1mm. The RGB images and depth maps were registered using camera intrinsics to ensure accurate pixel correspondence.

[0027] A raw point cloud dataset of peanut plants is constructed based on the acquired clear RGB-D image sequence. For each frame of the RGB-D image, the depth value Z and corresponding pixel coordinates (u, v) of each pixel in the depth map are used, combined with the camera intrinsic parameter matrix, to calculate the point coordinates (X, Y, Z) in 3D space: X = (u-cx)×Z / fx, Y = (v-cy)×Z / fy. Simultaneously, the color information (R, G, B) of the corresponding pixels in the RGB image is mapped onto the 3D point to form a color point cloud. This process is repeated for each frame to generate a raw point cloud dataset containing both position and color information.

[0028] The original point cloud dataset is preprocessed, and a bilateral filtering algorithm is applied to remove noise points. Bilateral filtering considers both spatial distance and depth differences to prevent blurring of plant edge features. Specifically, for each point p, a set of points N(p) with radius r is searched within its neighborhood. The spatial weights are calculated as ws = exp(-||pq||). 2 / 2σs 2 ) and depth weight wd=exp(-|Zp-Zq| 2 / 2σd 2 (where q∈N(p)), the coordinates of point p are updated by weighted average. The parameters σs and σd control the spatial smoothness and depth similarity sensitivity, respectively, and are usually set to 5mm and 2mm.

[0029] Next, the plant point cloud and background point cloud were separated using a color thresholding method and the RANSAC plane fitting algorithm. A threshold range for green plants was set in the HSV color space (H: 35-85, S: 40-255, V: 20-255) to extract potential plant point clouds. Then, the RANSAC algorithm was used to fit the ground plane, identifying and removing points less than the threshold (e.g., 10mm) as background points, while retaining the plant point cloud.

[0030] To achieve registration and fusion of multi-frame point cloud data, the Iterative Closest Point (ICP) algorithm is employed. The first frame of the point cloud is selected as the reference frame. For each subsequent frame, initial correspondences are obtained through feature point extraction and matching (such as FPFH features). The rigid body transformation matrix is ​​then iteratively optimized until the mean square error is less than a preset threshold (e.g., 0.5 mm) or the maximum number of iterations (e.g., 50 times) is reached. Finally, all point clouds are transformed to the same coordinate system, and voxel mesh filtering (2 mm voxel size) is used for downsampling to reduce data redundancy.

[0031] Based on the preprocessed point cloud dataset, the three-dimensional spatial coordinates of the peanut plant's main stem, branches, and fruits were extracted. A region growing method and a cylinder fitting algorithm were used to identify the main stem and branches. First, seed points were selected from the bottom of the point cloud. Region growing was then performed based on normal consistency and spatial continuity to form the main stem point cloud. Then, the RANSAC cylinder fitting algorithm was used to extract the centerline of the main stem, obtaining its three-dimensional spatial coordinates. Branch identification employed a similar method, but branching points were detected from the main stem point cloud as seed points. For fruit identification, combining color features (mature fruits are yellowish-brown) and geometric features (approximately ellipsoidal), a Euclidean clustering algorithm was used to segment the potential fruit point cloud. Then, ellipsoidal fitting was used to obtain the fruit's position, size, and orientation information. Through these steps, accurate extraction of the three-dimensional spatial coordinates of the peanut plant's main stem, branches, and fruits was achieved.

[0032] In one optional implementation, based on the three-dimensional spatial coordinate information, a Kalman filter algorithm is used to match the three-dimensional coordinates of the fruit at adjacent time points, and the calculation of the change in the centroid position and the rate of change in volume of the matched fruit includes: The three-dimensional spatial coordinate information of peanut plant fruits at adjacent time points is obtained. Based on the three-dimensional spatial coordinate information, a fruit state vector is constructed. The state vector contains the position coordinates and motion velocity components of the fruit. The Kalman filter algorithm is used to predict the fruit state vector to obtain the predicted position of the fruit at the next time point. The matching probability is calculated using the predicted position and the actual collected three-dimensional coordinate information. The correspondence between fruits at adjacent time points is established based on the matching probability to determine the optimal matching fruit pair. The centroid coordinates are calculated for the optimal matching fruit pair, and the change in centroid position at adjacent time points is calculated based on the centroid coordinates. Obtain the volume data of the optimal matching fruit pair at adjacent time points, and calculate the volume change rate based on the volume data.

[0033] The three-dimensional spatial coordinates of peanut plants' fruits at adjacent time points are acquired using a 3D imaging device. This information includes the position coordinates (x, y, z) of each fruit at a specific moment. Based on the acquired 3D coordinate information, a state vector is constructed for each fruit. The state vector contains six components: three position coordinates (x, y, z) and three velocity components (vx, vy, vz). The position coordinates represent the fruit's position in three-dimensional space, while the velocity components represent its motion trend.

[0034] The Kalman filter algorithm is applied to predict the fruit's state vector. The Kalman filter algorithm consists of two main steps: prediction and update. In the prediction step, based on the current state vector and the motion model, the predicted position of the fruit at the next time step is calculated. The motion model assumes that the fruit moves at a constant speed, meaning that the position change is proportional to the speed and the time interval. The prediction equation can be expressed as: the predicted value of the state vector equals the state transition matrix multiplied by the current state vector.

[0035] The matching probability is calculated using the predicted location and the actual 3D coordinates acquired at the next moment. The matching probability is determined by calculating the Mahalanobis distance between the predicted and actual locations. The Mahalanobis distance takes into account the positional differences and their covariance, providing a probability score for each pair of matches. The smaller the distance, the higher the matching probability.

[0036] Based on the calculated matching probabilities, a correspondence is established between fruits at adjacent time steps. The Hungarian algorithm is then used to solve the multi-objective matching problem. This algorithm finds the global optimum by minimizing the overall matching cost. For each fruit at the current time step, the algorithm finds the best matching pair among all fruits at the next time step, forming the optimal matching fruit pair.

[0037] For each optimally matched fruit pair, the centroid coordinates are calculated. These coordinates are obtained by calculating the weighted average of the fruit's surface or volume distribution, representing the center point of the fruit's overall position. Based on the centroid coordinates at two consecutive time points, the change in centroid position is calculated, i.e., the Euclidean distance between the centroid positions at adjacent time points.

[0038] The volume data of the optimal matching fruit pair at adjacent time points is obtained. The volume data is calculated from the spatial coordinate point cloud using 3D reconstruction technology. Specifically, a closed volume is constructed around the point cloud using a convex hull algorithm or a spherical fitting method, and then the volume of this closed volume is calculated. Based on the volume data at two consecutive time points, the volume change rate, i.e., the ratio of the volume change to the initial volume, is calculated.

[0039] In practical applications, appropriate thresholds can be set to filter valid matching results. For example, when the matching probability is lower than a certain threshold, it is considered that no reliable match can be found; when the change in the centroid position exceeds the expected range, it indicates that the matching is wrong or the fruit is disturbed by external forces; when the volume change rate is abnormal, it indicates measurement error or abnormal fruit growth.

[0040] The Kalman filter algorithm updates the state vector and its covariance matrix after matching is complete, based on the actual observed fruit positions. The updated state vector serves as the basis for the next prediction, thus forming a recursive process.

[0041] In a practical application example, five fruits on a peanut plant were monitored for seven consecutive days, with three-dimensional data collected daily. The correspondence between the fruits was established using the method described above. The results showed that the matching accuracy using the Kalman filter algorithm reached 95%, significantly higher than the accuracy achieved by using only the nearest neighbor matching method. By calculating the change in centroid position, it was observed that the positional changes of most fruits followed a certain pattern, reflecting the drooping growth characteristic of peanut fruits. The volume change rate data clearly demonstrated the differences in growth rate of fruits at different developmental stages, providing a basis for agronomic management.

[0042] This method is particularly suitable for monitoring fruit growth under dense planting conditions, effectively solving the problems of overlapping and shading of fruits caused by high plant density. By continuously tracking the growth status of individual fruits, the impact of environmental factors and agronomic practices on peanut yield can be accurately assessed, providing a scientific basis for optimizing planting management.

[0043] In one optional implementation, determining the fruit growth trajectory based on the change in centroid position and the rate of volume change, and repairing the fruit growth trajectory when tracking is interrupted due to occlusion, includes: The direction and speed of the fruit's movement are calculated based on the change in the centroid position, and the growth rate of the fruit is calculated in combination with the volume change rate. Based on the direction of movement, the speed of movement, and the growth rate, a sequence of fruit growth states is constructed, and the sequence of fruit growth states is determined as the fruit growth trajectory. The continuity of the fruit growth trajectory is monitored, and when an interruption in tracking is detected, the sequence of the fruit growth state before the interruption is obtained. Based on the fruit growth state sequence before the interruption, trajectory repair is performed. The fruit movement trend and growth trend before the interruption are calculated, and multiple candidate search points are set in the occluded area. The candidate search points are scored according to the movement trend and the growth trend, and the candidate search point with the highest score is selected as the trajectory connection point. The continuity of the fruit growth trajectory is restored using the trajectory connection point to obtain the repaired fruit growth trajectory.

[0044] The direction and velocity of the fruit's motion are calculated based on the change in its center of mass position. At two consecutive time points t and t+1, the coordinates of the fruit's center of mass are obtained through 3D reconstruction as (x_t, y_t, z_t) and (x_{t+1}, y_{t+1}, z_{t+1}), respectively. The change in the center of mass position can be expressed as a vector (Δx, Δy, Δz) = (x_{t+1} - x_t, y_{t+1} - y_t, z_{t+1} - z_t). The direction of the fruit's motion is a unit vector of this vector, and the velocity is the magnitude of this vector divided by the time interval.

[0045] The growth rate of the fruit is calculated by combining the rate of volume change. At two consecutive time points, the fruit volumes are V_t and V_{t+1}, and the rate of volume change is (V_{t+1}-V_t) / V_t. The growth rate of the fruit can be expressed as the rate of volume change divided by the time interval.

[0046] Based on the direction of movement, speed of movement, and growth rate, a sequence of fruit growth states is constructed. The fruit growth state at each time point can be represented as a quintuple (x, y, z, v, g), where (x, y, z) are the coordinates of the fruit's centroid, v is the speed of movement, and g is the growth rate. Arranging the fruit growth states at multiple time points in chronological order forms a sequence of fruit growth states, which is thus determined as the fruit growth trajectory.

[0047] During fruit growth trajectory monitoring, it is necessary to monitor the continuity of the trajectory. If the time interval between two consecutive time points t and t+1 exceeds a preset threshold, or if no corresponding fruit is detected at time point t+1, it is determined that the tracking is interrupted. At this time, the fruit growth state sequence before the interruption is obtained, that is, the growth state sequence from time point 1 to t {(x_1, y_1, z_1, v_1, g_1), (x_2, y_2, z_2, v_2, g_2), ..., (x_t, y_t, z_t, v_t, g_t)}.

[0048] Trajectory repair is performed based on the fruit growth state sequence before the interruption. The movement trend of the fruit before the interruption is calculated. By performing linear regression on the centroid coordinates of n time points before the interruption, the changing trend of the centroid position of the fruit is obtained, and the centroid coordinates of future time points are predicted. Specifically, the changes of x, y, and z coordinates with time are linearly fitted to obtain three fitting functions: x=f_x(t), y=f_y(t), and z=f_z(t).

[0049] The growth trend of the fruit before interruption is calculated. By fitting the volume at n time points before the interruption using exponential or polynomial methods, the trend of fruit volume change is obtained, and the volume at future time points is predicted. Specifically, the function V=f_V(t) can be used to fit the change in volume over time.

[0050] Multiple candidate search points are set in the occluded area. The occluded area can be determined by the predicted direction of fruit movement. m candidate search points {P_1, P_2, ..., P_m} are evenly distributed in the area, and each search point corresponds to a hypothetical fruit centroid position.

[0051] Candidate search points are scored based on their movement and growth trends. For each candidate search point P_i, the distance d_i between it and the predicted centroid position is calculated, along with the difference δV_i between the volume assumed to be the fruit centroid and the predicted volume. Taking both factors into account, the candidate search points are scored using a scoring function designed as: Score_i = w_1 * (1 / d_i) + w_2 * (1 / δV_i), where w_1 and w_2 are weighting coefficients.

[0052] The candidate search point with the highest score is selected as the trajectory connection point. The coordinates of the candidate search point P_max with the highest score, the predicted movement speed, and the growth speed are used as the fruit growth state (x_{t+1}, y_{t+1}, z_{t+1}, v_{t+1}, g_{t+1}) at the interruption time point.

[0053] The continuity of the fruit growth trajectory is restored by using trajectory connection points. The sequence of fruit growth states before the interruption is merged with the growth states corresponding to the trajectory connection points to obtain the repaired fruit growth trajectory. If the occlusion lasts for multiple time points, the above process can be repeated to gradually restore the trajectory continuity.

[0054] In practical applications, the weighting coefficients in the scoring function can be adjusted according to the orchard environment and fruit characteristics. For example, for fruits that grow slowly but change position significantly, the value of w_1 can be increased; for fruits that grow rapidly but have a relatively fixed position, the value of w_2 can be increased. Furthermore, when calculating movement and growth trends, different regression models, such as linear, multinomial, or exponential models, can be selected to more accurately describe the growth characteristics of the fruit.

[0055] The above technical solution enables accurate tracking and repair of fruit growth trajectory, maintaining trajectory continuity even when fruit is obscured, thus providing reliable data support for fruit growth monitoring and yield prediction.

[0056] In one optional implementation, growth analysis is performed on the fruit along its growth trajectory, morphological characteristic parameters of the fruit during its growth process are extracted, and a growth curve is established using the morphological characteristic parameters, including: Three-dimensional point cloud data of fruit in the fruit growth trajectory is obtained, fruit morphological feature parameters are extracted based on the three-dimensional point cloud data, a local coordinate system is constructed by calculating the main axis direction of the fruit, and the fruit morphological feature parameters are calculated in the local coordinate system. Record the morphological characteristic parameters of fruits at different growth stages to establish a time-series characteristic dataset; perform piecewise fitting on the time-series characteristic dataset, determine growth nodes based on the volume growth rate and morphological change rate, and generate growth curves by fitting piecewise polynomial functions between adjacent growth nodes.

[0057] To acquire 3D point cloud data of a fruit along its growth trajectory, images can be captured using a multi-view camera or a depth camera. During acquisition, it's crucial to ensure that all surfaces of the fruit are fully captured to construct a complete 3D point cloud model. For round fruits like apples, at least 12 images from different perspectives are typically required; for elongated fruits like cucumbers, more perspectives along the long axis are needed. The acquired image data is then converted into a 3D point cloud using structured light algorithms or multi-view stereo vision algorithms, while simultaneously removing noise and outliers to improve point cloud quality.

[0058] Based on the acquired 3D point cloud data, the principal axis direction of the fruit is determined using principal component analysis (PCA). The covariance matrix of the point cloud is calculated, and then eigenvalue decomposition is performed on this matrix. The eigenvector corresponding to the largest eigenvalue is the principal axis direction of the fruit. A local coordinate system is established based on this principal axis direction, with the Z-axis set as the principal axis direction. The top of the fruit is selected as the origin, and the X-axis and Y-axis are perpendicular to the Z-axis and to each other, forming a right-handed coordinate system.

[0059] In the established local coordinate system, the morphological parameters of the fruit are calculated, including: volume, surface area, length, maximum diameter, centroid position, and curvature distribution. Volume is calculated using the convex hull voxelization method, converting the point cloud into a voxel mesh, counting the number of voxels occupied and multiplying by the voxel volume; surface area is obtained by summing the areas of all triangular facets after triangulating the point cloud; fruit length is the maximum distance along the principal axis; maximum diameter is the maximum cross-sectional diameter perpendicular to the principal axis; centroid position is obtained by calculating the average coordinates of all points; curvature distribution is achieved by fitting a local surface and calculating the principal curvature values.

[0060] To analyze the fruit's growth process, it is necessary to collect 3D point cloud data of the same fruit regularly, ideally weekly, from fruit set to maturity. For fast-growing fruits like cucumbers, the frequency can be shortened to every 3 days; for fruits with longer growth cycles like apples, the frequency can be extended to every 10 days. After each collection, morphological feature parameters are extracted using the method described above, and the collection time is recorded to form a time-series feature dataset. This dataset contains various morphological parameters of the fruit at different growth stages and their corresponding timestamp information.

[0061] Before segmenting and fitting the constructed time-series feature dataset, key nodes in fruit growth are identified, and the volume growth rate between adjacent time points is calculated using the formula: the difference between the current volume and the volume at the previous time point divided by the volume at the previous time point. Simultaneously, the rate of morphological change is calculated, including the rate of change of characteristic parameters such as aspect ratio and surface area to volume ratio. Growth nodes are identified when a significant inflection point appears in the volume growth rate or morphological change rate. Fruit growth typically consists of 3-4 stages: initial growth stage, rapid expansion stage, slowing growth stage, and ripening stage.

[0062] After determining the growth nodes, piecewise polynomial functions are used to fit between adjacent growth nodes. For each growth stage, a polynomial function of appropriate order is selected based on its characteristics. The initial growth stage typically exhibits exponential growth, so a second-order polynomial can be used; the rapid expansion stage exhibits linear or slightly above-linear growth, so a first- or second-order polynomial can be used; the growth rate decreases in the slowing growth stage, so a third-order polynomial can be used; in the mature stage, volume changes are not significant, but morphological parameters still change, so a low-order polynomial can be used. The least squares method is used to determine the polynomial coefficients during fitting, ensuring that the values ​​of adjacent polynomial functions are continuous at the growth nodes.

[0063] By connecting the piecewise polynomial functions obtained from the fitting, a complete fruit growth curve is formed. This curve describes the changes in various morphological parameters of the fruit over time throughout its growth process, from fruit set to maturity. The application value of the growth curve lies in predicting the growth trend of the fruit, assisting agricultural producers in yield estimation and harvest time planning. For example, the volume growth curve can predict the final size of the fruit at maturity; the curve showing the change in the center of gravity can predict the changes in the fruit's hanging posture, providing a basis for harvesting operations.

[0064] By comparing the growth curves of fruits from different varieties and under different cultivation conditions, key factors affecting fruit growth can be identified, providing decision support for agricultural production. For example, comparing the growth curves of tomato fruits under normal and water-deficient conditions reveals that the fruit volume growth rate significantly decreases during the rapid expansion phase under water-deficient conditions, while the surface area to volume ratio increases, indicating a change in fruit morphology. This information helps agricultural producers optimize irrigation strategies and improve water resource utilization efficiency.

[0065] In one optional implementation, the time-series feature dataset is piecewise fitted, growth nodes are determined based on the volume growth rate and morphological change rate, and a growth curve is generated by fitting a piecewise polynomial function between adjacent growth nodes, including: The volume growth rate and morphological change rate are calculated for the time-series feature dataset, and the abrupt change points of the volume growth rate and morphological change rate are identified as growth nodes; data segments between adjacent growth nodes are determined, and the start and end times of each data segment are recorded; curve fitting is performed on each data segment, and the curve fitting adopts a piecewise polynomial function. The piecewise polynomial functions obtained from fitting each data segment are connected to generate a complete fruit growth curve.

[0066] In monitoring fruit growth, analyzing and processing the collected time-series characteristic datasets is crucial for accurately reflecting the changing patterns during fruit development. Based on the differences in fruit characteristics at different growth stages, a piecewise fitting method can be used to construct growth curves, which can more accurately describe the entire developmental process of the fruit from immature to mature.

[0067] The time-series feature dataset is first preprocessed to ensure data continuity and reliability. Preprocessing includes outlier removal and missing value imputation to guarantee the accuracy of subsequent analysis. The preprocessed dataset contains data on the changes in parameters such as fruit volume and shape over time.

[0068] Based on the preprocessed time-series data, the volume growth rate and morphological change rate are calculated. The volume growth rate can be obtained by dividing the volume difference between adjacent time points by the time interval, representing the rate of increase in fruit volume per unit time. The morphological change rate is obtained by calculating the change in morphological characteristic parameters (such as aspect ratio, roundness, etc.) between adjacent time points and dividing by the time interval, reflecting the rate of change in fruit morphology.

[0069] The calculated volume growth rate and morphological change rate are analyzed to identify abrupt change points. Abrupt change points can be identified using a sliding window method combined with standard deviation analysis. When the growth rate or change rate at a certain time point exceeds twice the standard deviation of the average value of the preceding and following time windows, it can be marked as a potential abrupt change point. By comprehensively analyzing the abrupt changes in volume growth rate and morphological change rate, the location of growth nodes is determined.

[0070] Growth nodes typically correspond to key transition points in fruit growth and development, such as the transition from cell division to cell expansion, or from fruit enlargement to ripening. Each growth node records its corresponding time information and relevant feature parameter values, serving as the basis for piecewise fitting.

[0071] Based on the determined growth nodes, the entire time series data is divided into multiple data segments. The start and end times of each data segment are recorded, and all sampled data points within that time segment are extracted to prepare for subsequent segmented fitting.

[0072] Curve fitting is performed separately for each data segment using a polynomial function. The appropriate polynomial order is selected based on the complexity of the data segment. For data segments exhibiting exponential growth in the early stages, a second or third-order polynomial can be chosen; for data segments showing a gradual plateau in the middle and later stages of growth, a first or second-order polynomial can be selected. During the fitting process, the least squares method is used to determine the polynomial coefficients, minimizing the sum of squared errors between the fitted curve and the actual data points.

[0073] To ensure a smooth transition of the fitted curve between adjacent data segments, constraints are applied at the junctions of the piecewise polynomial function. These constraints include continuity of function values ​​and continuity of the first derivative, ensuring that the curve has no jumps or abrupt changes at the growth nodes. By solving for the polynomial coefficients that satisfy these constraints, a smooth piecewise polynomial function is obtained.

[0074] By connecting the piecewise polynomial functions fitted to each data segment, a complete fruit growth curve is formed. This curve accurately reflects the volume change pattern of the fruit at different growth stages, providing a basis for fruit growth prediction and management decisions.

[0075] In practical applications, taking apple fruit growth as an example, data such as the diameter and height of the fruit are collected weekly to calculate volume and morphological characteristics. Analysis reveals three distinct growth nodes in the apple's development: the first occurs approximately 20 days after pollination, corresponding to the transition from cell division to cell expansion; the second occurs approximately 60 days after pollination, corresponding to the transition from rapid expansion to slow expansion; and the third occurs approximately 100 days after pollination, corresponding to the beginning of ripening. Between these three growth nodes, 3rd, 2nd, and 1st order polynomial functions are used for fitting, respectively, resulting in piecewise growth curves that accurately describe the entire growth process of the apple from immature fruit to maturity.

[0076] The fruit growth curve constructed using this segmented fitting method can not only accurately reflect the stage characteristics of fruit growth, but also provide a scientific basis for precision agricultural management, such as optimizing irrigation strategies and determining appropriate harvesting times, thereby improving fruit yield and quality.

[0077] A second aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0078] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0079] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic multi-angle detection and statistical analysis of peanut plant fruits based on point cloud mapping, characterized in that, include: RGB-D image sequences of peanut plants were acquired using a depth camera to establish a point cloud dataset of peanut plants. After preprocessing the point cloud dataset, three-dimensional spatial coordinate information was extracted. Based on the three-dimensional spatial coordinate information, the Kalman filter algorithm is used to match the three-dimensional coordinates of the fruit at adjacent time points, calculate the change in the centroid position and the rate of change in volume of the matched fruit, determine the fruit growth trajectory based on the change in the centroid position and the rate of change in volume, and repair the fruit growth trajectory when tracking is interrupted due to occlusion. The growth trajectory of the fruit is analyzed to extract morphological characteristic parameters of the fruit during the growth process. A growth curve is established using the morphological characteristic parameters. The fruit growth rate is calculated and the growth stages are divided based on the growth curve. At the same time, abnormal growth states are identified and marked. A dynamic monitoring report is generated based on the fruit growth rate, the growth stages and the abnormal growth states.

2. The method according to claim 1, characterized in that, RGB-D image sequences of peanut plants were acquired using a depth camera to establish a point cloud dataset of peanut plants. After preprocessing the point cloud dataset, three-dimensional spatial coordinate information was extracted, including: Obtain the intrinsic and extrinsic calibration parameters of the depth camera, and fix the depth camera above the peanut plant collection area; The exposure parameters of the depth camera are adjusted by an adaptive exposure time control strategy. The adaptive exposure time control strategy dynamically calculates the optimal exposure time based on the image brightness histogram distribution, and a clear RGB-D image is acquired according to the optimal exposure time. The original point cloud dataset of peanut plants was constructed based on the clear RGB-D image sequence. The original point cloud dataset is preprocessed by using a bilateral filtering algorithm to remove noise points, preserve plant edge features, extract plant point clouds, remove background point clouds, and register and fuse multiple frames of point cloud data. The three-dimensional spatial coordinate information of the main stem, branches and fruits of peanut plants was extracted based on the preprocessed point cloud dataset.

3. The method according to claim 1, characterized in that, Based on the aforementioned three-dimensional spatial coordinate information, a Kalman filter algorithm is used to match the three-dimensional coordinates of the fruit at adjacent time points. The calculation of the change in the centroid position and the rate of change in volume of the matched fruit includes: The three-dimensional spatial coordinate information of peanut plant fruits at adjacent time points is obtained. Based on the three-dimensional spatial coordinate information, a fruit state vector is constructed. The state vector contains the position coordinates and motion velocity components of the fruit. The Kalman filter algorithm is used to predict the fruit state vector to obtain the predicted position of the fruit at the next time point. The matching probability is calculated using the predicted position and the actual collected three-dimensional coordinate information. The correspondence between fruits at adjacent time points is established based on the matching probability to determine the optimal matching fruit pair. The centroid coordinates are calculated for the optimal matching fruit pair, and the change in centroid position at adjacent time points is calculated based on the centroid coordinates. Obtain the volume data of the optimal matching fruit pair at adjacent time points, and calculate the volume change rate based on the volume data.

4. The method according to claim 1, characterized in that, The fruit growth trajectory is determined based on the change in centroid position and the rate of volume change, and the fruit growth trajectory is repaired when tracking is interrupted due to occlusion, including: The direction and speed of the fruit's movement are calculated based on the change in the centroid position, and the growth rate of the fruit is calculated in combination with the volume change rate. Based on the direction of movement, the speed of movement, and the growth rate, a sequence of fruit growth states is constructed, and the sequence of fruit growth states is determined as the fruit growth trajectory. The continuity of the fruit growth trajectory is monitored, and when an interruption in tracking is detected, the sequence of the fruit growth state before the interruption is obtained. Based on the fruit growth state sequence before the interruption, trajectory repair is performed. The fruit movement trend and growth trend before the interruption are calculated, and multiple candidate search points are set in the occluded area. The candidate search points are scored according to the movement trend and the growth trend, and the candidate search point with the highest score is selected as the trajectory connection point. The continuity of the fruit growth trajectory is restored using the trajectory connection point to obtain the repaired fruit growth trajectory.

5. The method according to claim 1, characterized in that, Growth analysis is performed on the fruits along their growth trajectory, and morphological characteristic parameters of the fruits during the growth process are extracted. A growth curve is then established using these morphological characteristic parameters, including: Three-dimensional point cloud data of fruit in the fruit growth trajectory is obtained, fruit morphological feature parameters are extracted based on the three-dimensional point cloud data, a local coordinate system is constructed by calculating the main axis direction of the fruit, and the fruit morphological feature parameters are calculated in the local coordinate system. Record the morphological characteristic parameters of fruits at different growth stages to establish a time-series characteristic dataset; perform piecewise fitting on the time-series characteristic dataset, determine growth nodes based on the volume growth rate and morphological change rate, and generate growth curves by fitting piecewise polynomial functions between adjacent growth nodes.

6. The method according to claim 5, characterized in that, The time-series feature dataset is piecewise fitted, and growth nodes are determined based on the volume growth rate and morphological change rate. A piecewise polynomial function is used to fit growth curves between adjacent growth nodes, including: The volume growth rate and morphological change rate are calculated for the time-series feature dataset, and the abrupt change points of the volume growth rate and morphological change rate are identified as growth nodes; data segments between adjacent growth nodes are determined, and the start and end times of each data segment are recorded; curve fitting is performed on each data segment, and the curve fitting adopts a piecewise polynomial function. The piecewise polynomial functions obtained from fitting each data segment are connected to generate a complete fruit growth curve.

7. A multi-angle automatic detection and statistical system for peanut plant fruits based on point cloud mapping, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to acquire RGB-D image sequences of peanut plants through a depth camera, establish a point cloud dataset of peanut plants, and extract three-dimensional spatial coordinate information after preprocessing the point cloud dataset. The second unit is used to match the three-dimensional coordinates of the fruit at adjacent time points using the Kalman filter algorithm based on the three-dimensional spatial coordinate information, calculate the change in the centroid position and the rate of change in volume of the matched fruit, determine the fruit growth trajectory based on the change in the centroid position and the rate of change in volume, and repair the fruit growth trajectory when tracking is interrupted due to occlusion. The third unit is used to perform growth analysis on the fruits in the fruit growth trajectory, extract morphological characteristic parameters of the fruits during the growth process, establish a growth curve using the morphological characteristic parameters, calculate the fruit growth rate and divide the growth stages based on the growth curve, and identify and mark abnormal growth states; and generate a dynamic monitoring report based on the fruit growth rate, the growth stages and the abnormal growth states.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images