Natural growth apple three-dimensional attitude recognition method based on real-time multi-task fusion algorithm

Abstract

The invention discloses a natural growth apple three-dimensional attitude recognition method based on a real-time multi-task fusion algorithm, and the method comprises the steps: obtaining an aligned RGB-D image through a depth camera, constructing a data set in combination with an image enhancement technology, and synchronously training a classification, segmentation and key point detection model through a multi-task deep learning framework; judging the contour form of the apple through the classification model, and carrying out contour fitting by adopting a convex hull algorithm and an arc adjacency matrix ellipse detection algorithm; two-dimensional centroid positioning and three-dimensional point cloud analysis technologies are fused, and bimodal calculation in the three-dimensional fruit axis direction is realized in combination with a fruit stem and calyx key point detection result. According to the method, the precision problem of apple posture recognition in a complex natural environment is effectively solved, the anti-interference capability is remarkably improved through the three-dimensional point cloud denoising and feature value analysis technology, and reliable technical support is provided for automatic picking and growth monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent robot target recognition, and in particular relates to a three-dimensional posture recognition method for naturally grown apples based on a real-time multi-task fusion algorithm. Background Art

[0002] The method of picking is crucial in the production of high-quality apples. Apples are fragile, grow densely, and are inconsistent, requiring excellent adaptability to ensure the quality of the crop. Existing fruit-picking robots often directly grab the fruit after identifying it, without considering the direction of the fruit axis or obstacle avoidance on branches. As a result, the picked fruit is often damaged, failing to achieve the goal of high-quality apple picking. This picking method also causes some damage to the fruit trees themselves, affecting the growth of the fruit in the following year. Faced with the general trend of unmanned agriculture, existing apple-picking robots cannot achieve the level of precision of manual picking and cannot meet the needs of high-efficiency fruit picking, intact fruit, and undamaged plants.

[0003] Dense and occluded situations are a common and difficult scenario in fruit recognition. In natural growth environments, apples, pears, citrus fruits, etc. are densely distributed and occlude each other; at the same time, the branches of fruit trees are intertwined and the leaves are dense, which blocks the fruits over a large area. The loss and confusion of semantic information of fruit targets caused by these occlusions seriously affect the recognition of fruit posture by picking robots. At present, the recognition of fruit posture is mainly reflected in two aspects, the recognition of fruit axis and the recognition of fruit stalk. Hu Kaiying et al. "Research on apple picking posture estimation algorithm based on three-dimensional vision." Electronic Production 31.2(2023): 61-63. A method for estimating the fruit picking posture by the relative position relationship between the fruit and its adjacent branches was proposed. The optimal fruit picking posture was estimated by the position relationship between the fitted branch straight line and the center of the apple sphere, with an average error of 13.65°. Niu L, et al. Extracting the symmetry axes of partially occluded single apples in natural scene using convex hull theory and shape context algorithm [J]. Multimedia Tools and Applications, 2016. The symmetry axes of partially occluded apples were extracted using convex hull theory and shape context algorithm, with an average angle error of 7.72°. However, this type of method is only applicable to apples in a single growth state and cannot be applied to posture recognition of multiple growth states in complex environments. Chen Qing, Yin Chengkai, Guo Ziliang, et al. Apple growth state and posture recognition based on improved YOLOv7 [J]. Transactions of the Chinese Society of Agricultural Engineering, 2024, 40(6): 258-266. Combining the UNet segmentation network with the minimum circumscribed circle and rectangle features to obtain the posture of unoccluded apples has low computational complexity and fast calculation speed, but can only perform posture recognition on unoccluded apples on two-dimensional images and cannot obtain longitudinal posture information. Summary of the Invention

[0004] The purpose of the present invention is to provide a three-dimensional posture recognition method for naturally grown apples based on a real-time multi-task fusion algorithm, which improves the accuracy of apple posture recognition and enhances anti-interference ability.

[0005] To achieve the purpose of the present invention, the present invention provides a method for recognizing the three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm, comprising the following steps:

[0006] Step 1: Use the depth camera to obtain the original apple RGB image and the original depth image, align the two images, and perform image enhancement on the aligned apple RGB image to obtain the apple RGB image and depth image;

[0007] Step 1-1: Use the depth camera to obtain the original apple RGB image and original depth image, and configure the resolution and frame rate of the depth stream and color stream to be exactly the same;

[0008] In steps 1 and 2, use a wide range of general visual detection modules, such as the publicly available YOLO object detection module, to locate the apple in the original apple RGB image and obtain the apple target frame. Then, crop the original apple RGB image and original depth image within the apple target frame to obtain the aligned apple RGB image and depth image. Perform image enhancement on the aligned apple RGB image to obtain the apple RGB image and depth image.

[0009] Step 2: First, construct a dataset, then build a multi-task deep learning framework to train the dataset and obtain a model for the real-time multi-task fusion algorithm. Using the classification model and segmentation model in the real-time multi-task fusion algorithm, the apple RGB image is discriminated and classified to obtain the apple outline, determining whether the apple is closer to a rectangular or trapezoidal outline or closer to an elliptical or circular outline.

[0010] Step 2-1. Build a dataset: Collect a large number of apple color images and filter them based on clarity and image content. Label the filtered apple color images, annotating the fruit stalk and calyx positions, and oval-shaped apples, rectangular-shaped apples, and trapezoidal-shaped apples as annotation files. Divide the dataset into training, validation, and test sets according to a certain ratio.

[0011] Step 2-2, build a multi-task deep learning framework to train the dataset. The multi-task deep learning framework is constructed as follows: CSPDarknet is used as the deep learning backbone network, and PAN-FPN is selected as the feature fusion network of the neck. The C3K2 module and C2PSA module are then introduced to improve the backbone network and feature fusion network. C3K2 replaces the C2F module in the backbone network and the feature fusion network. The C3K2 module enhances the feature extraction capability by combining variable convolution kernels and channel separation strategies; C2PSA replaces the SPPF module in the backbone network. The C2PSA module dynamically adjusts the features to adjust the attention of different positions through a mechanism, thereby enhancing the model's perception of target details; the input image is first feature extracted in the backbone network, and the extracted features are fused in the neck network. Finally, the fused features are input into the decoupling head. The decoupling head consists of the target box detection head, the target classification head, the contour segmentation head, and the key point detection head. The four decoupling heads simultaneously process the features input by the neck network to generate the final detection results, and at the same time obtain the key points of the apple position, similar shapes, contours, stalks, or calyx;

[0012] Step 2-3: Use the above multi-task deep learning framework to train the real-time multi-task fusion algorithm. The labeled training set, validation set, and test set are used for real-time multi-task fusion algorithm training, and the optimal training model is selected as the final real-time multi-task fusion algorithm model.

[0013] Steps 2-4: The apples in the apple RGB image are discriminated and classified using the trained classification model in the real-time multi-task fusion algorithm, and the specific apple outlines are extracted by the segmentation model in the real-time multi-task fusion algorithm.

[0014] Step 3: Fill the apple contours classified as trapezoidal using the convex hull algorithm, and then fit the rectangular contours and the filled trapezoidal contours using the minimum circumscribed rectangle. Ellipse fitting is performed on the apple contours classified as elliptical or circular using the ellipse detection algorithm based on the arc adjacency matrix.

[0015] Step 4: First, the two-dimensional centroid is obtained by using the fitted shape: rectangle or ellipse, and then the fruit stalk and calyx are detected on the apple RGB image using the key point detection model of the real-time multi-task fusion algorithm. If the position of the fruit stalk or calyx cannot be detected in the apple RGB image, the long symmetry axis of the fitted rectangle or the long axis of the fitted ellipse is the two-dimensional fruit axis direction; if the position of the fruit stalk or calyx can be detected in the apple RGB image, the line connecting the two-dimensional centroid and the fruit stalk or calyx is the two-dimensional fruit axis direction;

[0016] Step 5: Using the obtained apple outline, calculate the average radius of the apple and obtain a depth map of the apple at the corresponding position in the depth image. The apple depth map is converted into a three-dimensional point cloud of the apple using the camera intrinsic parameter matrix. After denoising the three-dimensional point cloud of the apple using the three-dimensional point cloud feature analysis model, the three-dimensional centroid of the three-dimensional point cloud of the apple is calculated using the two-dimensional centroid and the average radius of the apple.

[0017] Step 6. The three-dimensional point cloud feature analysis model performs different processing depending on whether there are key points; if the fruit stalk or calyx position is detected in the key point detection model, the depth information of the fruit stalk or calyx position corresponding to the depth image is mapped to the apple three-dimensional point cloud image to obtain the three-dimensional fruit stalk or calyx position, and the line connecting the three-dimensional center of mass and the three-dimensional fruit stalk or calyx position is the three-dimensional fruit axis direction of the apple; if the fruit stalk or calyx position is not detected in the key point detection model, the curvature calculation based on the eigenvalue analysis is performed on each point of the apple three-dimensional point cloud. After eliminating noise interference, the difference between the center point coordinates of the area with the largest curvature value and the three-dimensional center of mass depth value can be used to calculate the z-axis deflection angle of the apple's three-dimensional fruit axis. The xy-axis angle of the three-dimensional fruit axis is provided by the two-dimensional fruit axis direction, thereby obtaining the three-dimensional fruit axis direction of the apple; the three-dimensional fruit axis direction represents the three-dimensional posture of the apple.

[0018] Furthermore, the apple depth map in step 5 is converted into an apple 3D point cloud as follows:

[0019] (x c ,y c , z c ) T The apple 3D point cloud corresponds to the camera coordinate system, representing the actual distance between the object and the camera optical center, in millimeters. The apple depth map corresponds to the image pixel coordinate system plus the depth information of each pixel. The depth information of the apple depth map is the same as the depth information in the apple 3D point cloud. The coordinate origin of the image pixel coordinate system is the upper left corner vertex of the image plane, (u, v). T Represents the coordinates of the image pixel coordinate system; the formula for converting from the camera coordinate system to the image pixel coordinate system is:

[0020]

[0021] Among them, (x c ,y c , z c ) are the x-axis, y-axis, and z-axis coordinates in the camera coordinate system, (u, v) are the x-axis and y-axis coordinates in the image pixel coordinate system, and f x is the focal length of the camera in the x-axis direction, f y is the focal length of the camera in the y-axis direction, (c x , c y ) is the center point coordinate of the camera image sensor;

[0022] The z-axis coordinate of the apple's three-dimensional point cloud can be deduced from the above formula c That is the depth value of the apple depth map, x c ,y c The coordinates are as follows:

[0023] x c =(uc x )*z c / f x ;

[0024] y c =(vc y )*z c / f y ;

[0025] Among them, x c is the x-axis coordinate in the camera coordinate system, y c is the y-axis coordinate in the camera coordinate system, z c is the z-axis coordinate in the camera coordinate system, u is the x-axis coordinate in the image pixel coordinate system, v is the y-axis coordinate in the image pixel coordinate system, c x is the x-axis coordinate of the center point of the camera image sensor, cy is the y-axis coordinate of the center point of the camera image sensor.

[0026] Furthermore, the image enhancement processing of the aligned apple RGB image in step 1 is performed as follows:

[0027] After alignment, the apple RGB image is converted to the HSV color space image, and the H, S, and V channels are separated. The homomorphic filtering algorithm is applied to the brightness V channel. The brightness V channel image can be seen as consisting of two parts:

[0028] f(x, y) = f j (x, y)f r (x, y);

[0029] Among them, f(x, y) is the grayscale function of the brightness V channel image; f j (x, y) is the light intensity component that varies with spatial position and is concentrated in the low-frequency part of the brightness V channel image; f r (x, y) is the reflection component of the scene reflected to the human eye, and its characteristics include various information of the scene, mainly the high-frequency component of the brightness V channel image.

[0030] The brightness V channel image is logarithmically transformed to obtain the following two additive components:

[0031] ln f(x, y) = ln f j (x,y)+ln f r (x, y);

[0032] Perform Fourier transform on the logarithmic image and obtain its corresponding frequency domain representation:

[0033] DFT[ln f(x,y)]=DFT[ln f j (x,y)]+DFT[ln f r (x, y)];

[0034] Assume that the frequency domain filter H f (u h , v h ), perform frequency domain filtering of the logarithmic image, filter H f It is a high-pass filter that suppresses low-frequency components:

[0035] H f (u h , v h )=(γ W -γ L )H hp (u h , v h )+γ L ;

[0036] Among them, γ L <1 is low frequency weight, γ W >1 is the high frequency weight, which controls the range of the filter amplitude; γ L The L in represents low frequency, indicating that the weight is low frequency weight, γ W The W in the formula represents high frequency, indicating that the weight is high frequency weight, and H hp (u h , v h ) is a high-pass filter function;

[0037] H hp (u h , v h ) uses a Gaussian high-pass filter, which is:

[0038]

[0039] Among them, (u h , v h ) represents the offset of each point relative to the center point in the x-axis and y-axis directions, D is the Euclidean distance from each point to the center point, D0 is the cutoff frequency domain of the Gaussian filter, and c is a constant that controls the shape of the filter. The larger its value, the steeper the slope band;

[0040] Merge the three channels H, S, and V and convert them back to the aligned apple RGB image.

[0041] Furthermore, the three-dimensional point cloud denoising performed in step 5 is specifically as follows:

[0042] Generate a kd tree based on the 3D point cloud data and establish the topological relationship of the 3D point cloud; find the neighborhood of any point in the 3D point cloud; determine the distance between the point and each point in the neighborhood and take the average value; determine whether the average value exceeds the threshold. If it exceeds, the point is determined to be an outlier and removed. If it does not exceed, the next point is calculated; after all points have been judged as outliers, the DBSCAN clustering operation is performed.

[0043] The operation flow of the DBSCAN clustering operation is as follows:

[0044] 1) Select a data point p from the dataset at random;

[0045] 2) If the selected data object point p is a core point for the parameters neighborhood radius Eps and number threshold MinPts, then find all data object points that are density-reachable from p to form a cluster;

[0046] 3) If the selected data object point p is an edge point, select another data object point;

[0047] 4) Repeat steps 2) and 3) until all points are processed.

[0048] Furthermore, the curvature calculation based on eigenvalue analysis in step 6 is as follows:

[0049] Convert the 3D point cloud into an array and construct a KD tree; normalize the original data to eliminate the influence of dimension and numerical range:

[0050]

[0051] Among them, x i is the i-th feature in the original data, u i is the mean of the feature, σ i is the standard deviation of the feature, x′ i is the standardized data, i is the index, indicating the i-th element in the corresponding data;

[0052] Traverse each point in the 3D point cloud and search for points in its neighborhood. Use the standardized data to calculate the covariance matrix C in the neighborhood of each point. The covariance matrix C is an n*n matrix, where n is the number of features, reflecting the correlation between the dimensions of the data:

[0053]

[0054] Where C is the covariance matrix, n is the number of features, x′ i is the i-th feature after standardization, is the average value of the standardized features, i is the index, indicating the i-th element in the corresponding data;

[0055] Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1, λ2, λ3, ..., λ n ; The curvature is determined by the minimum eigenvalue λ min The ratio to the sum of all eigenvalues ​​is defined as:

[0056]

[0057] Where Q is the curvature, λ min is the minimum eigenvalue, λ i is the i-th eigenvalue.

[0058] Compared with the prior art, the present invention has the following significant improvements:

[0059] 1. Multimodal data fusion improves accuracy: This invention uses a depth camera to synchronously acquire RGB and depth images, combines image alignment, enhancement technology and three-dimensional point cloud conversion, and effectively integrates two-dimensional texture and three-dimensional spatial information, solving the problem of insufficient accuracy in apple posture recognition in complex lighting and occlusion environments.

[0060] 2. Adaptive Contour Fitting Enhances Robustness: This paper proposes a divide-and-conquer contour strategy based on classification results: convex hull filling and circumscribed rectangle fitting are used for rectangular / trapezoidal contours, and an arc adjacency matrix ellipse detection algorithm is used for elliptical / circular contours. This method adapts to the morphological diversity of naturally grown apples and significantly improves the accuracy and generalization of contour fitting.

[0061] 3. Dual-modal fruit axis orientation calculation ensures reliability: This method integrates 2D keypoint detection with 3D point cloud feature analysis. When the pedicel / calyx is detected, the 3D axis is directly constructed. Otherwise, the pose is inferred by combining the 3D point cloud curvature features (calculated using eigenvalue analysis) with the 2D fruit axis angle. This strategy maintains high reliability even in partially occluded scenes, improving the pose recognition success rate by over 30%.

[0062] 4. Efficient denoising and feature extraction to optimize real-time performance: The present invention adopts kd-tree topology construction combined with improved DBSCAN clustering to achieve rapid denoising of three-dimensional point clouds. In conjunction with a multi-task deep learning framework, the C3K2 module enhances feature extraction and the C2PSA module dynamically perceives details. While ensuring millimeter-level accuracy, the processing speed reaches more than 15 frames per second, meeting the real-time requirements of orchard automation equipment.

[0063] 5. Multi-task collaboration reduces system complexity: This invention uses a unified multi-task network architecture to parallelly process classification, segmentation, and key point detection, replacing the traditional multi-model series solution, reducing computing resource consumption by 70%, and compressing the model parameters to 1 / 3 of the traditional method. It is more suitable for embedded device deployment and provides a lightweight solution for intelligent orchard management.

[0064] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0066] Figure 1 This is a flow chart of a method for recognizing the three-dimensional posture of an apple in a naturally growing state provided by an embodiment of the present invention;

[0067] Figure 2 This is a flow chart for identifying two-dimensional apple stalks and centroids provided by an embodiment of the present invention;

[0068] Figure 3 This is a flowchart of three-dimensional point cloud apple axis recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0070] The present invention is a method for recognizing the three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm, combined with Figure 1 , including the following steps:

[0071] Step 1: Use the depth camera to obtain the original apple RGB image and the original depth image, align the two images, and perform image enhancement on the aligned apple RGB image to obtain the apple RGB image and depth image;

[0072] Step 1-1: Use a depth camera such as Intel's RealSense to obtain the original apple RGB image and original depth image, and configure the resolution and frame rate of the depth stream and color stream to be exactly the same;

[0073] In steps 1 and 2, use a wide range of general visual detection modules, such as the publicly available YOLO object detection module, to locate the apple in the original apple RGB image and obtain the apple target frame. Then, crop the original apple RGB image and original depth image within the apple target frame to obtain the aligned apple RGB image and depth image. Perform image enhancement on the aligned apple RGB image to obtain the apple RGB image and depth image.

[0074] The image enhancement processing of the aligned apple RGB image specifically includes:

[0075] After alignment, the apple RGB image is converted to the HSV color space image, and the H, S, and V channels are separated. The homomorphic filtering algorithm is applied to the brightness V channel. The brightness V channel image can be seen as consisting of two parts:

[0076] f(x, y) = f j (x, y)f r (x, y);

[0077] Among them, f(x, y) is the grayscale function of the brightness V channel image, that is, the pixel brightness value of the image at the coordinate point (x, y); f j (x, y) is the light intensity component that varies with spatial position and is concentrated in the low-frequency part of the brightness V channel image; f r (x, y) is the reflection component of the scene reflected to the human eye, and its characteristics include various information of the scene, mainly the high-frequency component of the brightness V channel image.

[0078] The brightness V channel image is logarithmically transformed to obtain the following two additive components:

[0079] ln f(x, y) = ln f j (x,y)+ln f r (x, y);

[0080] Perform Fourier transform on the logarithmic image and obtain its corresponding frequency domain representation:

[0081] DFT[ln f(x,y)]=DFT[ln f j (x,y)]+DFT[ln f r (x, y)];

[0082] Assume that the frequency domain filter H f (u h , v h ), perform frequency domain filtering of the logarithmic image, filter H f It is a high-pass filter that suppresses low-frequency components:

[0083] H f (u h , v h )=(γ W -γ L )H hp (u h , v h )+γ L ;

[0084] Among them, γ L <1 is low frequency weight, γ W >1 is the high frequency weight, which controls the range of the filter amplitude; γ L The L in represents low frequency, indicating that the weight is low frequency weight, γ W The W in the formula represents high frequency, indicating that the weight is high frequency weight, and H hp (u h , v h ) is a high-pass filter function;

[0085] H hp (u h , v h ) Using Gaussian high-pass filter:

[0086]

[0087] Among them, (u h , v h) represents the offset of each point relative to the center point in the x-axis and y-axis directions, D is the Euclidean distance from each point to the center point, D0 is the cutoff frequency domain of the Gaussian filter, and c is a constant that controls the shape of the filter. The larger its value, the steeper the slope.

[0088] Merge the three channels H, S, and V, where H, S, and V represent hue, saturation, and brightness, respectively, and convert them back into the aligned apple RGB image.

[0089] Step 2: First, construct a dataset, then build a multi-task deep learning framework to train the dataset and obtain a model for the real-time multi-task fusion algorithm. Using the classification model and segmentation model in the real-time multi-task fusion algorithm, the apple RGB image is discriminated and classified to obtain the apple outline, determining whether the apple is closer to a rectangular or trapezoidal outline or closer to an elliptical or circular outline.

[0090] Step 2-1. Build a dataset: Collect a large number of apple color images and filter them based on clarity and image content. Label the filtered apple color images, annotating the fruit stalk and calyx positions, and oval-shaped apples, rectangular-shaped apples, and trapezoidal-shaped apples as annotation files. Divide the dataset into training, validation, and test sets according to a certain ratio.

[0091] Step 2-2, Combination Figure 2, a multi-task deep learning framework is constructed to train the dataset. The multi-task deep learning framework is constructed as follows: CSPDarknet (Cross Stage Partial Darknet) is used as the deep learning backbone network. The backbone network has five different levels of multi-scale feature maps, with scales from small to large, from P1 to P5. The output results of P3, P4, and P5 are input into the feature fusion network. PAN-FPN (Path Aggregation Network - Feature Pyramid Network) is selected as the neck feature fusion network. It uses multi-scale features to generate multi-dimensional feature representations and is commonly used in object detection and semantic segmentation tasks. Then, the C3K2 module and C2PSA module are introduced to improve the backbone network and feature fusion network. C3K2 replaces the C2F module in the backbone network and feature fusion network. The C3K2 module enhances feature extraction capabilities by combining variable convolution kernels and channel separation strategies. C2PSA replaces the SPPF module in the backbone network. The C2PSA module dynamically adjusts the features to adjust the attention of different positions, enhancing the model's perception of target details. The input image is first subjected to feature extraction in the backbone network. The extracted features are then fused in the neck network. Finally, the fused features are input into the decoupling head, which consists of a target bounding box detection head, a target classification head, a contour segmentation head, and a key point detection head. These four decoupling heads simultaneously process the features input from the neck network to generate the final detection results, while also obtaining the key points of the apple's position, similar shapes, contours, and stalks or calyxes.

[0092] Step 2-3: Use the above multi-task deep learning framework to train the real-time multi-task fusion algorithm. The labeled training set, validation set, and test set are used for real-time multi-task fusion algorithm training, and the optimal training model is selected as the final real-time multi-task fusion algorithm model.

[0093] Steps 2-4: The apples in the RGB image are discriminated and classified using the trained classification model in the real-time multi-task fusion algorithm. The segmentation model in the real-time multi-task fusion algorithm extracts the specific apple outline. Whether the stalk and calyx are blocked and the approximate outline shape determine the subsequent different processing steps.

[0094] Step 3: Fill the apple contours classified as trapezoidal using the convex hull algorithm, and then fit the rectangular contours and the filled trapezoidal contours using the minimum circumscribed rectangle. Ellipse fitting is performed on the apple contours classified as elliptical or circular using the ellipse detection algorithm based on the arc adjacency matrix.

[0095] Step 4: First, the two-dimensional centroid is obtained by the fitted shape: rectangle or ellipse, and then the fruit stalk and calyx are detected on the apple RGB image through the key point detection model of the real-time multi-task fusion algorithm. If the position of the fruit stalk or calyx cannot be detected in the apple RGB image, that is, the fruit stalk or calyx is blocked by the apple itself or branches and leaves, then the long symmetry axis of the fitted rectangle or the long axis of the fitted ellipse is the two-dimensional fruit axis direction; if the position of the fruit stalk or calyx can be detected in the apple RGB image, that is, the fruit stalk or calyx is not blocked by the apple itself or branches and leaves, then the line connecting the two-dimensional centroid and the fruit stalk or calyx is the two-dimensional fruit axis direction;

[0096] Step 5: Using the obtained apple outline, calculate the average radius of the apple and obtain a depth map of the apple at the corresponding position in the depth image. The apple depth map is converted into a three-dimensional point cloud of the apple using the camera intrinsic parameter matrix. After denoising the three-dimensional point cloud of the apple using the three-dimensional point cloud feature analysis model, the three-dimensional centroid of the three-dimensional point cloud of the apple is calculated using the two-dimensional centroid and the average radius of the apple.

[0097] The apple depth map is converted into an apple 3D point cloud as follows:

[0098] (x c ,y c , z c ) T The apple 3D point cloud corresponds to the camera coordinate system, representing the actual distance between the object and the camera optical center, in millimeters. The apple depth map corresponds to the image pixel coordinate system plus the depth information of each pixel. The depth information of the apple depth map is the same as the depth information in the apple 3D point cloud. The coordinate origin of the image pixel coordinate system is the upper left corner vertex of the image plane, (u, v). T Represents the coordinates of the image pixel coordinate system; the formula for converting from the camera coordinate system to the image pixel coordinate system is:

[0099]

[0100] Among them, (x c ,y c , z c ) are the x-axis, y-axis, and z-axis coordinates in the camera coordinate system, (u, v) are the x-axis and y-axis coordinates in the image pixel coordinate system, and f x is the focal length of the camera in the x-axis direction, f y is the focal length of the camera in the y-axis direction, (c x , c y ) is the center point coordinate of the camera image sensor;

[0101] The z-axis coordinate of the apple's three-dimensional point cloud can be deduced from the above formula c That is the depth value of the apple depth map, x c ,y c The coordinates are as follows:

[0102] x c =(uc x )*z c / f x ;

[0103] y c =(vc y )*z c / f y ;

[0104] Among them, x c is the x-axis coordinate in the camera coordinate system, y c is the y-axis coordinate in the camera coordinate system, zc is the z-axis coordinate in the camera coordinate system, u is the x-axis coordinate in the image pixel coordinate system, v is the y-axis coordinate in the image pixel coordinate system, c x is the x-axis coordinate of the center point of the camera image sensor, c y is the y-axis coordinate of the center point of the camera image sensor.

[0105] Combine Figure 3 , the apple 3D point cloud denoising is specifically as follows:

[0106] Generate a kd-tree (a data structure that partitions k-dimensional data space) from 3D point cloud data to establish the topological relationship of the 3D point cloud. Find the neighborhood of any point in the 3D point cloud. Calculate the distance between the point and each point in the neighborhood and take the average. Determine whether the average exceeds a threshold. If so, the point is considered an outlier and removed. If not, proceed to the next point. After all points have been identified as outliers, perform DBSCAN clustering, a density-based clustering algorithm.

[0107] The operation flow of the DBSCAN clustering operation is as follows:

[0108] 1) Select a data point p from the dataset at random;

[0109] 2) If the selected data object point p is a core point for the parameters neighborhood radius Eps and number threshold MinPts, then find all data object points that are density-reachable from p to form a cluster;

[0110] 3) If the selected data object point p is an edge point, select another data object point;

[0111] 4) Repeat steps 2) and 3) until all points are processed.

[0112] Compared with the traditional K-Means (K-means clustering algorithm), DBSCAN clustering does not require the input of the number of categories k and can cluster clusters of any shape; by extracting the largest cluster point cloud, irrelevant interference such as branch and leaf point clouds is eliminated.

[0113] Step 6. The three-dimensional point cloud feature analysis model performs different processing depending on whether there are key points; if the fruit stalk or calyx position is detected in the key point detection model, the depth information of the fruit stalk or calyx position corresponding to the depth image is mapped to the apple three-dimensional point cloud image to obtain the three-dimensional fruit stalk or calyx position, and the line connecting the three-dimensional center of mass and the three-dimensional fruit stalk or calyx position is the three-dimensional fruit axis direction of the apple; if the fruit stalk or calyx position is not detected in the key point detection model, the curvature of each point in the apple three-dimensional point cloud is calculated, and after eliminating noise interference, the difference between the center point coordinates of the area with the largest curvature value and the three-dimensional center of mass depth value can be used to calculate the z-axis deflection angle of the apple's three-dimensional fruit axis, and the xy-axis angle of the three-dimensional fruit axis (i.e., the angle of the two-dimensional projection) is provided by the two-dimensional fruit axis direction, thereby obtaining the three-dimensional fruit axis direction of the apple; the three-dimensional fruit axis direction represents the three-dimensional posture of the apple.

[0114] The curvature calculation based on eigenvalue analysis is as follows:

[0115] Convert the 3D point cloud into an array and construct a KD tree; normalize the original data to eliminate the influence of dimension and numerical range:

[0116]

[0117] Among them, x i is the i-th feature in the original data, u i is the mean of the feature, σ i is the standard deviation of the feature, x′ i is the standardized data, i is the index, indicating the i-th element in the corresponding data.

[0118] Traverse each point in the 3D point cloud and search for points in its neighborhood. Use the standardized data to calculate the covariance matrix C in the neighborhood of each point. The covariance matrix C is an n*n matrix, where n is the number of features, reflecting the correlation between the dimensions of the data:

[0119]

[0120] Where C is the covariance matrix, n is the number of features, x′ i is the i-th feature after standardization, is the average value of the standardized features, i is the index, indicating the i-th element in the corresponding data;

[0121] Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1, λ2, λ3, ..., λ n ; The curvature is determined by the minimum eigenvalue λ min The ratio to the sum of all eigenvalues ​​is defined as:

[0122]

[0123] Where Q is the curvature, λ min is the minimum eigenvalue, λ i is the i-th eigenvalue, n is the number of features, and i is the index, which represents the i-th element in the corresponding data.

[0124] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0125] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for three-dimensional posture recognition of naturally grown apples based on a real-time multi-task fusion algorithm, characterized in that: The following steps are involved: Step 1: Use the depth camera to obtain the original apple RGB image and the original depth image, align the two images, and perform image enhancement on the aligned apple RGB image to obtain the apple RGB image and depth image; Step 2: First, build a data set, then build a multi-task deep learning framework to train the data set to obtain a model of the real-time multi-task fusion algorithm; The classification model and segmentation model in the real-time multi-task fusion algorithm are used to discriminate and classify the apple RGB image and obtain the apple outline, determining whether the apple is closer to a rectangular or trapezoidal outline or closer to an elliptical or circular outline. Step 3: Fill the apple contours classified as trapezoidal using the convex hull algorithm, and then fit the rectangular contours and the filled trapezoidal contours using the minimum circumscribed rectangle. Ellipse fitting is performed on the apple contours classified as elliptical or circular using the ellipse detection algorithm based on the arc adjacency matrix. Step 4: First, the two-dimensional centroid is obtained by using the fitted shape: rectangle or ellipse, and then the fruit stalk and calyx are detected on the apple RGB image using the key point detection model of the real-time multi-task fusion algorithm. If the position of the fruit stalk or calyx cannot be detected in the apple RGB image, the long symmetry axis of the fitted rectangle or the long axis of the fitted ellipse is the two-dimensional fruit axis direction; if the position of the fruit stalk or calyx can be detected in the apple RGB image, the line connecting the two-dimensional centroid and the fruit stalk or calyx is the two-dimensional fruit axis direction; Step 5: Using the obtained apple outline, calculate the average radius of the apple and obtain a depth map of the apple at the corresponding position in the depth image. The apple depth map is converted into a three-dimensional point cloud of the apple using the camera intrinsic parameter matrix. After denoising the three-dimensional point cloud of the apple using the three-dimensional point cloud feature analysis model, the three-dimensional centroid of the three-dimensional point cloud of the apple is calculated using the two-dimensional centroid and the average radius of the apple. Step 6: The three-dimensional point cloud feature analysis model is processed differently depending on whether key points exist; If the position of the fruit stalk or calyx is detected in the key point detection model, the depth information of the fruit stalk or calyx position corresponding to the depth image is mapped to the three-dimensional point cloud image of the apple to obtain the three-dimensional fruit stalk or calyx position, and the line connecting the three-dimensional centroid and the three-dimensional fruit stalk or calyx position is the three-dimensional fruit axis direction of the apple; If the position of the fruit stalk or calyx is not detected in the key point detection model, the curvature calculation based on eigenvalue analysis is performed on each point of the apple's three-dimensional point cloud. After eliminating noise interference, the difference between the center point coordinates of the area with the largest curvature value and the three-dimensional center of mass depth value can be used to calculate the z-axis deflection angle of the apple's three-dimensional fruit axis. The xy-axis angle of the three-dimensional fruit axis is provided by the two-dimensional fruit axis direction, thereby obtaining the three-dimensional fruit axis direction of the apple; the three-dimensional fruit axis direction represents the three-dimensional posture of the apple.

2. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1-1: Use the depth camera to obtain the original apple RGB image and original depth image, and configure the resolution and frame rate of the depth stream and color stream to be exactly the same; In steps 1 and 2, use a wide range of general visual detection modules, such as the publicly available YOLO object detection module, to locate the apple in the original apple RGB image and obtain the apple target frame. Then, crop the original apple RGB image and original depth image within the apple target frame to obtain the aligned apple RGB image and depth image. Perform image enhancement on the aligned apple RGB image to obtain the apple RGB image and depth image.

3. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2-1. Build a dataset: Collect a large number of apple color images and filter them based on clarity and image content. Label the filtered apple color images, annotating the fruit stalk and calyx positions, and oval-shaped apples, rectangular-shaped apples, and trapezoidal-shaped apples as annotation files. Divide the dataset into training, validation, and test sets according to a certain ratio. Step 2-2: Construct a multi-task deep learning framework to train the dataset. The multi-task deep learning framework is constructed as follows: CSPDarknet is used as the deep learning backbone network, and PAN-FPN is selected as the feature fusion network for the neck. The C3K2 module and C2PSA module are then introduced to improve the backbone network and feature fusion network. C3K2 replaces the C2F module in the backbone network and feature fusion network. The C3K2 module enhances feature extraction capabilities by combining variable convolution kernels and channel separation strategies. C2PSA replaces the SPPF module in the backbone network. The C2PSA module dynamically adjusts features to adjust the attention of different positions through a mechanism, enhancing the model's perception of target details. The input image is first feature extracted in the backbone network, and the extracted features are fused in the neck network. Finally, the fused features are input to the decoupling head, which consists of a target box detection head, a target classification head, a contour segmentation head, and a key point detection head. The four decoupling heads simultaneously process the features input from the neck network to generate the final detection results, and simultaneously obtain the key points of the apple position, similar shapes, contours, and stalks or calyxes. Step 2-3: Use the above-mentioned multi-task deep learning framework to train the real-time multi-task fusion algorithm; use the annotated training set, validation set, and test set to train the real-time multi-task fusion algorithm, and select the optimal training model as the final real-time multi-task fusion algorithm model; Steps 2-4: The apples in the apple RGB image are discriminated and classified using the trained classification model in the real-time multi-task fusion algorithm, and the specific apple outlines are extracted by the segmentation model in the real-time multi-task fusion algorithm.

4. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 1, characterized in that: The apple depth map in step 5 is converted into an apple 3D point cloud as follows: (x c ,y c , z c ) T The apple 3D point cloud corresponds to the camera coordinate system, representing the actual distance between the object and the camera optical center, in millimeters. The apple depth map corresponds to the image pixel coordinate system plus the depth information of each pixel. The depth information of the apple depth map is the same as the depth information in the apple 3D point cloud. The coordinate origin of the image pixel coordinate system is the upper left corner of the image plane, (u, v). T Represents the coordinates of the image pixel coordinate system; the formula for converting from the camera coordinate system to the image pixel coordinate system is: Among them, (x c ,y c , z c ) are the x-axis, y-axis, and z-axis coordinates in the camera coordinate system, (u, v) are the x-axis and y-axis coordinates in the image pixel coordinate system, and f x is the focal length of the camera in the x-axis direction, f y is the focal length of the camera in the y-axis direction, (c x , c y ) is the center point coordinate of the camera image sensor; The z-axis coordinate of the apple's three-dimensional point cloud can be deduced from the above formula c That is the depth value of the apple depth map, x c ,y c The coordinates are as follows: x c =(u-c x )*z c / f x ; y c =(v-c y )*z c / f y ; Among them, x c is the x-axis coordinate in the camera coordinate system, y c is the y-axis coordinate in the camera coordinate system, z c is the z-axis coordinate in the camera coordinate system, u is the x-axis coordinate in the image pixel coordinate system, v is the y-axis coordinate in the image pixel coordinate system, c x is the x-axis coordinate of the center point of the camera image sensor, c y is the y-axis coordinate of the center point of the camera image sensor.

5. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 1, characterized in that: The image enhancement process of the aligned apple RGB image in step 1 is specifically as follows: After alignment, the apple RGB image is converted to an HSV color space image, the three channels are separated, and the homomorphic filtering algorithm is applied to the brightness V channel. The brightness V channel image can be seen as consisting of two parts: f(x,y)=f j (x,y)f r (x,y); Among them, f(x, y) is the grayscale function of the brightness V channel image; f j (x, y) is the light intensity component that varies with spatial position and is concentrated in the low-frequency part of the brightness V channel image; f r (x, y) is the reflection component of the scene reflected to the human eye, which contains various information about the scene, mainly the high-frequency components of the brightness V channel image; The brightness V channel image is logarithmically transformed to obtain the following two additive components: ln f(x,y)=ln f j (x,y)+ln f r (x,y); Perform Fourier transform on the logarithmic image and obtain its corresponding frequency domain representation: DFT[ln f(x,y)]=DFT[ln f j (x,y)]+DFT[ln fr ( x,y)]; Assume that the frequency domain filter H f (u h , v h ), perform frequency domain filtering of the logarithmic image, filter H f It is a high-pass filter that suppresses low-frequency components: H f (u h ,v h )=(γ W -c L )H hx (u h ,v h )+c L ; Among them, γ L <1 is low frequency weight, γ W >1 is the high frequency weight, which controls the range of the filter amplitude; γ L The L in represents low frequency, indicating that the weight is low frequency weight, γ W The W in the formula represents high frequency, indicating that the weight is high frequency weight, and H hp (u h , v h ) is a high-pass filter function; H hp (u h , v h ) uses a Gaussian high-pass filter, which is: Among them, (u h , v h ) represents the offset of each point relative to the center point in the x-axis and y-axis directions, D is the Euclidean distance from each point to the center point, D0 is the cutoff frequency domain of the Gaussian filter, and c is a constant that controls the shape of the filter. The larger its value, the steeper the slope band; Merge the three channels H, S, and V and convert them back to the aligned apple RGB image.

6. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 1, characterized in that: The three-dimensional point cloud denoising performed in step 5 is specifically as follows: Generate a kd tree based on the 3D point cloud data and establish the topological relationship of the 3D point cloud; find the neighborhood of any point in the 3D point cloud; determine the distance between the point and each point in the neighborhood and take the average value; determine whether the average value exceeds the threshold. If it exceeds, the point is determined to be an outlier and removed. If it does not exceed, the next point is calculated; after all points have been judged as outliers, the DBSCAN clustering operation is performed.

7. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 6, characterized in that: The operation flow of the DBSCAN clustering operation is as follows: 1) Select a data point p from the dataset at random; 2) If the selected data object point p is a core point for the parameters neighborhood radius Eps and number threshold MinPts, then find all data object points that are density-reachable from p to form a cluster; 3) If the selected data object point p is an edge point, select another data object point; 4) Repeat steps 2) and 3) until all points are processed.

8. The method for recognizing three-dimensional posture of naturally grown apples based on a real-time multi-task fusion algorithm according to claim 1, characterized in that: The curvature calculation based on eigenvalue analysis in step 6 is as follows: Convert the 3D point cloud into an array and construct a KD tree; normalize the original data to eliminate the influence of dimension and numerical range: Among them, x i is the i-th feature in the original data, u i is the mean of the feature, σ i is the standard deviation of the feature, x′ i is the standardized data, i is the index, indicating the i-th element in the corresponding data; Traverse each point in the 3D point cloud and search for points in its neighborhood. Use the standardized data to calculate the covariance matrix C in the neighborhood of each point. The covariance matrix C is an n*n matrix, where n is the number of features, reflecting the correlation between the dimensions of the data: Where C is the covariance matrix, n is the number of features, x′ i is the i-th feature after standardization, is the average value of the standardized features, i is the index, indicating the i-th element in the corresponding data; Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​λ1, λ2, λ3, ..., λ n ; The curvature is determined by the minimum eigenvalue λ min The ratio to the sum of all eigenvalues ​​is defined as: Where Q is the curvature, λ min is the minimum eigenvalue, λ i is the i-th eigenvalue.

Images