Fruit stem pose recognition method used for pomelo fruit picking apparatus

Abstract

Disclosed in the present invention is a fruit stem pose recognition method used for a pomelo fruit picking apparatus, the method comprising the following steps: preprocessing image data of a scene to be recognized, so as to obtain preprocessed image data, the image data comprising an RGB map and a depth map; inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth map of a single pomelo fruit; on the basis of the depth map of a single pomelo fruit, recognizing an elliptical fruit pose feature region, and further using the Hough ellipse detection algorithm to obtain an ellipse tilt angle of the fruit pose feature region; and, by means of the ellipse tilt angle of the fruit pose feature region, calculating and obtaining the position and orientation of a pomelo fruit stem. The method integrates a plurality of advanced technologies to achieve optimization of a whole process from image data preprocessing to fruit stem pose accurate recognition, thus improving the recognition accuracy and working efficiency of automated pomelo fruit picking apparatuses, and reducing the mis-picking rate and the fruit damage rate.

Description

A fruit stem pose recognition method applied to a grapefruit picking device TECHNICAL FIELD

[0001] The present application relates to the field of grapefruit picking, more particularly to a fruit stem pose recognition method applied to a grapefruit picking device. BACKGROUND

[0002] In the process of modern agricultural automation, fruit picking robots as an important part of intelligent agricultural equipment have great significance for improving orchard operation efficiency and reducing labor costs. However, the automatic picking technology for specific fruits such as grapefruit still faces many challenges, especially in fruit recognition and accurate positioning. Existing grapefruit picking devices mostly rely on simple color recognition or fixed shape matching strategies, which are not effective in complex natural environments.

[0003] Currently, the orchard environment is natural and complex, with varying light intensity and direction over time and weather, as well as natural shading of leaves and fruits, which makes simple color or shape-based recognition algorithms susceptible to interference and unable to work stably under various lighting and shading conditions, resulting in a significant decrease in recognition accuracy. Existing solutions focus more on RGB image processing and fail to fully integrate depth information to enhance the spatial perception of fruits. The lack of depth data support makes it difficult for the system to accurately distinguish between foreground fruits and background branches and leaves, especially in dense fruit or complex depth change conditions, resulting in large fruit pose estimation errors.

[0004] Although deep learning models have achieved remarkable results in image segmentation, their direct application in complex orchard scenarios is limited by high model complexity and computational requirements, affecting real-time performance. The segmentation effect of small-scale or partially obscured fruits by the model is not good, which affects the subsequent fruit pose judgment.

[0005] At the same time, traditional fruit pose estimation methods usually rely on fixed shape assumptions or simplified geometric models, which cannot fully consider the diversity of fruit actual shape, especially the shape changes during fruit ripening. This leads to inaccurate pose recognition and affects the effective formulation of the grasping strategy. In actual orchard operations, dynamic backgrounds (such as wind-moving leaves) and static disturbances (such as other fruits and branches) can seriously interfere with the accuracy of image recognition. Existing technologies often rely on static background modeling or simple motion compensation when dealing with these disturbances, and have limited adaptability to complex dynamic scenes.

[0006] In addition, the accurate positioning of the fruit stem, which is a key point for picking, is crucial. However, existing positioning methods often rely on simplified geometric reasoning or template matching, ignoring the complexity of fruit pose and the diversity of fruit stem shape, resulting in a high failure rate of grasping.

[0007] Therefore, how to design a fruit stem pose recognition method applied to a grapefruit picking device, improve the recognition accuracy and working efficiency of the grapefruit picking automatic device, and reduce the mispicking rate and fruit damage rate is a problem that a person skilled in the art urgently needs to solve. SUMMARY

[0008] Therefore, the present application provides a fruit stem pose recognition method applied to a grapefruit picking device, which integrates various advanced technologies and realizes the whole-process optimization from image data preprocessing to accurate fruit stem pose recognition, so that the grapefruit picking device can better adapt to the complex environment of the orchard and provide strong support for the automatic development of intelligent agriculture.

[0009] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0010] A fruit stem pose recognition method applied to a grapefruit picking device, comprising the following steps:

[0011] S1, preprocessing image data of a scene to be recognized to obtain preprocessed image data; the image data includes an RGB image and a depth image;

[0012] S2, inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth image of a single grapefruit;

[0013] S3, based on the depth image of the single grapefruit, recognizing an elliptical fruit attitude feature region, and combining a Hough ellipse detection algorithm to obtain an elliptical tilt angle of the fruit attitude feature region;

[0014] S4, calculating the position and attitude of the grapefruit stem through the elliptical tilt angle of the fruit attitude feature region.

[0015] In the S1, the preprocessing includes:

[0016] S11, using a space-time joint embedding network to align the RGB image and the depth image in the image data, ensuring the consistency of any pixel in the spatial dimension and the temporal dimension;

[0017] S12, based on the aligned image data, filtering and smoothing the depth image in the image data through an adaptive bilateral morphological filter to obtain filtered and smoothed image data;

[0018] S13, combining the actual working radius and working height of the grapefruit picking device to set a depth threshold, filtering the pixels in the depth image in the filtered and smoothed image data to generate a mask image;

[0019] S14. The mask image and the RGB image in the filtered and smoothed image data are superimposed to filter out the color pixels outside the sampling range and obtain the superimposed image data.

[0020] S15. Perform grapefruit contour enhancement on the superimposed image data to obtain preprocessed image data.

[0021] Furthermore, S11 includes:

[0022] S111. Using the spatiotemporal attention module in the spatiotemporal joint embedding network, the optical flow information and depth differences between consecutive image frames are analyzed to perform dynamic region recognition.

[0023] S112. Combine the auxiliary layer in the spatiotemporal joint embedding network to capture time-series data of the dynamic region;

[0024] S113. Based on the results of dynamic region identification and time series analysis, implement an adaptive weight fusion strategy to adjust the fusion weights of RGB and depth maps in the image data and enhance the dependence on color image features.

[0025] S114. The gradient descent algorithm is used to perform multi-scale fusion of the RGB image and the depth image after adaptive weight fusion, so as to achieve accurate alignment from the whole to the local.

[0026] Furthermore, S12 includes:

[0027] S121. Based on the local features of the depth map in the image data, determine the kernel size of the adaptive bilateral morphological filter;

[0028] S122. Perform depth adaptive weight optimization, assigning higher weights to pixels that are close to each other;

[0029] S123. Perform morphological operations on the depth map in the depth-adaptive weighted image data; in the morphological operations, dynamically select and execute combinations of opening and closing operations and adjust parameters.

[0030] S124. Establish a feedback mechanism for the difference between images before and after filtering, analyze and process the results of morphological operations, and dynamically adjust the filtering parameters to prevent over-smoothing.

[0031] Furthermore, S15 includes:

[0032] S151. Employ a dynamic background suppression strategy to analyze optical flow information and inter-frame differences in the superimposed image data, update the background model, and remove background interference and dynamic change factors.

[0033] S152. Based on the image data after removing background interference and dynamic change factors, and combined with the occlusion attention mechanism, the occluded area and the non-occluded area are identified.

[0034] S153. After identifying the occluded and unoccluded areas, the image data is used to enhance the visualization of the grapefruit outline at different scales using a multi-scale feature pyramid network and an edge detection algorithm to obtain an initial outline-enhanced image.

[0035] S154. An adaptive contour thinning network is introduced to intelligently thin the initial contour enhancement image, identify and correct contour defects caused by occlusion or image noise, and obtain preprocessed image data.

[0036] Furthermore, in S2, the instance segmentation model adopts a U-Net architecture that combines depthwise separable convolution with a self-attention mechanism, including: an encoder, an attention module, a decoder, and an output layer;

[0037] The encoder uses a ResNet bottleneck module with depthwise separable convolutions and adds a multi-task learning branch for depth map prediction, edge map prediction, and stem position prediction; the decoder uses skip connections; and the attention module incorporates a dynamic context enhancement strategy to dynamically adjust channel and spatial attention weights based on image content.

[0038] Furthermore, in S2, the instance segmentation model optimization includes: before training the instance segmentation model, a lightweight depthwise enhancement strategy is adopted to perform lightweight transformation on the depth map in the image data;

[0039] During the training of the instance segmentation model, an adaptive loss function is used; the adaptive loss function includes: class imbalance loss, boundary clarity loss, and depth consistency loss.

[0040] Furthermore, in step S3, identifying the elliptical fruit posture feature region includes:

[0041] S311. Analyze the depth value range of a single pomelo's depth map using a sorting algorithm, set a depth threshold for the feature region, and remove pixels from non-target regions.

[0042] S312. Combine morphological optimization with the depth map of a single pomelo fruit after removing the pixels of the target region to refine the boundary of the pomelo fruit region;

[0043] S313. Based on shape and texture features, the depth map of a single pomelo fruit after the boundary of the pomelo fruit region is refined is filtered to obtain an elliptical fruit posture feature region.

[0044] Furthermore, in step S3, obtaining the elliptical tilt angle of the fruit posture feature region includes:

[0045] S321. Based on the fruit posture feature region, perform edge detection and extract the contour line of the fruit posture feature region;

[0046] S322. Smooth the outline to obtain the smoothed outline of the fruit posture feature area.

[0047] S323. The Hough transform ellipse detection algorithm is used to analyze and process the smoothed contour lines of the fruit posture feature region and identify the ellipse parameters of the fruit posture feature region contour.

[0048] S324. Based on the ellipse parameters of the fruit posture feature region contour, extract and verify the ellipse tilt angle until the preset conditions are met, and output the ellipse tilt angle.

[0049] Furthermore, S4 includes:

[0050] S41. Based on the ellipse tilt angle, rotate the depth map of a single pomelo fruit to obtain a depth map of a single pomelo fruit with the stem pointing vertically upwards.

[0051] S42. Draw the outer rectangle of the depth map of a single pomelo with the stem pointing vertically upwards, and determine the pixel length and width of the outer rectangle;

[0052] S43. Based on the two-dimensional coordinates (u, v) of the center point of the circumscribed rectangle in the image, and the depth value d of the center point, perform a transformation from the pixel coordinate system to the camera coordinate system to obtain the three-dimensional coordinates (X, v) of the center point in the camera coordinate system. c ,Y c Z c The conversion formula is: Z c =d

[0053] Among them, f x and f y c is the camera focal length. x and c y The pixel coordinates of the camera's optical center;

[0054] S44. Take any two points on either side of the center point of the circumscribed rectangle, and use the transformation formula to calculate the three-dimensional coordinates of the two points, construct the proportional relationship between the pixel length and the actual length, and determine the length and width of the circumscribed rectangle; the length and width of the circumscribed rectangle are the longitudinal diameter H and the transverse diameter W of the grapefruit.

[0055] S45. Based on the three-dimensional coordinates (X) of the center point in the camera coordinate system c ,Y c Z c ) and the longitudinal diameter H and transverse diameter W of the pomelo, to obtain the coordinates of the cutting point (X). q ,Yq Z q The calculation formula is:

[0056] T is the translation matrix:

[0057] Where θ is the angle of fruit tilt.

[0058] As can be seen from the above technical solution, compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0059] 1. Image preprocessing steps, including standardization, noise reduction, and resizing, optimized the quality of the original image data, ensuring the accuracy of subsequent processing. Simultaneously, the use of RGB and depth maps provided rich visual and spatial information, which is crucial for fruit identification in complex orchard environments.

[0060] 2. By adopting the optimized instance segmentation model, especially the U-Net architecture that integrates depthwise separable convolution and self-attention mechanism, it can efficiently and accurately segment individual grapefruits, significantly improving processing speed and segmentation accuracy.

[0061] 3. By analyzing the depth map of a single pomelo and combining it with the Hough ellipse detection algorithm, we can not only accurately identify the elliptical posture feature region of the fruit, but also directly extract the tilt angle of the fruit, ensuring robustness under different lighting conditions and fruit arrangement complexity.

[0062] 4. By utilizing the elliptical tilt angle of the fruit's posture feature area, the position and posture of the fruit stalk can be accurately calculated. This effectively addresses the challenge of fruit identification under shading conditions, improves the success rate of harvesting, reduces damage to the fruit, and ensures harvesting efficiency and fruit quality. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0064] Figure 1 is a flowchart of the fruit stem pose recognition method for pomelo harvesting equipment provided by an embodiment of the present invention.

[0065] Figure 2 is a preprocessing flowchart provided in an embodiment of the present invention;

[0066] Figure 3 is a flowchart of the process for identifying the elliptical fruit posture feature region according to an embodiment of the present invention;

[0067] Figure 4 is a flowchart of obtaining the elliptical tilt angle of the fruit posture feature region according to an embodiment of the present invention;

[0068] Figure 5 is a schematic diagram of the outline of the fruit posture feature region extracted according to an embodiment of the present invention;

[0069] Figure 6 is a flowchart of the calculation of the position and posture of the pomelo fruit stalk provided in an embodiment of the present invention;

[0070] Figure 7 is a schematic diagram of the fruit shape characteristics of pomelo provided in an embodiment of the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] As shown in Figure 1, this embodiment provides a method for recognizing the posture of the fruit stalk in a pomelo harvesting device, including the following steps:

[0073] S1. Preprocess the image data of the scene to be identified to obtain preprocessed image data; the image data includes: RGB image and depth image;

[0074] S2. Input the preprocessed image data into the optimized instance segmentation model to obtain the depth map of a single grapefruit;

[0075] S3. Based on the depth map of the single pomelo, identify the elliptical fruit posture feature region, and combine the Hough ellipse detection algorithm to obtain the ellipse tilt angle of the fruit posture feature region.

[0076] S4. Calculate the position and posture of the pomelo stem by using the elliptical tilt angle of the fruit posture feature region.

[0077] This method integrates multiple advanced technologies to optimize the entire process from image data preprocessing to accurate identification of fruit stem pose, thereby improving the recognition accuracy and working efficiency of automated pomelo harvesting equipment and reducing the rate of misharvesting and fruit damage.

[0078] The following provides a further detailed explanation of each of the above steps:

[0079] As shown in Figure 2, in this embodiment S1, the preprocessing includes:

[0080] S11. Align the RGB and depth maps in the image data using a spatiotemporal joint embedding network to ensure the consistency of any pixel in both spatial and temporal dimensions; specifically including:

[0081] S111. Using the spatiotemporal attention module in the spatiotemporal joint embedding network, analyze the optical flow information and depth differences between consecutive image frames to perform dynamic region identification; S112. Combining the auxiliary layer in the spatiotemporal joint embedding network, capture time-series data of the dynamic region; S113. Based on the results of dynamic region identification and time-series analysis, implement an adaptive weight fusion strategy to adjust the fusion weights of the RGB image and depth image in the image data, enhancing the dependence on color image features; S114. Use the gradient descent algorithm to perform multi-scale fusion of the RGB image and depth image after adaptive weight fusion to achieve accurate alignment from the whole to the local.

[0082] It achieves spatiotemporal consistency optimization of image data through spatiotemporal joint embedding network. By using dynamic information analysis, adaptive fusion strategy and multi-scale fusion technology, it ensures high synchronization and alignment between RGB image and depth image in the early stage of processing, providing accurate input data for subsequent processing steps and improving the robustness and accuracy of the entire fruit stem pose recognition system.

[0083] S12. Based on the aligned image data, an adaptive bilateral morphological filter is used to filter and smooth the depth map in the image data to obtain filtered and smoothed image data; specifically including:

[0084] S121. Determine the kernel size of the adaptive bilateral morphological filter based on the local features of the depth map in the image data; S122. Perform depth-adaptive weight optimization, assigning higher weights to pixels that are close to each other; S123. Perform morphological operations on the depth map in the image data after depth-adaptive weight optimization; in the morphological operations, dynamically select and execute combinations of opening and closing operations and adjust parameters; S124. Establish a feedback mechanism for the difference between the images before and after filtering, analyze and process the results of the morphological operations, and dynamically adjust the filtering parameters to prevent over-smoothing.

[0085] This step first analyzes the local features of the depth map and intelligently determines the filter kernel size to adapt to the feature requirements of different regions. Next, a depth-adaptive weight optimization strategy is implemented, dynamically adjusting weights based on the spatial relationships between pixels to enhance the influence of neighboring pixels, thereby smoothing the image while preserving edge details. Based on this, morphological operations are performed, dynamically selecting and fine-tuning the combination and parameters of opening and closing operations to adapt to changes in image content, effectively removing noise while maintaining structural integrity. Finally, a feedback mechanism is established to monitor the processing effect by establishing the difference between the images before and after filtering, adjusting the filtering parameters in a timely manner to avoid over-smoothing and ensure that the processed result is both clean and retains necessary detail information.

[0086] S13. Based on the actual working radius and working height of the pomelo harvesting equipment, a depth threshold is set, and pixel filtering is performed on the depth map in the filtered and smoothed image data to generate a mask image.

[0087] In the specific process of determining rather than dynamically adjusting the depth threshold, an initial depth threshold range is first set based on the physical specifications of the robotic arm and prior knowledge of the orchard's operating environment. This range needs to consider the maximum operating radius of the robotic arm, the expected size of the fruit (such as a pomelo), and its typical hanging height in the tree canopy. Using statistical analysis of historical harvesting data, the average fruit diameter and its distribution range are determined. Combined with the growth habits of the orchard trees and the density of branches and leaves, a fixed depth threshold is estimated that can cover most of the effective fruit area while excluding background clutter. This threshold is used to filter pixels in the depth map, retaining depth information that falls within the preset range and may belong to the target fruit, thereby assisting in subsequent fruit identification and localization tasks.

[0088] S14. Overlay the RGB image in the mask image and the filtered and smoothed image data, filtering out color pixels outside the sampling range to obtain the overlaid image data; specifically including:

[0089] Based on the previously set depth threshold, a binary mask image is generated. In this mask image, pixels within the robotic arm's working range are marked as white (or set to 1), indicating that these areas are of interest or valid; while pixels outside the working range are marked as black (or set to 0), indicating that they should be ignored or filtered out.

[0090] The generated binary mask image is then compared pixel-by-pixel with the original color image. Specifically, a pixel-level logical AND operation can be used, which performs a bitwise logical AND operation between each pixel value in the mask image and the corresponding pixel value in the color image. Only the color image pixels corresponding to pixels with a value of 1 (i.e., valid) in the mask image are retained, while pixels with a mask value of 0 are set to the background color or transparent (in some software or algorithms, this is implemented as a zero value).

[0091] After completing the above logic and operations, the "filtering" step has been completed. The pixel information corresponding to all pixels marked as 0 in the mask image (i.e., those outside the picking area) has been hidden or changed to the background color in the color image. This process can be considered as directly removing unnecessary information at the image processing level. Through the above overlay and filtering process, not only is the analysis area effectively defined and the computational load reduced, but the accuracy of detecting grapefruit and its stem in subsequent processing steps is also improved, especially under conditions of complex backgrounds and occlusion.

[0092] S15. Perform grapefruit contour enhancement on the superimposed image data to obtain preprocessed image data. This includes:

[0093] S151. Employing a dynamic background suppression strategy, the optical flow information and inter-frame differences in the superimposed image data are analyzed to update the background model and remove background interference and dynamic change factors. S152. Based on the image data after removing background interference and dynamic change factors, an occlusion attention mechanism is used to identify occluded and unoccluded regions. S153. For the image data after identifying occluded and unoccluded regions, a multi-scale feature pyramid network is used in conjunction with an edge detection algorithm to enhance the visualization of the grapefruit outline at different scales, obtaining an initial outline-enhanced image. S154. An adaptive outline thinning network is introduced to intelligently thin the initial outline-enhanced image, identifying and correcting outline defects caused by occlusion or image noise, obtaining preprocessed image data.

[0094] Through the orderly execution of this series of steps, not only was background interference effectively suppressed and occluded areas accurately identified and processed, but the visualization effect of the grapefruit outline was also significantly enhanced at multiple scales. Finally, highly accurate pre-processed image data was obtained through an intelligent thinning network, providing a solid foundation for the pose detection and localization of grapefruit stems under occlusion.

[0095] In this embodiment S2, the instance segmentation model adopts the U-Net architecture that combines depthwise separable convolution with self-attention mechanism, including: encoder, attention module, decoder and output layer;

[0096] The encoder uses a ResNet bottleneck module with depthwise separable convolutions and adds a multi-task learning branch for depth map prediction, edge map prediction, and stem position prediction; the decoder uses skip connections; and the attention module incorporates a dynamic context enhancement strategy to dynamically adjust channel and spatial attention weights based on image content.

[0097] The instance segmentation model optimization includes: before training the instance segmentation model, a lightweight depthwise enhancement strategy is adopted to perform lightweight transformations on the depth map in the image data;

[0098] During the training of the instance segmentation model, an adaptive loss function is used; the adaptive loss function includes: class imbalance loss, boundary clarity loss, and depth consistency loss.

[0099] The specific process of obtaining a depth map of a single grapefruit includes:

[0100] In the feature extraction and processing stage, rich features are extracted using the bottleneck module of ResNet with depthwise separable convolution, key information is enhanced by the dynamic context-enhanced attention module, and the segmentation accuracy is improved in the decoding stage by combining skip connection technology.

[0101] Lightweight depthwise augmentation and adaptive loss function optimization strategies are implemented during model training to enhance the model's adaptability to depth variations and the accuracy of segmentation boundaries.

[0102] The post-processing steps include generating a binarized mask to distinguish the target fruit from the background, accurately extracting the depth region of a single fruit, and eliminating edge noise through an erosion algorithm to finally obtain a high-quality depth map of a single fruit.

[0103] When observing a grapefruit in a depth map, pixels of the same depth value appear in concentric rings. Therefore, near the equator of the grapefruit, from the shoulder to the base, there exists a region surrounded by a ring of pixels of the same depth value, which appears as an ellipse in the depth image.

[0104] As shown in Figure 3, in this embodiment S3, identifying the elliptical fruit posture feature region includes:

[0105] S311. Analyze the depth value range of a single pomelo depth map using a sorting algorithm, set a depth threshold for the feature region, and remove pixels from non-target regions. S312. Combine morphological optimization with the single pomelo depth map after removing target region pixels to refine the pomelo region boundary. S313. Based on shape and texture features, filter the single pomelo depth map after refining the pomelo region boundary to obtain an elliptical fruit posture feature region.

[0106] As shown in Figure 4, the elliptical tilt angle of the fruit posture feature region is obtained, including:

[0107] S321. Based on the fruit posture feature region, perform edge detection and extract the outline of the fruit posture feature region; as shown in Figure 5, the blue area in the upper layer image is the grapefruit with the stem obscured, and the white ellipsoid in the lower layer image is the fruit posture feature region.

[0108] S322. Smooth the outline to obtain the smoothed outline of the fruit posture feature area.

[0109] S323. The Hough transform ellipse detection algorithm is used to analyze and process the smoothed contour lines of the fruit posture feature region and identify the ellipse parameters of the fruit posture feature region contour.

[0110] S324. Based on the ellipse parameters of the fruit posture feature region contour, extract and verify the ellipse tilt angle until the preset conditions are met, and output the ellipse tilt angle.

[0111] This step enables precise identification of the posture feature regions of individual grapefruits and accurate extraction of the elliptical tilt angle, significantly improving the accuracy and reliability of fruit detection. First, through intelligent analysis and processing of the depth map, non-fruit pixels are effectively removed and fruit boundaries are optimized. Combined with shape and texture information filtering, elliptical fruit regions are precisely defined. Within these precise fruit feature regions, meticulous edge detection and smoothing ensure clear contours. Then, the Hough transform ellipse detection algorithm is used to precisely locate the contour parameters of the fruit posture feature regions, and the fruit's tilt angle is calculated accordingly. This entire process not only reduces background interference but also improves the accuracy of fruit posture judgment through multi-step optimization, providing a solid technical foundation for applications such as automated harvesting and fruit classification.

[0112] As shown in Figures 6 and 7, in this embodiment S4, the following are included:

[0113] S41. Based on the ellipse tilt angle, rotate the depth map of a single pomelo fruit to obtain a depth map of a single pomelo fruit with the stem pointing vertically upwards.

[0114] S42. Draw the outer rectangle of the depth map of a single pomelo with the stem pointing vertically upwards, and determine the pixel length and width of the outer rectangle;

[0115] S43. Based on the two-dimensional coordinates (u, v) of the center point of the circumscribed rectangle in the image, and the depth value d of the center point, perform a transformation from the pixel coordinate system to the camera coordinate system to obtain the three-dimensional coordinates (X, v) of the center point in the camera coordinate system. c ,Y c Z c The conversion formula is: Z c =d

[0116] Among them, f x and f y c is the camera focal length. x and c y The pixel coordinates of the camera's optical center;

[0117] S44. Take any two points on either side of the center point of the circumscribed rectangle, and use the transformation formula to calculate the three-dimensional coordinates of the two points, construct the proportional relationship between the pixel length and the actual length, and determine the length and width of the circumscribed rectangle; the length and width of the circumscribed rectangle are the longitudinal diameter H and the transverse diameter W of the grapefruit.

[0118] S45. Based on the three-dimensional coordinates (X) of the center point in the camera coordinate system c ,Y c Z c ) and the longitudinal diameter H and transverse diameter W of the pomelo, to obtain the coordinates of the cutting point (X). q ,Y q Z q The calculation formula is:

[0119] T is the translation matrix:

[0120] Where θ is the angle of fruit tilt.

[0121] In this step, the image viewing angle is adjusted according to the elliptical tilt angle of the fruit to ensure the stem is vertical. Then, a circumscribed rectangle is constructed and mapped to the camera coordinate system, accurately obtaining the three-dimensional coordinates of the fruit's center and key points of the circumscribed rectangle. Using these coordinate points and camera parameters, not only are the actual physical dimensions of the fruit (vertical and horizontal diameters) calculated, but the coordinates of the cutting point are also determined through advanced geometric calculations. This strategy effectively overcomes measurement errors caused by changes in fruit posture, achieving precise mapping from two-dimensional images to three-dimensional space, and significantly enhancing the accuracy of grapefruit size and position determination.

[0122] In summary, the fruit stem pose recognition method for pomelo harvesting equipment in this embodiment achieves efficient recognition of fruit stem pose through a series of sophisticated image processing and machine learning techniques. From the precise alignment of RGB and depth maps using a spatiotemporal joint embedding network in the preprocessing stage, to adaptive filtering and depth map optimization, image quality and the accuracy of subsequent processing are effectively improved. The instance segmentation model integrates depthwise separable convolution and self-attention mechanisms, combined with lightweight depthwise enhancement and adaptive loss function optimization, significantly enhancing the model's robustness and segmentation accuracy in complex orchard environments. Through refined elliptical pose feature recognition and Hough transform, the pose information of the fruit is accurately obtained, and the transformation from two-dimensional image to three-dimensional space is completed accordingly, accurately locating the position and pose of the fruit stem. Ultimately, this method not only provides precise coordinates of the fruit stem for pomelo harvesting robots, but also provides strong technical support for achieving automated and intelligent fruit harvesting, greatly improving harvesting efficiency and fruit selection accuracy, reducing the need for manual intervention, and providing key technical support for the intelligent upgrading of modern agriculture.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0124] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fruit stalk pose recognition method applied to a grapefruit picking device, characterized in that, The method comprises the following steps: S1, preprocessing image data of a to-be-identified scene to obtain preprocessed image data; The image data comprises an RGB image and a depth image; S2, inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth image of a single pomelo fruit; S3, identifying an elliptical fruit posture feature region based on the depth image of the single pomelo fruit, and combining a Hough ellipse detection algorithm to obtain an elliptical tilt angle of the fruit posture feature region; S4, calculating the position and posture of the pomelo fruit stem based on the elliptical tilt angle of the fruit posture feature region.

2. The fruit stem pose recognition method applied to the grapefruit picking device according to claim 1, characterized in that, In S1, the preprocessing comprises: S11, aligning the RGB image and the depth image in the image data by using a space-time joint embedding network to ensure the consistency of any pixel in the spatial dimension and the temporal dimension; S12, filtering and smoothing the depth image in the image data by using an adaptive bilateral morphological filter based on the aligned image data to obtain filtered and smoothed image data; S13, setting a depth threshold based on the actual working radius and working height of a pomelo picking device, filtering the pixels in the depth image in the filtered and smoothed image data, and generating a mask image; S14, superimposing the mask image and the RGB image in the filtered and smoothed image data to filter out color pixels outside the picking range, and obtaining superimposed image data; S15, performing pomelo contour enhancement on the superimposed image data to obtain preprocessed image data.

3. The fruit stem pose recognition method applied to the grapefruit picking device according to claim 2, characterized in that, The S11 comprises: S111, analyzing the optical flow information and depth difference between consecutive image frames by using a space-time attention module in the space-time joint embedding network to perform dynamic region identification; S112, capturing time series data of the dynamic region by using an auxiliary layer in the space-time joint embedding network; S113, adjusting the fusion weight of the RGB image and the depth image in the image data based on the results of dynamic region identification and time series analysis to enhance the dependence on color image features by implementing an adaptive weight fusion strategy; S114, performing multi-scale fusion on the RGB image and the depth image after adaptive weight fusion by using a gradient descent algorithm to realize accurate alignment from the whole to the local.

4. The fruit stem pose recognition method for grape picking equipment according to claim 2, characterized in that, The S12 comprises: S121, determining the filter kernel size of the adaptive bilateral morphological filter based on the local features of the depth image in the image data; S122, performing depth adaptive weight optimization to give higher weight to pixels with similar distances; S123, performing morphological operations on the depth image in the image data after depth adaptive weight optimization; in the morphological operations, the combination and parameter adjustment of open-close operations are dynamically selected and executed; S124, establishing a feedback mechanism for the difference between the images before and after filtering, analyzing and processing the execution results of the morphological operations, and dynamically adjusting the filtering parameters to prevent over-smoothing.

5. The fruit stem pose recognition method for grape picking equipment according to claim 2, characterized in that, The S15 comprises: S151, analyzing the optical flow information and inter-frame difference in the superimposed image data by using a dynamic background suppression strategy, updating the background model, and removing background interference and dynamic change factors; S152, based on the image data after removing background interference and dynamic factors, combined with the shielding attention mechanism, the shielding area and the non-shielding area are identified; S153, the image data after identifying the shielding area and the non-shielding area is processed by using a multi-scale feature pyramid network combined with an edge detection algorithm to strengthen the outline of the pomelo fruit at different scales, and an initial outline enhanced image is obtained; S154, an adaptive contour thinning network is introduced to intelligently thin the initial contour enhanced image, identify and correct the contour defects caused by shielding or image noise, and obtain preprocessed image data.

6. The fruit stem pose recognition method for grape picking equipment according to claim 1, characterized in that, In S2, the instance segmentation model adopts a U-Net architecture combined with depth separable convolution and self-attention mechanism, including an encoder, an attention module, a decoder and an output layer. The encoder uses a ResNet bottleneck module of depth separable convolution, and a multi-task learning branch is additionally provided for depth map prediction, edge map prediction and fruit stem position prediction; the decoder adopts a skip connection; the attention module is combined with a dynamic context enhancement strategy to dynamically adjust the channel and spatial attention weights according to the image content.

7. The fruit stem pose recognition method for grape picking equipment according to claim 1, characterized in that, In S2, the instance segmentation model optimization includes: before training the instance segmentation model, a lightweight depthwise enhancement strategy is used to implement lightweight transformation on the depth map in the image data; During the training process of the instance segmentation model, an adaptive loss function is used; the adaptive loss function includes: class imbalance loss, boundary clarity loss and depth consistency loss. 8.The fruit stalk pose recognition method applied to the grapefruit picking device according to claim 1, characterized in that, In S3, the fruit posture feature region of the ellipse is identified, including: S311, the depth value interval of a single pomelo depth map is analyzed by a sorting algorithm, the depth threshold of the feature region is set, and the non-target region pixels are removed; S312, the single pomelo depth map after removing the target region pixels is combined with morphological optimization to refine the pomelo region boundary; S313, based on shape and texture features, the single pomelo depth map after refining the pomelo region boundary is screened to obtain the fruit posture feature region of the ellipse. 9.The fruit stalk pose recognition method applied to the grapefruit picking device according to claim 1, characterized in that, In S3, the elliptical inclination angle of the fruit posture feature region is obtained, including: S321, based on the fruit posture feature region, the edge is detected to extract the contour line of the fruit posture feature region; S322, the contour line is smoothed to obtain the smoothed contour line of the fruit posture feature region; S323, the smoothed contour line of the fruit posture feature region is analyzed and processed by using a Hough transform ellipse detection algorithm to identify the ellipse parameters of the fruit posture feature region contour; S324, based on the ellipse parameters of the fruit posture feature region contour, the elliptical inclination angle is extracted and verified until the preset condition is met, and the elliptical inclination angle is output.

10. The fruit stem pose recognition method for grape picking equipment according to claim 1, characterized in that, S4 includes: S41, based on the elliptical inclination angle, the single pomelo depth map is rotated to obtain a single pomelo depth map with the fruit stem vertically upward; S42, a circumscribed rectangle of the single pomelo depth map with the fruit stem vertically upward is drawn, and the pixel length and width of the circumscribed rectangle are determined; S43, based on the two-dimensional coordinates (u, v) of the center point of the circumscribed positive rectangle in the image, and the depth value d of the center point, pixel coordinate system to camera coordinate system conversion is performed to obtain the three-dimensional coordinates (X c ,Y c ,Z c ) of the center point in the camera coordinate system; the conversion formula is: Z c = d where f x and f y are the camera focal lengths, c x and c y are the pixel coordinates of the camera optical centers; S44, two points are taken on both sides of the center point of the circumscribed rectangle, and the three-dimensional coordinates of the two points are calculated by using the conversion formula to construct the proportional relationship between the pixel length and the actual length, and the length and width of the circumscribed rectangle are determined; the length and width of the circumscribed rectangle are the vertical diameter H and the horizontal diameter W of the pomelo fruit. S45, based on the three-dimensional coordinates (X c ,Y c ,Z c ) of the center point in the camera coordinate system and the longitudinal diameter H and the transverse diameter W of the pomelo fruit, the coordinates (X q ,Y q ,Z q ) of the cutting point are obtained; the calculation formula is: T is a translation matrix: Wherein, θ is the fruit inclination angle.

Images