A fruit stem posture recognition method for pomelo picking equipment

Abstract

The present invention discloses a method for recognizing the position and posture of a fruit stalk applied to a pomelo fruit picking device, comprising the following steps: preprocessing the image data of a scene to be recognized to obtain preprocessed image data; the image data includes: an RGB image and a depth image; inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth image of a single pomelo fruit; based on the depth image of a single pomelo fruit, recognizing an elliptical fruit posture feature region, and combining the Hough ellipse detection algorithm to obtain the elliptical inclination angle of the fruit posture feature region; calculating the position and posture of the fruit stalk of the pomelo fruit through the elliptical inclination angle of the fruit posture feature region. The method realizes the optimization of the entire process from image data preprocessing to accurate recognition of the position and posture of the fruit stalk by integrating a variety of advanced technologies, thereby improving the recognition accuracy and work efficiency of the pomelo fruit picking automation equipment, and reducing the mispick rate and fruit damage rate.

Description

Technical Field

[0001] The present invention relates to the field of pomelo fruit picking, and more particularly to a fruit stem posture recognition method applied to pomelo fruit picking equipment. Background Art

[0002] In the process of modern agricultural automation, fruit picking robots, as an important part of intelligent agricultural machinery and equipment, are of great significance for improving orchard operation efficiency and reducing labor costs. However, the automated picking technology for specific fruits such as pomelo still faces many challenges, especially in fruit recognition and precise positioning. Most existing pomelo picking devices rely on simple color recognition or fixed shape matching strategies, which are incapable of meeting the needs in complex and changing natural environments.

[0003] At present, the orchard environment is natural and complex, and the light intensity and direction change with time and weather. In addition, the natural occlusion of leaves and fruits makes the recognition algorithm based on simple colors or shapes easily interfered and unable to work stably under various lighting and occlusion conditions, and the recognition accuracy rate drops significantly. Existing solutions focus more on RGB image processing and fail to fully integrate depth information to enhance the spatial perception ability of fruits. The lack of support for depth data makes it difficult for the system to accurately distinguish between foreground fruits and background branches and leaves, especially when the fruits are dense or the depth changes are complex, resulting in large errors in fruit posture estimation.

[0004] Although deep learning models have achieved remarkable results in the field of image segmentation, when directly applied to complex scenes in orchards, the model complexity and computing requirements are too high, affecting real-time performance. The model does not perform well on small-scale or partially occluded fruits, which affects the subsequent judgment of fruit posture.

[0005] At the same time, traditional fruit posture estimation methods usually rely on fixed shape assumptions or simplified geometric models, which cannot fully consider the diversity of the actual morphology of fruits, especially the changes in shape during the ripening process of fruits. This leads to inaccurate posture recognition and affects the effective formulation of grasping strategies. In actual orchard operations, dynamic backgrounds (such as leaves moving in the wind) and static interferences (such as other fruits and branches) can seriously interfere with the accuracy of image recognition. When dealing with these interferences, existing technologies often rely on static background modeling or simple motion compensation, and have limited adaptability to complex dynamic scenes.

[0006] In addition, the fruit stalk is the key point for picking, and its accurate positioning is crucial. However, existing positioning methods often rely on simplified geometric inference or template matching, ignoring the complexity of fruit posture and the diversity of fruit stalk morphology, resulting in a high grasping failure rate.

[0007] Therefore, how to design a fruit stalk posture recognition method for pomelo fruit picking equipment, improve the recognition accuracy and work efficiency of pomelo fruit picking automation equipment, and reduce the mispicking rate and fruit damage rate is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0008] In view of this, the present invention provides a fruit stalk posture recognition method applied to pomelo fruit picking equipment, which integrates multiple advanced technologies and realizes the full process optimization from image data preprocessing to accurate recognition of fruit stalk posture, so that the pomelo fruit picking equipment can better adapt to the complex environment of the orchard and provide strong support for the automated development of smart agriculture.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A fruit stem posture recognition method applied to a pomelo fruit picking device comprises the following steps:

[0011] S1. Preprocessing the image data of the scene to be identified to obtain preprocessed image data; the image data includes: an RGB image and a depth image;

[0012] S2, inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth map of a single pomelo fruit;

[0013] S3, based on the depth map of the single pomelo fruit, identifying the elliptical fruit posture feature area, and combining the Hough ellipse detection algorithm to obtain the ellipse inclination angle of the fruit posture feature area;

[0014] S4. Calculate the position and posture of the pomelo fruit stem through the ellipse inclination angle of the fruit posture characteristic area.

[0015] Wherein, in said S1, the preprocessing includes:

[0016] S11. Use the spatiotemporal joint embedding network to align the RGB image and depth image in the image data to ensure the consistency of any pixel in the spatial dimension and the temporal dimension;

[0017] S12, based on the aligned image data, filtering and smoothing the depth map in the image data by using an adaptive bilateral morphological filter to obtain filtered and smoothed image data;

[0018] S13, combining the actual working radius and working height of the pomelo fruit picking equipment, setting a depth threshold, filtering pixels of the depth map in the filtered and smoothed image data, and generating a mask map;

[0019] S14, superimposing the mask image and the RGB image in the image data after filtering and smoothing, filtering out color pixels outside the picking range, and obtaining the image data after superimposing;

[0020] S15, performing pomelo fruit contour enhancement on the image data after the superposition processing to obtain pre-processed image data.

[0021] Furthermore, the S11 includes:

[0022] S111, using the spatiotemporal attention module in the spatiotemporal joint embedding network to analyze the optical flow information and depth difference between consecutive image frames to perform dynamic area recognition;

[0023] S112, combining the auxiliary layer in the spatiotemporal joint embedding network to capture time series data of the dynamic region;

[0024] S113. Based on the results of dynamic region recognition and time series analysis, an adaptive weight fusion strategy is implemented to adjust the fusion weights of the RGB image and the depth image in the image data to enhance the reliance on color image features.

[0025] S114. Use a gradient descent algorithm to perform multi-scale fusion of the RGB image and the depth image after adaptive weight fusion to achieve accurate alignment from the whole to the part.

[0026] Furthermore, the S12 includes:

[0027] S121, determining a filter kernel size of an adaptive bilateral morphological filter based on local features of the depth map in the image data;

[0028] S122, performing deep adaptive weight optimization to assign higher weights to pixels with similar distances;

[0029] S123, performing morphological operation on the depth map in the image data after depth adaptive weight optimization; in the morphological operation, dynamically selecting and executing a combination of opening and closing operations and parameter adjustment;

[0030] S124. Establish a feedback mechanism for the image difference before and after filtering, analyze and process the execution results of the morphological operation, and dynamically adjust the filtering parameters to prevent over-smoothing.

[0031] Furthermore, the S15 includes:

[0032] S151, adopting a dynamic background suppression strategy, analyzing the optical flow information and inter-frame differences in the image data after superposition processing, updating the background model, and removing background interference and dynamic change factors;

[0033] S152, based on the image data after removing background interference and dynamic change factors, combined with the occlusion attention mechanism, identify the occluded area and the non-occluded area;

[0034] S153, for the image data after the occluded area and the non-occluded area are identified, a multi-scale feature pyramid network is used in combination with an edge detection algorithm to perform visualization enhancement of the pomelo fruit contour at different scales to obtain an initial contour enhanced image;

[0035] S154, introducing an adaptive contour refinement network, performing intelligent refinement processing on the initial contour enhanced image, identifying and correcting contour defects caused by occlusion or image noise, and obtaining preprocessed image data.

[0036] Furthermore, in S2, the instance segmentation model adopts a U-Net architecture combining deep separable convolution with a self-attention mechanism, including: an encoder, an attention module, a decoder, and an output layer;

[0037] Among them, the encoder uses the ResNet bottleneck module with deep separable convolution and adds a multi-task learning branch for depth map prediction, edge map prediction and fruit stem position prediction; the decoder adopts skip connection; the attention module is combined with a dynamic context enhancement strategy to dynamically adjust the channel and spatial attention weights according to the image content.

[0038] Furthermore, in S2, the instance segmentation model optimization includes: before training the instance segmentation model, a lightweight depth wise enhancement strategy is used to perform a lightweight transformation on the depth map in the image data;

[0039] During the instance segmentation model training process, an adaptive loss function is used; the adaptive loss function includes: category imbalance loss, boundary clarity loss and depth consistency loss.

[0040] Furthermore, in S3, identifying the oval fruit posture feature area includes:

[0041] S311, analyzing the depth value interval of a single pomelo fruit depth map through a sorting algorithm, setting a depth threshold of a feature area, and removing pixels in non-target areas;

[0042] S312, combining morphological optimization with the single pomelo fruit depth map from which pixels in the target area are removed, to refine the pomelo fruit area boundary;

[0043] S313. Based on shape and texture features, the depth map of a single pomelo fruit after the pomelo fruit region boundary is refined is screened to obtain an elliptical fruit posture feature region.

[0044] Furthermore, in S3, obtaining the ellipse inclination angle of the fruit posture feature area includes:

[0045] S321, performing edge detection based on the fruit posture feature area to extract the contour line of the fruit posture feature area;

[0046] S322, smoothing the contour line to obtain a smoothed contour line of the fruit posture feature area;

[0047] S323, using the Hough transform ellipse detection algorithm to analyze and process the contour line of the fruit posture characteristic area after smoothing, and identify the ellipse parameters of the contour of the fruit posture characteristic area;

[0048] S324, based on the ellipse parameters of the contour of the fruit posture feature area, extract and verify the ellipse tilt angle until the preset conditions are met, and output the ellipse tilt angle.

[0049] Furthermore, the S4 includes:

[0050] S41, rotating the single pomelo fruit depth map based on the ellipse inclination angle to obtain a single pomelo fruit depth map with the fruit stem facing vertically upward;

[0051] S42, drawing a circumscribed rectangle of a depth map of a single pomelo fruit with a fruit stalk facing vertically upward, and determining the pixel length and width of the circumscribed rectangle;

[0052] S43, based on the two-dimensional coordinates u, v of the center point of the circumscribed regular rectangle in the image and the depth value d of the center point, convert the pixel coordinate system to the camera coordinate system to obtain the three-dimensional coordinates X of the center point in the camera coordinate system c ,Y c ,Z c ; The conversion formula is:

[0053]

[0054] Z, C d

[0055] Among them, f x and f y is the focal length of the camera, c x and c y is the pixel coordinate of the camera optical center;

[0056] S44, randomly selecting two points on both sides of the center point of the circumscribed rectangle, and using the conversion formula to calculate the three-dimensional coordinates of the two points, constructing a proportional relationship between the pixel length conversion and the actual length, and determining the length and width of the circumscribed rectangle; the length and width of the circumscribed rectangle are the longitudinal diameter H and the transverse diameter W of the pomelo fruit;

[0057] S45, based on the three-dimensional coordinates X of the center point in the camera coordinate system c ,Y c ,Zc And the vertical diameter H and horizontal diameter W of the pomelo fruit, obtain the cutting point coordinate X q ,Y q ,Z q ; The calculation formula is:

[0058]

[0059] T is the translation matrix:

[0060]

[0061] Where θ is the tilt angle of the fruit.

[0062] It can be seen from the above technical solution that compared with the prior art, the technical solution of the present invention has the following advantages:

[0063] Beneficial effects:

[0064] 1. Through image preprocessing steps, including standardization, denoising, and resizing, the quality of the original image data is optimized to ensure the accuracy of subsequent processing. At the same time, the use of RGB images and depth maps provides rich visual and spatial information, which is crucial for fruit recognition in complex orchard environments.

[0065] 2. The optimized instance segmentation model, especially the U-Net architecture that integrates deep separable convolution and self-attention mechanism, can efficiently and accurately segment individual pomelo fruits, significantly improving the processing speed and segmentation accuracy.

[0066] 3. By analyzing the depth map of a single pomelo fruit and combining it with the Hough ellipse detection algorithm, not only can the elliptical posture feature area of ​​the fruit be accurately identified, but the inclination angle of the fruit can also be directly extracted, ensuring robustness under different lighting conditions and the complexity of fruit arrangement.

[0067] 4. By using the elliptical inclination angle of the fruit posture feature area, the position and posture of the fruit stalk can be accurately calculated, which can effectively deal with the problem of fruit recognition under occlusion, improve the success rate of picking, reduce damage to the fruit, and ensure picking efficiency and fruit quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0069] Figure 1A flow chart of a method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device provided in an embodiment of the present invention;

[0070] Figure 2 A preprocessing flow chart provided for an embodiment of the present invention;

[0071] Figure 3 A flow chart of identifying an elliptical fruit posture feature area provided by an embodiment of the present invention;

[0072] Figure 4 A flow chart of obtaining the ellipse tilt angle of a fruit posture feature area provided in an embodiment of the present invention;

[0073] Figure 5 A schematic diagram of the contour line of the fruit posture feature area extracted according to an embodiment of the present invention;

[0074] Figure 6 A flow chart for calculating and obtaining the position and posture of a pomelo fruit stem provided in an embodiment of the present invention;

[0075] Figure 7 This is a schematic diagram of the characteristics of a pomelo fruit shape provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] like Figure 1 As shown, this embodiment provides a fruit stem posture recognition method applied to a pomelo fruit picking device, comprising the following steps:

[0078] S1. Preprocessing the image data of the scene to be identified to obtain preprocessed image data; the image data includes: an RGB image and a depth image;

[0079] S2, inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth map of a single pomelo fruit;

[0080] S3, based on the depth map of the single pomelo fruit, identifying the elliptical fruit posture feature area, and combining the Hough ellipse detection algorithm to obtain the ellipse inclination angle of the fruit posture feature area;

[0081] S4. Calculate the position and posture of the pomelo fruit stem through the ellipse inclination angle of the fruit posture characteristic area.

[0082] This method integrates a variety of advanced technologies to achieve full-process optimization from image data preprocessing to accurate recognition of fruit stalk posture, improves the recognition accuracy and work efficiency of pomelo fruit picking automation equipment, and reduces the mispicking rate and fruit damage rate.

[0083] The following is a further detailed description of each of the above steps:

[0084] like Figure 2 As shown, in this embodiment S1, the preprocessing includes:

[0085] S11. Use the spatiotemporal joint embedding network to align the RGB image and depth image in the image data to ensure the consistency of any pixel in the spatial dimension and the temporal dimension; specifically including:

[0086] S111. Use the spatiotemporal attention module in the spatiotemporal joint embedding network to analyze the optical flow information and depth difference between consecutive image frames to identify dynamic regions; S112. Combine the auxiliary layer in the spatiotemporal joint embedding network to capture time series data of the dynamic region; S113. Based on the results of dynamic region identification and time series analysis, implement an adaptive weight fusion strategy to adjust the fusion weights of the RGB image and depth image in the image data to enhance the reliance on color image features; S114. Use the gradient descent algorithm to perform multi-scale fusion of the RGB image and depth image after adaptive weight fusion to achieve precise alignment from the whole to the part.

[0087] It optimizes the spatiotemporal consistency of image data through a joint spatiotemporal embedding network. It uses dynamic information analysis, adaptive fusion strategy and multi-scale fusion technology to ensure the high synchronization and alignment of RGB images and depth images in the early stage of processing, providing accurate input data for subsequent processing steps and improving the robustness and accuracy of the entire fruit stalk posture recognition system.

[0088] S12, based on the aligned image data, filtering and smoothing the depth map in the image data by using an adaptive bilateral morphological filter to obtain filtered and smoothed image data; specifically comprising:

[0089] S121. Determine the filter kernel size of the adaptive bilateral morphological filter based on the local features of the depth map in the image data; S122. Perform depth adaptive weight optimization to assign higher weights to pixels with similar distances; S123. Perform morphological operations on the depth map in the image data after the depth adaptive weight optimization; in the morphological operations, dynamically select and execute a combination of opening and closing operations and parameter adjustments; S124. Establish a feedback mechanism for the difference between images before and after filtering, analyze and process the results of the morphological operations, and dynamically adjust the filter parameters to prevent over-smoothing.

[0090] This step first analyzes the local features of the depth map and intelligently determines the filter kernel size to meet the feature requirements of different regions. Then, a deep adaptive weight optimization strategy is implemented to dynamically adjust the weights according to the spatial relationship between pixels, strengthen the influence of neighboring pixels, and smooth the image while maintaining edge details. On this basis, morphological operations are performed to dynamically select and fine-tune the combination and parameters of opening and closing operations to adapt to changes in image content, effectively remove noise while maintaining structural integrity. Finally, by establishing a feedback mechanism for the difference between images before and after filtering, the processing effect is monitored and the filtering parameters are adjusted in time to avoid over-smoothing, ensuring that the processing results are both clean and retain the necessary detail information.

[0091] S13, combining the actual working radius and working height of the pomelo fruit picking equipment, setting a depth threshold, filtering pixels of the depth map in the filtered and smoothed image data, and generating a mask map;

[0092] In the specific process of determining rather than dynamically adjusting the depth threshold, an initial depth threshold range is first set based on the physical specifications of the robot arm and prior knowledge of the orchard operating environment. This range needs to take into account the maximum operating radius of the robot arm, the expected size of the fruit (such as pomelo), and its typical hanging height in the crown. Using statistical analysis of historical picking data, the average value of the fruit diameter and its distribution range are determined. Combined with the growth habits and branch density of orchard trees, a fixed depth threshold that can cover most of the effective fruit area and exclude background clutter is estimated. This threshold is used to filter pixels in the depth map and retain depth information that is within the preset range and may belong to the target fruit, thereby assisting subsequent fruit recognition and positioning tasks.

[0093] S14, superimposing the mask image and the RGB image in the image data after filtering and smoothing, filtering out the color pixels outside the picking range, and obtaining the image data after superimposing; specifically comprising:

[0094] According to the depth threshold set previously, a binary mask is generated. In this mask, the pixels within the robot's operating range are marked as white (or set to 1), indicating that these areas are of interest or valid; while the pixels outside the operating range are marked as black (or set to 0), indicating that they should be ignored or filtered out.

[0095] The generated binary mask image is operated pixel by pixel with the original color image. Specifically, a pixel-level logical AND operation can be used, that is, each pixel value of the mask image is logically ANDed with the corresponding color image pixel value, and only the color image pixels corresponding to the pixels with a value of 1 (valid) in the mask image are retained, and the pixels with a mask value of 0 are set to the background color or transparent (implemented as a zero value in some software or algorithms).

[0096] After completing the above logical AND operation, the "filtering" step has been completed, and the corresponding pixel information in the color image of all pixels marked as 0 in the mask image (i.e., the part not within the picking range) has been hidden or changed to the background color. This process can be regarded as directly eliminating unnecessary information at the image processing level. Through the above superposition and filtering process, not only the analysis area is effectively limited and the amount of calculation is reduced, but also the accuracy of the detection of pomelo fruit and its stalk in the subsequent processing steps is improved, especially under the conditions of complex background and occlusion.

[0097] S15, performing pomelo fruit contour enhancement on the image data after the superposition processing to obtain pre-processed image data.

[0098] S151. A dynamic background suppression strategy is adopted to analyze the optical flow information and inter-frame differences in the image data after superposition processing, update the background model, and remove background interference and dynamic change factors; S152. Based on the image data after removing background interference and dynamic change factors, combined with the occlusion attention mechanism, occluded areas and non-occluded areas are identified; S153. For the image data after the occluded areas and non-occluded areas are identified, a multi-scale feature pyramid network is used, combined with an edge detection algorithm, to visualize and enhance the pomelo contour at different scales to obtain an initial contour enhanced image; S154. An adaptive contour refinement network is introduced to perform intelligent refinement processing on the initial contour enhanced image, identify and correct contour defects caused by occlusion or image noise, and obtain preprocessed image data.

[0099] Through the orderly execution of this series of steps, it not only effectively suppresses background interference, accurately identifies and processes occluded areas, but also significantly enhances the visualization effect of pomelo fruit contours at multiple scales. Finally, highly accurate preprocessed image data is obtained through the intelligent refinement network, which provides a solid foundation for the detection and positioning of pomelo fruit stalk posture under occluded state.

[0100] In this embodiment S2, the instance segmentation model adopts a U-Net architecture combining deep separable convolution and self-attention mechanism, including: an encoder, an attention module, a decoder and an output layer;

[0101] Among them, the encoder uses the ResNet bottleneck module with deep separable convolution and adds a multi-task learning branch for depth map prediction, edge map prediction and fruit stem position prediction; the decoder adopts skip connection; the attention module is combined with a dynamic context enhancement strategy to dynamically adjust the channel and spatial attention weights according to the image content.

[0102] The instance segmentation model optimization includes: before training the instance segmentation model, a lightweight depth-wise enhancement strategy is used to perform lightweight transformation on the depth map in the image data;

[0103] During the instance segmentation model training process, an adaptive loss function is used; the adaptive loss function includes: category imbalance loss, boundary clarity loss and depth consistency loss.

[0104] The specific process of obtaining the depth map of a single pomelo fruit includes:

[0105] In the feature extraction and processing stage, the ResNet bottleneck module with deep separable convolution is used to extract rich features, the key information is strengthened through the attention module with dynamic context enhancement, and the segmentation accuracy is improved in the decoding stage by combining the skip connection technology.

[0106] Lightweight depth-wise enhancement and adaptive loss function optimization strategies are implemented in the model training process to enhance the model's adaptability to depth changes and the accuracy of segmentation boundaries.

[0107] The post-processing steps include generating a binary mask to distinguish the target fruit from the background, accurately extracting the depth area of ​​a single fruit, and eliminating edge noise through an erosion algorithm, finally obtaining a high-quality single fruit depth map.

[0108] Observing the pomelo fruit in the depth map, the pixels with the same depth value are surrounded in circles. Therefore, near the equator of the pomelo fruit, from the shoulder to the base of the fruit, there is an area surrounded by a circle of pixels with the same depth value, which appears as an ellipse in the depth image.

[0109] like Figure 3 As shown, in this embodiment S3, identifying the oval fruit posture feature area includes:

[0110] S311. Analyze the depth value range of a single pomelo fruit depth map through a sorting algorithm, set the depth threshold of the feature area, and remove pixels in non-target areas; S312. Combine morphological optimization with the single pomelo fruit depth map that removes pixels in the target area to refine the pomelo fruit area boundary; S313. Based on shape and texture features, screen the single pomelo fruit depth map after the pomelo fruit area boundary is refined to obtain an elliptical fruit posture feature area.

[0111] like Figure 4 As shown, the ellipse inclination angle of the fruit posture feature area is obtained, including:

[0112] S321, based on the fruit posture feature area, edge detection is performed to extract the contour line of the fruit posture feature area; Figure 5 As shown, the blue area in the upper image is the pomelo fruit with the stalk blocked, and the white ellipsoid in the lower image is the characteristic area of ​​the fruit posture.

[0113] S322, smoothing the contour line to obtain a contour line of the fruit posture feature area after smoothing;

[0114] S323, using the Hough transform ellipse detection algorithm to analyze and process the contour line of the fruit posture characteristic area after smoothing, and identify the ellipse parameters of the contour of the fruit posture characteristic area;

[0115] S324, based on the ellipse parameters of the contour of the fruit posture feature area, extract and verify the ellipse tilt angle until the preset conditions are met and output the ellipse tilt angle.

[0116] This step realizes the precise identification of the posture feature area of ​​a single pomelo fruit and the accurate extraction of the inclination angle of the ellipse, significantly improving the accuracy and reliability of fruit detection. First, through the intelligent analysis and processing of the depth map, non-fruit pixels are effectively removed and the fruit boundary is optimized. Combined with shape and texture information screening, the elliptical fruit area is accurately defined. In the precise fruit feature area, the contour line is ensured to be clear through meticulous edge detection and smoothing. The Hough transform ellipse detection algorithm is then used to accurately locate the contour parameters of the fruit posture feature area, and the inclination angle of the fruit is calculated accordingly. The entire process not only reduces background interference, but also improves the accuracy of fruit posture judgment through multi-step optimization, providing a solid technical foundation for applications such as automated picking and fruit classification.

[0117] like Figure 6 and Figure 7 As shown, in this embodiment S4, it includes:

[0118] S41, rotating the single pomelo fruit depth map based on the ellipse inclination angle to obtain a single pomelo fruit depth map with the fruit stem facing vertically upward;

[0119] S42, drawing a circumscribed regular rectangle of a depth map of a single pomelo fruit with a fruit stalk facing vertically upward, and determining the pixel length and width of the circumscribed regular rectangle;

[0120] S43, based on the two-dimensional coordinates u, v of the center point of the circumscribed regular rectangle in the image and the depth value d of the center point, convert the pixel coordinate system to the camera coordinate system to obtain the three-dimensional coordinates X of the center point in the camera coordinate system c ,Y c ,Z c ; The conversion formula is:

[0121]

[0122] Z, C d

[0123] Where fx and fy are the focal lengths of the camera, cx and cy are the pixel coordinates of the optical center of the camera;

[0124] S44, randomly selecting two points on both sides of the center point of the circumscribed rectangle, and using the conversion formula to calculate the three-dimensional coordinates of the two points, constructing a proportional relationship between the pixel length conversion and the actual length, and determining the length and width of the circumscribed rectangle; the length and width of the circumscribed rectangle are the longitudinal diameter H and the transverse diameter W of the pomelo fruit;

[0125] S45, based on the three-dimensional coordinates X of the center point in the camera coordinate system c ,Y c ,Z c As well as the longitudinal diameter H and transverse diameter W of the pomelo fruit, the coordinates of the cutting point Xq, Yq, Zq are obtained; the calculation formula is:

[0126]

[0127] T is the translation matrix:

[0128]

[0129] Where θ is the tilt angle of the fruit.

[0130] In this step, the image viewing angle is adjusted according to the elliptical inclination angle of the fruit to ensure that the fruit stem is vertical, and then a circumscribed rectangle is constructed and mapped to the camera coordinate system, and the three-dimensional coordinates of the fruit center and the key points of the circumscribed rectangle are accurately obtained. Using these coordinate points and camera parameters, not only the actual physical size of the fruit (vertical diameter and horizontal diameter) is calculated, but also the coordinates of the cutting point are determined through advanced geometric operations. This strategy effectively overcomes the measurement error caused by the change of the fruit posture, realizes the accurate mapping from two-dimensional image to three-dimensional space, and significantly enhances the accuracy of the measurement of the size and position of pomelo fruit.

[0131] In summary, the fruit stalk posture recognition method applied to the pomelo fruit picking device in this embodiment realizes the efficient recognition of the fruit stalk posture in the pomelo fruit picking device through a series of precise image processing and machine learning technologies. From the precise alignment of the RGB image and the depth image by the spatiotemporal joint embedding network in the preprocessing stage to the adaptive filtering and depth image optimization, the image quality and the accuracy of subsequent processing are effectively improved. The instance segmentation model adopted integrates the deep separable convolution and self-attention mechanism, combined with the lightweight deep wise enhancement and adaptive loss function optimization, which significantly enhances the robustness and segmentation accuracy of the model in a complex orchard environment. Through the fine elliptical posture feature recognition and Hough transform, the posture information of the fruit is accurately obtained, and the conversion from two-dimensional image to three-dimensional space is completed accordingly, and the position and posture of the fruit stalk are accurately located. In the end, this method can not only provide the precise coordinates of the fruit stalk for the pomelo fruit picking robot, but also provide strong technical support for the realization of automated and intelligent fruit picking, greatly improve the picking efficiency and the accuracy of fruit selection, reduce the need for manual intervention, and provide key technical support for the intelligent upgrading of modern agriculture.

[0132] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0133] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing the position and posture of a fruit stem applied to a grapefruit picking device, characterized in that: The following steps are involved: S1, preprocessing the image data of the scene to be identified to obtain preprocessed image data; The image data includes: an RGB image and a depth image; S2, inputting the preprocessed image data into an optimized instance segmentation model to obtain a depth map of a single pomelo fruit; S3, based on the depth map of the single pomelo fruit, identifying the elliptical fruit posture feature area, and combining the Hough ellipse detection algorithm to obtain the ellipse inclination angle of the fruit posture feature area; S4, calculating the position and posture of the pomelo fruit stem according to the ellipse inclination angle of the fruit posture characteristic area; The S4 comprises: S41, rotating the single pomelo fruit depth map based on the ellipse inclination angle to obtain a single pomelo fruit depth map with the fruit stem facing vertically upward; S42, drawing a circumscribed regular rectangle of a depth map of a single pomelo fruit with a fruit stalk facing vertically upward, and determining the pixel length and width of the circumscribed regular rectangle; S43, based on the two-dimensional coordinates of the center point of the circumscribed regular rectangle in the image ( u,v ) , and the depth value d of the center point, convert the pixel coordinate system to the camera coordinate system to obtain the three-dimensional coordinates of the center point in the camera coordinate system ( X c ,Y c ,Z c) ; The conversion formula is: Z c =d Among them, f x and f y is the focal length of the camera, c x and c y is the pixel coordinate of the camera optical center; S44, randomly selecting two points on both sides of the center point of the circumscribed rectangle, and using the conversion formula to calculate the three-dimensional coordinates of the two points, constructing a proportional relationship between the pixel length conversion and the actual length, and determining the length and width of the circumscribed rectangle; the length and width of the circumscribed rectangle are the longitudinal diameter H and the transverse diameter W of the pomelo fruit; S45, based on the three-dimensional coordinates of the center point in the camera coordinate system ( X c ,Y c ,Z c) As well as the vertical diameter H and horizontal diameter W of the pomelo fruit, obtain the cutting point coordinates ( X q ,Y q ,Z q ) ; The calculation formula is: T is the translation matrix: Where θ is the tilt angle of the fruit.

2. According to claim 1, a method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device is characterized in that: In S1, the preprocessing includes: S11. Use the spatiotemporal joint embedding network to align the RGB image and depth image in the image data to ensure the consistency of any pixel in the spatial dimension and the temporal dimension; S12, based on the aligned image data, filtering and smoothing the depth map in the image data by using an adaptive bilateral morphological filter to obtain filtered and smoothed image data; S13, combining the actual working radius and working height of the pomelo fruit picking equipment, setting a depth threshold, filtering pixels of the depth map in the filtered and smoothed image data, and generating a mask map; S14, superimposing the mask image and the RGB image in the image data after filtering and smoothing, filtering out color pixels outside the picking range, and obtaining the image data after superimposing; S15, performing pomelo fruit contour enhancement on the image data after the superposition processing to obtain pre-processed image data.

3. According to claim 2, a method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device is characterized in that: The S11 includes: S111, using the spatiotemporal attention module in the spatiotemporal joint embedding network to analyze the optical flow information and depth difference between consecutive image frames to perform dynamic area recognition; S112, combining the auxiliary layer in the spatiotemporal joint embedding network to capture time series data of the dynamic region; S113. Based on the results of dynamic region recognition and time series analysis, an adaptive weight fusion strategy is implemented to adjust the fusion weights of the RGB image and the depth image in the image data to enhance the reliance on color image features. S114. Use a gradient descent algorithm to perform multi-scale fusion of the RGB image and the depth image after adaptive weight fusion to achieve accurate alignment from the whole to the part.

4. The method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device according to claim 2, characterized in that: The S12 comprises: S121, determining a filter kernel size of an adaptive bilateral morphological filter based on local features of the depth map in the image data; S122, performing deep adaptive weight optimization to assign higher weights to pixels with similar distances; S123, performing morphological operation on the depth map in the image data after depth adaptive weight optimization; in the morphological operation, dynamically selecting and executing a combination of opening and closing operations and parameter adjustment; S124. Establish a feedback mechanism for the image difference before and after filtering, analyze and process the execution results of the morphological operation, and dynamically adjust the filtering parameters to prevent over-smoothing.

5. The method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device according to claim 2, characterized in that: The S15 comprises: S151, adopting a dynamic background suppression strategy, analyzing the optical flow information and inter-frame differences in the image data after superposition processing, updating the background model, and removing background interference and dynamic change factors; S152, based on the image data after removing background interference and dynamic change factors, combined with the occlusion attention mechanism, identify the occluded area and the non-occluded area; S153, for the image data after the occluded area and the non-occluded area are identified, a multi-scale feature pyramid network is used in combination with an edge detection algorithm to perform visualization enhancement of the pomelo fruit contour at different scales to obtain an initial contour enhanced image; S154, introducing an adaptive contour refinement network, performing intelligent refinement processing on the initial contour enhanced image, identifying and correcting contour defects caused by occlusion or image noise, and obtaining preprocessed image data.

6. The method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device according to claim 1, characterized in that: In S2, the instance segmentation model adopts a U-Net architecture that combines deep separable convolution with a self-attention mechanism, including: an encoder, an attention module, a decoder, and an output layer; Among them, the encoder uses the ResNet bottleneck module with deep separable convolution and adds a multi-task learning branch for depth map prediction, edge map prediction and fruit stem position prediction; the decoder adopts skip connection; the attention module is combined with a dynamic context enhancement strategy to dynamically adjust the channel and spatial attention weights according to the image content.

7. The method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device according to claim 1, characterized in that: In S2, the instance segmentation model optimization includes: before training the instance segmentation model, using a lightweight depth wise enhancement strategy to perform a lightweight transformation on the depth map in the image data; During the instance segmentation model training process, an adaptive loss function is used; the adaptive loss function includes: category imbalance loss, boundary clarity loss and depth consistency loss.

8. The method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device according to claim 1, characterized in that: In S3, identifying the oval fruit posture feature area includes: S311, analyzing the depth value interval of a single pomelo fruit depth map through a sorting algorithm, setting a depth threshold of a feature area, and removing pixels in non-target areas; S312, combining morphological optimization with the single pomelo fruit depth map from which pixels in the target area are removed, to refine the pomelo fruit area boundary; S313. Based on shape and texture features, the depth map of a single pomelo fruit after the pomelo fruit region boundary is refined is screened to obtain an elliptical fruit posture feature region.

9. The method for recognizing the position and posture of a fruit stem applied to a pomelo fruit picking device according to claim 1, characterized in that: In S3, obtaining the ellipse inclination angle of the fruit posture feature area includes: S321, performing edge detection based on the fruit posture feature area to extract the contour line of the fruit posture feature area; S322, smoothing the contour line to obtain a contour line of the fruit posture feature area after smoothing; S323, using the Hough transform ellipse detection algorithm to analyze and process the contour line of the fruit posture characteristic area after smoothing, and identify the ellipse parameters of the contour of the fruit posture characteristic area; S324, based on the ellipse parameters of the contour of the fruit posture feature area, extract and verify the ellipse tilt angle until the preset conditions are met and output the ellipse tilt angle.

Images