Method and device for realizing close-range perspective estimation of serial tomatoes based on depth enhancement and geometric constraint, processor and readable storage medium thereof

Abstract

The present application relates to a kind of based on depth enhancement and geometric constraint realization string tomato close-range view angle estimation method, comprising the following steps: using depth camera to shoot the color image and depth image of long-range, by adaptive image filter processing, all string tomatoes to be picked in image are detected by TOLO model;The related coordinates of detection frame and color image are input, and the segmentation mask of string tomato is generated, the color image is input to generate depth feature map and is segmented using mask, two depth images are fused, and the string tomato point cloud of curvature feature enhancement is obtained;Input enhanced string tomato point cloud, combined with point cloud projection feature, according to geometric constraint, the optimal close-range view angle is estimated.The method, device, processor and readable storage medium of the present application based on depth enhancement and geometric constraint realization string tomato close-range view angle estimation, by providing robust and rich semantic information view angle guide, lay the foundation for subsequent picking operation, significantly improve picking success rate and system stability.

Description

Technical Field

[0001] This invention relates to the field of agricultural robot harvesting technology, and more particularly to the field of bunch tomato harvesting robots. Specifically, it refers to a method, device, processor, and readable storage medium for estimating the close-up view of bunch tomatoes based on depth enhancement and geometric constraints. Background Technology

[0002] In the field of tomato-harvesting robots, although depth cameras have been widely used in target detection tasks, the point cloud data obtained in long-distance imaging scenarios often exhibits sparsity, making it difficult to accurately characterize features such as the curvature of the fruit surface. This problem stems from the physical limitations of depth sensing, hardware performance bottlenecks, and external environmental interference, which restrict the robotic arm's accurate perception of the target's spatial morphology. When the robotic arm performs fine operations such as grasping or cutting, the system is highly dependent on the fine-grained semantic segmentation results and high-precision 3D structural information of the target. Therefore, the system needs to actively search for and locate a near-field observation viewpoint that provides sufficient visual semantic information during the switching between near and far views, so that the optimal harvesting posture can be further derived based on the imaging data from this viewpoint.

[0003] However, existing research often treats distant detection and close-up harvesting as independent stages. Most methods, after distant localization, simply approach the target at a fixed distance, ignoring the influence of the observation perspective, or directly estimate the harvesting pose based on the distant RGB-D image. Such strategies have two limitations: first, the loss of depth texture information due to depth loss significantly affects the accuracy of pose estimation; second, the dependence of close-up harvesting on fine-grained semantic information is not effectively modeled. Therefore, there is an urgent need to propose a method that can estimate the close-up observation perspective that maximizes intra-string fruit visibility from distant images with only low-precision feature information, thus providing reliable input for high-precision harvesting operations. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, processor and its readable storage medium for estimating the close-up view of a bunch of tomatoes based on depth enhancement and geometric constraints, which has a high harvesting success rate, good system stability and wide applicability.

[0005] To achieve the above objectives, the present invention provides a method, apparatus, processor, and readable storage medium for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints, as follows: The method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints is characterized by the following steps: (1) Using a depth camera to capture color and depth images of the distant view under the initial pose of the robotic arm, the images are processed by an adaptive image filter, and the TOLO model is used to detect all the bunches of tomatoes to be picked in the image and return the relevant coordinates of the detection box. (2) Input the relevant coordinates of the detection box and the color image, generate the segmentation mask of the tomato string, input the color image to generate the depth feature map and use the mask for segmentation, fuse the two depth images, and obtain the tomato string point cloud with enhanced curvature features; (3) Input the enhanced string of tomato point cloud, combine the point cloud projection features, and estimate the optimal close-up view based on geometric constraints.

[0006] Preferably, step (1) specifically includes the following steps: (1.1) Initialize the robotic arm's distant shooting pose, make the X and Z axes of the camera coordinate system parallel to the ground, and the Y axis perpendicular to the ground, mount the camera on the end effector of the robotic arm, and capture RGB-D images; (1.2) After filtering the background pixels and their depth using an adaptive image filter, the images are input into the TOLO model to identify and locate the tomato cluster; (1.3) The TOLO model infers all the detection boxes of the tomato clusters based on the input RGB-D image, and returns the coordinates of the top left vertex, the bottom right vertex, and the center point of each detection box in order from left to right and from top to bottom.

[0007] Preferably, the adaptive image filter in step (1.2) achieves an adaptive foreground focusing effect based on depth information, specifically including the following steps: (1.2.1) Filter out all valid depth values ​​greater than 0 from the depth map and flatten them into a one-dimensional array; (1.2.2) The K-means clustering algorithm was used to perform cluster analysis on the effective depth values, and three cluster centers were obtained, which correspond to the near, middle and far depth levels in the scene respectively; (1.2.3) Construct a full-map label matrix with the same size as the depth map, and map the cluster labels corresponding to the effective depth regions to the matrix so that each pixel in the full map is assigned a depth level label of near, middle, or far. (1.2.4) Generate a mask that retains only the near-field cluster label region, and filter the RGB-D image based on the mask.

[0008] Preferably, step (1.3) further includes the steps of designing and training the TOLO model, specifically including the following steps: (1.3.1) Replace the Stem Layer of the YOLOv11 model with the dual-modal fusion module CDFormer; (1.3.2) Replace Stage Layer 1 of the YOLOv11 model with the multimodal attention mechanism RGB-DAttention; (1.3.3) Replace the feature extraction module C3K2 of the YOLOv11 model with FDW-C3K2; (1.3.4) Add a weighted occlusion loss function, RepulsionLoss, to the loss function of the YOLOv11 model to complete the design of the TOLO model; (1.3.5) Use the color image, depth image and corresponding labels processed by the adaptive image filter to train the TOLO model, update the network weight values ​​and obtain the trained TOLO model; (1.3.6) The trained TOLO model is used to detect the input color image and depth image after adaptive image filtering, and the detected tomato detection boxes are returned.

[0009] Preferably, step (2) specifically includes the following steps: (2.1) The coordinates of the upper left corner, the lower right corner and the center of each detection box are taken as input and fed into the Segment Anything model to obtain the tomato segmentation mask in the region. (2.2) Based on the original color image input to the Depth Anything model, a high-precision depth feature map is generated, and the depth feature map is cropped using the segmentation mask of each tomato. (2.3) The original depth map is semantically cropped using the segmentation mask output by the Segment Anything model and fused with the depth feature map. Combined with camera parameters, a refined point cloud representation with enhanced curvature features is generated.

[0010] Preferably, step (3) specifically includes the following steps: (3.1) Define the reference pose coordinate system for the string of tomatoes, where, The axis extends from top to bottom along the main stem of the fruit bunch. The axis is perpendicular to the direction of the fishbone-like structure on the surface of the tomato cluster. Shaft shaft and The cross product of the axes determines the direction; (3.2) Based on the tomato pixel mask obtained in (2.1), fit its main downward direction due to gravity in the image space, and extract the slope of the line corresponding to this direction. A preliminary estimate of the orientation of the fruit clusters was made. (3.3) Enhance the point cloud along The axial direction is divided into 3 equally spaced segments, and the point cloud of each segment is projected onto... - For a plane, extract its outer contour edge points and fit a straight line to obtain multiple candidate linear slopes; calculate the projection length of all points on the plane along the slope direction, and select the slope corresponding to the longest projected line segment as denoted as . ; (3.4) Enhance the point cloud along The axis is divided into 5 equally spaced segments, and each segment is projected onto... - For each projection point set, fit its longest line segment on the plane and extract the corresponding slope as a candidate; by calculating the projection length of all points in this linear direction, select the slope corresponding to the longest projection, denoted as . .

[0011] (3.5) Select the center of the enhanced point cloud as the mass of the bunch of tomatoes. Define the coordinate system on the tomato string. The direction vector is , The direction vector is Based on the three slopes , , Calculated , , , , and Calculate the pose that maximizes the visibility of fruits within a string of tomatoes. and take the distance from the particle. The position is the zero point of the camera coordinate system, and the pose matrix required for the robotic arm to move to the next pose is returned.

[0012] The device for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints is characterized by comprising: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints.

[0013] The processor for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints is characterized in that the processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the various steps of the aforementioned method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints.

[0014] The computer-readable storage medium is characterized in that it stores a computer program that can be executed by a processor to implement the various steps of the above-described method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints.

[0015] The present invention employs a method, apparatus, processor, and readable storage medium for estimating the close-up view of a bunch of tomatoes based on depth enhancement and geometric constraints. As a key bridge connecting distant detection and close-up harvesting, it provides robust and semantically rich view guidance, laying the foundation for subsequent harvesting operations and significantly improving the harvesting success rate and system stability. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention.

[0017] Figure 2 This is a schematic diagram illustrating the implementation of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention.

[0018] Figure 3 This is a schematic diagram of the steps of the TOLO target detection model for the method of estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention.

[0019] Figure 4 This is a schematic diagram of the network structure of the TOLO model for the method of estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention.

[0020] Figure 5 This is a schematic diagram of the CDFormer network structure for the method of estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention.

[0021] Figure 6 This is a schematic diagram of the RGB-D Attention network structure of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention.

[0022] Figure 7 This is a schematic diagram of the FDW-C3K2 network structure of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to the present invention. Detailed Implementation

[0023] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0024] The present invention provides a method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints, comprising the following steps: (1) Using a depth camera to capture color and depth images of the distant view under the initial pose of the robotic arm, the images are processed by an adaptive image filter, and the TOLO model is used to detect all the bunches of tomatoes to be picked in the image and return the relevant coordinates of the detection box. (2) Input the relevant coordinates of the detection box and the color image, generate the segmentation mask of the tomato string, input the color image to generate the depth feature map and use the mask for segmentation, fuse the two depth images, and obtain the tomato string point cloud with enhanced curvature features; (3) Input the enhanced string of tomato point cloud, combine the point cloud projection features, and estimate the optimal close-up view based on geometric constraints.

[0025] In a preferred embodiment of the present invention, step (1) specifically includes the following steps: (1.1) Initialize the robotic arm's distant shooting pose, make the X and Z axes of the camera coordinate system parallel to the ground, and the Y axis perpendicular to the ground, mount the camera on the end effector of the robotic arm, and capture RGB-D images; (1.2) After filtering the background pixels and their depth using an adaptive image filter, the images are input into the TOLO model to identify and locate the tomato cluster; (1.3) The TOLO model infers all the detection boxes of the tomato clusters based on the input RGB-D image, and returns the coordinates of the top left vertex, the bottom right vertex, and the center point of each detection box in order from left to right and from top to bottom.

[0026] In a preferred embodiment of the present invention, the adaptive image filter in step (1.2) achieves an adaptive foreground focusing effect based on depth information, specifically including the following steps: (1.2.1) Filter out all valid depth values ​​greater than 0 from the depth map and flatten them into a one-dimensional array; (1.2.2) The K-means clustering algorithm was used to perform cluster analysis on the effective depth values, and three cluster centers were obtained, which correspond to the near, middle and far depth levels in the scene respectively; (1.2.3) Construct a full-map label matrix with the same size as the depth map, and map the cluster labels corresponding to the effective depth regions to the matrix so that each pixel in the full map is assigned a depth level label of near, middle, or far. (1.2.4) Generate a mask that retains only the near-field cluster label region, and filter the RGB-D image based on the mask.

[0027] In a preferred embodiment of the present invention, step (1.3) further includes the steps of designing and training the TOLO model, specifically including the following steps: (1.3.1) Replace the Stem Layer of the YOLOv11 model with the dual-modal fusion module CDFormer; (1.3.2) Replace Stage Layer 1 of the YOLOv11 model with the multimodal attention mechanism RGB-DAttention; (1.3.3) Replace the feature extraction module C3K2 of the YOLOv11 model with FDW-C3K2; (1.3.4) Add a weighted occlusion loss function, RepulsionLoss, to the loss function of the YOLOv11 model to complete the design of the TOLO model; (1.3.5) Use the color image, depth image and corresponding labels processed by the adaptive image filter to train the TOLO model, update the network weight values ​​and obtain the trained TOLO model; (1.3.6) The trained TOLO model is used to detect the input color image and depth image after adaptive image filtering, and the detected tomato detection boxes are returned.

[0028] In a preferred embodiment of the present invention, step (2) specifically includes the following steps: (2.1) The coordinates of the upper left corner, the lower right corner and the center of each detection box are taken as input and fed into the Segment Anything model to obtain the tomato segmentation mask in the region. (2.2) Based on the original color image input to the Depth Anything model, a high-precision depth feature map is generated, and the depth feature map is cropped using the segmentation mask of each tomato. (2.3) The original depth map is semantically cropped using the segmentation mask output by the Segment Anything model and fused with the depth feature map. Combined with camera parameters, a refined point cloud representation with enhanced curvature features is generated.

[0029] In a preferred embodiment of the present invention, step (3) specifically includes the following steps: (3.1) Define the reference pose coordinate system for the string of tomatoes, where, The axis extends from top to bottom along the main stem of the fruit bunch. The axis is perpendicular to the direction of the fishbone-like structure on the surface of the tomato cluster. Shaft shaft and The cross product of the axes determines the direction; (3.2) Based on the tomato pixel mask obtained in (2.1), fit its main downward direction due to gravity in the image space, and extract the slope of the line corresponding to this direction. A preliminary estimate of the orientation of the fruit clusters was made. (3.3) Enhance the point cloud along The axial direction is divided into 3 equally spaced segments, and the point cloud of each segment is projected onto... - For a plane, extract its outer contour edge points and fit a straight line to obtain multiple candidate linear slopes; calculate the projection length of all points on the plane along the slope direction, and select the slope corresponding to the longest projected line segment as denoted as . ; (3.4) Enhance the point cloud along The axis is divided into 5 equally spaced segments, and each segment is projected onto... - For each projection point set, fit its longest line segment on the plane and extract the corresponding slope as a candidate; by calculating the projection length of all points in this linear direction, select the slope corresponding to the longest projection, denoted as . .

[0030] (3.5) Select the center of the enhanced point cloud as the mass of the bunch of tomatoes. Define the coordinate system on the tomato string. The direction vector is , The direction vector is Based on the three slopes , , Calculated , , , , and Calculate the pose that maximizes the visibility of fruits within a string of tomatoes. and take the distance from the particle. The position is the zero point of the camera coordinate system, and the pose matrix required for the robotic arm to move to the next pose is returned.

[0031] The present invention provides an apparatus for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints, wherein the apparatus comprises: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints.

[0032] The processor of the present invention for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints is configured to execute computer-executable instructions, which, when executed by the processor, implement the various steps of the above-described method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints.

[0033] The computer-readable storage medium of the present invention stores a computer program that can be executed by a processor to implement the various steps of the above-described method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints.

[0034] To address the point cloud sparsity issue caused by long-distance imaging with depth cameras and the need for fine-grained semantic features during the approach phase of a robotic arm's harvesting process, this invention proposes a close-range perspective estimation method for bunches of tomatoes based on depth enhancement and geometric constraints. First, an RGB-D dual-modal fusion target detection model, TOLO, is designed. Through dual-modal feature extraction and cross-modal attention enhancement, accurate localization of bunches of tomatoes is achieved. Second, the depth texture distribution of the bunches of tomatoes is optimized based on the Depth Anything and Segment Anything models and fused with the original depth image to enhance the curvature features of the point cloud surface. Finally, by analyzing the 3D projection characteristics of the enhanced point cloud and introducing geometric constraints based on prior physical factors such as gravity, an optimal close-range perspective estimation algorithm is proposed to guide the robotic arm to approach the target in a posture that maximizes the visibility of fruits within the bunch (e.g., a herringbone configuration).

[0035] This invention is based on a method for estimating the close-up viewpoint of a string of tomatoes using depth enhancement and geometric constraints, such as... Figure 1 , 2 As shown, it mainly includes: Step S1: With the robotic arm in its initial pose, use a depth camera to capture color and depth images of the distant scene. After processing with an adaptive image filter, the TOLO model detects all the bunches of tomatoes to be picked in the image and returns the relevant coordinates of the detection box.

[0036] Step S2: Input the relevant coordinates of the detection box and the color image. The Segment Anything model generates a segmentation mask for the tomato cluster, and the Depth Anything model inputs the color image to generate a depth feature map and uses the mask for segmentation. Merge the two depth images to obtain a tomato cluster point cloud with enhanced curvature features.

[0037] Step S3: Input the enhanced string of tomato point clouds, combine the point cloud projection features, and estimate the optimal close-up view based on geometric constraints.

[0038] Optimally, step S1, the detection of distant clusters of tomatoes, specifically includes the following steps: Step S1.1: Initialize the robotic arm's distant shooting pose, requiring the X and Z axes of the camera coordinate system to be parallel to the ground, and the Y axis to be perpendicular to the ground (i.e., conforming to the direction of gravity). Mount the Azure Kinect V2 camera on the robotic arm's end effector and capture RGB-D images.

[0039] Step S1.2: After filtering the background pixels and their depth using an adaptive image filter, input them into the TOLO model to identify and locate the tomato cluster.

[0040] Step S1.3: The TOLO model infers the detection boxes of all strings of tomatoes based on the input RGB-D image, and returns the coordinates of the top left vertex, bottom right vertex, and center point of each detection box in order from left to right and from top to bottom.

[0041] The adaptive image filter described above is used to achieve an adaptive foreground focusing effect based on depth information. First, all valid depth values ​​greater than 0 are selected from the depth map (excluding invalid depth data), and this value is flattened into a one-dimensional array. Second, the K-means clustering algorithm is used to cluster the valid depth values, resulting in three cluster centers corresponding to the near, middle, and far depth levels in the scene. A full-image label matrix with the same size as the depth map is constructed, and the cluster labels corresponding to the valid depth regions are mapped to this matrix, so that each pixel in the full image is assigned a label representing one of the near, middle, or far depth levels. Finally, a mask is generated that retains only the near-field cluster label regions (near-field regions are valid, middle and far-field regions are invalid), and the RGB-D image is filtered based on this mask.

[0042] The network structure of the TOLO model is optimized based on YOLOv11, and the design steps are as follows: Figure 3 As shown, the network structure is as follows Figure 4 As shown, firstly, the Stem Layer of the YOLOv11 model is replaced with the dual-modal fusion module CDFormer; next, Stage Layer 1 of the YOLOv11 model is replaced with the multimodal attention mechanism RGB-D Attention; then, the feature extraction module C3K2 of the YOLOv11 model is replaced with FDW-C3K2; finally, a weighted occlusion loss function, Repulsion Loss, is added to the loss function of the YOLOv11 model, thus completing the design of the TOLO model. After completing the model design, the TOLO model is trained using color images processed by adaptive image filters, depth images, and corresponding labels, updating the network weight values ​​to obtain the trained TOLO model. Finally, the trained TOLO model is used to detect the input color images and depth images processed by adaptive image filters, returning the identified tomato cluster detection boxes.

[0043] Optimally, step S2, the enhancement of the geometric features of the tomato point cloud, specifically includes the following steps: Step S2.1: Input the coordinates of the top left corner, bottom right corner, and center of each detection box into the Segment Anything model to obtain the tomato segmentation mask in that region.

[0044] Step S2.2: Based on the original color image input to the Depth Anything model, generate a high-precision depth feature map to represent the depth texture information of the fruit surface within the target fruit bunch. The depth feature map is then cropped using a segmentation mask for each tomato bunch.

[0045] Step S2.3: Use the segmentation mask output by the Segment Anything model to perform semantic cropping on the original depth map, and fuse it with the depth feature map. Combine the camera parameters to generate a refined point cloud representation with enhanced curvature features.

[0046] Ideally, step S3, the optimal close-up view estimation algorithm, specifically includes the following steps: Step S3.1: Define the reference pose coordinate system for the string of tomatoes, such as... Figure 2 As shown. Among them, The axis extends from top to bottom along the main stem of the fruit bunch. The axis is perpendicular to the direction of the fishbone-like structure on the surface of the tomato cluster. Shaft shaft and The cross product of the axes is determined to satisfy the requirements of the right-hand coordinate system.

[0047] Step S3.2: Based on the tomato pixel mask obtained in step S2.1, fit its main downward direction due to gravity in the image space, and extract the slope of the straight line corresponding to this direction. It is used to make a preliminary estimate of the orientation of the fruit bunch.

[0048] Step S3.3: Move the enhanced point cloud along... The point cloud is divided into three equally spaced segments along the axis. Each segment is projected onto... - For a plane, extract its outer contour edge points and fit a straight line to obtain multiple candidate linear slopes. Based on this, calculate the projection length of all points on the plane along the slope direction, and select the slope corresponding to the longest projected line segment as denoted as . .

[0049] Step S3.4: Further enhance the point cloud along... The axis is divided into 5 equally spaced segments, and each segment is projected onto... - For each projection point set, fit its longest line segment on the plane and extract the corresponding slope as a candidate. Calculate the projection length of all points along this linear direction, and select the slope corresponding to the longest projection, denoted as . .

[0050] Step S3.5: Select the center of the enhanced point cloud as the mass of the tomato cluster. Define the coordinate system on the tomato string. The direction vector is , The direction vector is Based on the three slopes , , Using the following formula: ; Calculate the pose that maximizes the visibility of fruits within a string of tomatoes. and take the distance from the particle. The current position is the zero point of the camera coordinate system. Return the pose matrix required for the robotic arm to move to the next pose: .

[0051] In a specific embodiment of this invention, the proposed close-range perspective estimation method for bunch-of-tomatoes based on depth enhancement and geometric constraints is applicable to bunch-of-tomatoes harvesting robots with "eyes in hands." Before running this algorithm, basic steps such as depth camera selection, camera parameter calibration, and hand-eye calibration are required. It should be noted that the initial pose of the bunch-of-tomatoes harvesting robot when capturing distant images should be referenced to the camera coordinate system (…). Figure 2 As shown), the camera coordinate system is required to be... axis, The axis is parallel to the ground. The axis is perpendicular to the ground (i.e., it aligns with the direction of gravity). This is done to facilitate subsequent estimations of the coordinate system established on the string of tomatoes. The camera coordinate system can always conform to the direction in which the bunch of tomatoes hangs down due to gravity. The axis serves as a reference for the direction of gravity.

[0052] Figure 1 This is a flowchart of the algorithm. Figure 2 This is a schematic diagram of the algorithm. The algorithm consists of three parts: First, target detection of a bunch of tomatoes is performed based on a distant image. Second, the depth texture distribution of the bunch of tomatoes is optimized using the Depth Anything and Segment Anything models to enhance the curvature features of the tomato point cloud and reconstruct the geometric details of the fruit surface. Finally, an optimal near-field perspective estimation algorithm is proposed based on the enhanced point cloud to guide the robotic arm to approach the target in a posture that maximizes the visibility of the fruits within the bunch.

[0053] For the first part of this algorithm, preprocessing of the color and depth images captured by the depth camera is required. Specifically, an adaptive image filter is designed to achieve foreground focusing and background noise suppression. First, effective pixels (depth values ​​greater than 0) in the depth map are extracted and converted into a one-dimensional array, providing a data foundation for subsequent clustering analysis. Next, the K-means clustering algorithm is used to cluster the one-dimensional depth array, with a cluster size of 3 to distinguish the three main depth levels in the scene: near, middle, and far. This clustering result reflects the distribution characteristics of different depth ranges in the scene. Subsequently, a full-image label matrix with the same size as the depth map is constructed, and the depth value of each effective pixel is mapped to one of the near, middle, or far labels according to its clustering result, achieving hierarchical labeling in the depth space. Finally, a near-field mask is extracted based on the clustering results (i.e., only the area with the near-field cluster label is retained, and its corresponding pixels are set to valid, while the rest are set to invalid). The color image and depth map are then subjected to pixel-level filtering based on the mask to remove interference information in the mid-to-far-field areas, thereby obtaining image data focused on the harvesting operation surface.

[0054] The preprocessed image is input into the designed TOLO object detection model, which infers the positions of all bunches of tomatoes within the picking area and outputs the coordinate information of the detection boxes (coordinates of the top-left vertex, bottom-right vertex, and center point of the detection box). The proposed TOLO object detection model follows the following... Figure 3 Improvements.

[0055] The backbone feature extraction network uses the CDFormer module as the starting point for dual-modal feature processing, performing co-encoding on RGB and depth images. The structure of the CDFormer module is as follows: Figure 5 As shown, when extracting RGB-D features, the traditional RGB-D dual-branch encoding is not used. Instead, the depth feature encoding branch is completely eliminated, retaining only a single RGB feature extraction backbone. The input RGB image undergoes RGB Patches Embedding, dividing the image into several image patches and mapping them to high-dimensional feature vectors. Simultaneously, Positional Encoding is introduced to inject spatial location information into each image patch, enabling the model to perceive the spatial arrangement relationship between image patches. Meanwhile, the input depth image is fed into the Depth Prior Generator module, which first divides the depth image into... Image patches, and build upon them Depth matrix To capture the self-similarity of deep structures, establish Position matrix The positional relationships of image patches are supplemented, and then fused through SE Blocks to generate a Depth Prior, providing prior knowledge in the depth domain for RGB-D cross-modal fusion. Subsequently, RGB features carrying positional information and the generated Depth Prior are input into the RGB-D Encoder. This encoder uses a Transformer as its core architecture, and its Geometry Self-Attention module, by introducing the Depth Prior, guides the attention mechanism to focus on the geometric relationship between RGB and depth data, enhancing the alignment and interaction of cross-modal features. Next, residual connections and layer normalization operations using Add&LayerNorm stabilize the training process and alleviate the gradient vanishing problem. Afterward, the feature flow undergoes a high-dimensional nonlinear transformation through the FeedForward network, and then a second Add&LayerNorm further optimizes the feature representation. In addition, a wavelet convolution module is introduced between the two RGB-D Encoders. By leveraging the time-frequency analysis capabilities of wavelet transform, it compensates for the shortcomings of traditional convolution in "global feature extraction" and "multi-scale representation," increasing the receptive field. CDFormer can output discriminative features that integrate RGB texture and depth geometry information, providing strong support for subsequent tasks such as RGB-D detection.

[0056] Depth Matrix The calculation formula is as follows: ; express OK Image blocks after uniform pooling.

[0057] Position matrix The calculation formula is as follows: ; express and Euclidean distance.

[0058] Following the CDFormer module is the RGB-D Attention module, such as... Figure 6 As shown, the RGB-D Attention module enhances the extraction capability of key features across different modalities through the cascaded collaboration of Multi-Channel self-Attention and spatial attention. The Multi-Channel self-Attention employs a dual-branch structure: one branch aggregates spatial dimensional information through global pooling to obtain a channel-level global representation; the other branch... Convolution and reshape operations transform features into... This provides the foundation for subsequent spatial weight calculations. Next, the features obtained from global pooling are divided into three parallel branches. Each branch is processed by a fully connected layer and the SiLU activation function, generating channels with a total number of... , , The feature vectors are used to capture multi-scale inter-channel dependencies (different numbers of channels correspond to channel interactions of different granularities). Meanwhile, After the features are activated by Softmax, attention weights for spatial location are generated (encoding the importance distribution of spatial dimensions). Subsequently, multi-scale channel features and spatial weights are aggregated through element-wise multiplication, and then processed through multiple... The depthwise separable convolutional module adjusts the channel dimensions and finally outputs the result through residual connections. The channel attention weights effectively enhance the channel expressive power of features.

[0059] For the YOLO architecture, the feature extraction module plays a crucial role in detection performance. Therefore, this invention proposes a lightweight and efficient FDW-C3K2 module to improve the C3K2 layer structure in YOLOv11, thereby enhancing the model's feature representation ability and detection accuracy in complex agricultural scenarios. Figure 7 As shown. This module integrates frequency-dynamic convolution and depthwise separable convolution. The specific process and improvement details are as follows: First, the input features are processed through... The convolutional layer performs initial channel dimension adjustment and feature compression to reduce subsequent computational complexity and provide adapted feature representations for parallel branch processing. Subsequently, a split operation divides the feature stream into two parallel branches, one of which achieves fine-grained, multi-stage feature extraction by cascading multiple FDW-C3K modules. Within each FDW-C3K module, the feature stream is first... Convolution completes the channel pre-adjustment, and then multiple FDW-Bottleneck units are concatenated: In the FDW-Bottleneck, Frequency-dependent dynamic convolution (FDConv) is used to capture dynamic features in the frequency domain, effectively adapting to the multi-scale and complex texture distribution of targets in agricultural scenes; subsequently... Depthily separable convolution (DWConv) significantly reduces the number of parameters and computational cost while ensuring feature interaction between channels, achieving a lightweight module design. Subsequent Batch Norm layers normalize the features, and the SiLU activation function introduces non-linearity to enhance feature expressiveness. Residual connections preserve original feature information, further improving feature robustness. Another branch directly passes the original feature stream, ensuring the integrity of the basic features. Finally, the features from the two branches are fused along the channel dimension using a Concat operation, integrating feature information from different branches, and then... The convolutional layer completes the final integration of channels and calibration of feature maps, outputting a feature representation that combines rich semantic information with lightweight characteristics. Compared to the original C3K2 layer of YOLOv11, FDW-C3K2 enhances the feature adaptability to complex agricultural scenarios through frequency dynamic convolution. It achieves a balance between computational efficiency and feature representation capability by leveraging depthwise separable convolution and residual structures, providing higher-quality feature input for the detection head, thereby improving the model's detection accuracy and inference efficiency in agricultural scenarios.

[0060] The total loss of the TOLO model consists of the bounding box loss. With category loss The weighted average is calculated using the following formula: ; in, , These are the weighting coefficients for the bounding box loss and the class loss, respectively, used to balance the contributions of the two types of losses.

[0061] The bounding box loss is CIoU Loss ( ), Distribution Focal Loss ( ), RepulsionLoss ( The weighted sum of ), that is: ; CIoU Loss introduces center distance and aspect ratio penalties on top of IoU to optimize bounding box localization accuracy. The formula is as follows: ; in, Represents the prediction box With real frame The intersection and union ratio; These are the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. The distance between the center of the predicted bounding box and the center of the ground truth bounding box; To surround and The length of the diagonal of the smallest bounding rectangle; The aspect ratio difference coefficient ( To predict the width and height of the bounding box, (The actual frame width and height). A coefficient to balance the aspect ratio penalty.

[0062] Distribution Focal Loss optimizes the probability distribution uncertainty of bounding box coordinates by shifting the predicted distribution towards the true offset. The formula is: ; in, The four dimensions of the bounding box ( Center offset, Center offset, Width, high); For the first time to be measured The probability distribution of dimensional offsets ( (where the number of intervals is the discretization interval of the distribution). For the first The distribution of dimensions (usually a Gaussian distribution around the true offset, i.e., the interval near the true offset). (Approaching 1, the rest approach 0).

[0063] Repulsion Loss is used to reduce the overlap between predicted bounding boxes and interference boxes (such as background boxes and ground truth boxes of adjacent objects). The formula is: ; in, This represents the total number of predicted boxes; Indicates the first The "attraction loss" between each predicted bounding box and its corresponding ground truth bounding box (usually reusing the IoU part of CIoU Loss); Indicates the first The interference boxes of the predicted boxes (such as those in the same image) (non-corresponding ground truth boxes or background candidate boxes with high overlap). Indicates the weight of the exclusion term; This indicates the overlap threshold.

[0064] The TOLO model uses Focal Loss (to address the class imbalance problem) for its class loss, and the formula is as follows: ; Where M represents the number of samples involved in the category calculation (positive samples are predicted boxes containing the target, and negative samples are background boxes). For sample labels ( Indicates a positive sample. (representing negative samples). For the first Each positive sample is predicted as the true class. The probability of; For the first The probability that a negative sample is predicted as the background class (category 0); For focusing parameters.

[0065] The second part of this algorithm, which focuses on enhancing the point cloud features of the tomato cluster, is as follows: First, the coordinates of the top left corner, bottom right corner, and center of each detection box are sequentially fed into the Segment Anything model to obtain the segmentation mask of the tomato cluster within that region. Then, based on the original color image input into the Depth Anything model, a high-precision depth map that can represent the geometric shape and depth texture information of the target fruit cluster is generated. Next, the original depth map and the high-precision depth map are cropped using the segmentation mask of each tomato cluster, and then fused using the depth texture transfer algorithm below, combined with camera parameters to generate a refined point cloud representation with enhanced curvature features.

[0066] Specifically, the deep texture feature transfer is illustrated in the following pseudocode: ; For the optimal close-up view estimation algorithm in the third part of this algorithm, the reference pose coordinate system of the tomato string is first defined, such as... Figure 2 As shown. Among them, The axis extends from top to bottom along the main stem of the fruit bunch. The axis is perpendicular to the direction of the fishbone-like structure on the surface of the tomato cluster. Shaft shaft and The cross product of the axes is determined to satisfy the requirements of the right-hand coordinate system. The origin of the coordinate system is the center coordinate of the tomato point cloud, calculated using the following formula: ; in The total number of points in the point cloud. For the first point cloud The three-dimensional coordinates of each point.

[0067] Next, the direction vectors of each coordinate system on the string of tomatoes need to be calculated based on the projection characteristics of the enhanced point cloud. Since the string of tomatoes is elongated, the projection along the Y-axis is the most obvious. Therefore, we will first focus on the string of tomatoes. Axis in - plane and - Calculate the slope of the projection onto the plane.

[0068] According to the principles of camera imaging, the captured image is a point cloud on the imaging plane. - Therefore, the projection onto the image is directly used, and the tomato pixel mask generated by the Segment Anything model is used to fit its main downward direction due to gravity in the image space. That is, the least squares method is used to fit a straight line for the non-zero pixels of the mask, and the slope of the straight line corresponding to that direction is extracted. Used for preliminary estimation of fruit bunches The direction of the axis.

[0069] Project the enhanced point cloud onto - plane, and along The axial direction is divided into three equally spaced segments. For each segment, the outer contour edge points are extracted and a straight line is fitted to obtain multiple candidate linear slopes. Based on this, the projection length of all points on the plane along the slope direction is calculated, and the slope corresponding to the longest projected line segment is denoted as . The optimal slope selection algorithm is shown below.

[0070] Project the tomato cluster point cloud onto - Plane view. For tomatoes with only one side of the fruit cluster visible in the camera's field of view, the point cloud density in that area is low, even resembling outliers, because the fruit edge often corresponds to a distant region or a sloping surface. When both sides of the fruit are visible in the camera's field of view, the curvature of the fruit surface will affect the point cloud projection. Two noticeable bulges are observed in the negative axial direction. Therefore, for - Planar projection represents the surface condition of the fruit bunch, and the coordinate system of the tomato bunch is calculated based on the projection points. Axis in - The slope of the line in the planar projection. Specifically, this will enhance the point cloud along... The axis is divided into 5 equally spaced segments, and each segment is projected onto... - For each projection point set, fit its longest line segment on the plane and extract the corresponding slope as a candidate. Calculate the projection length of all points along this linear direction, and select the slope corresponding to the longest projection, denoted as . .

[0071] ; Select the center of the enhanced point cloud as the center point of the tomato cluster. Define the coordinate system on the tomato string. The direction vector is , The direction vector is Based on the three slopes , , Using the following formula: ; Solving for: ; Calculate the pose that maximizes the visibility of fruits within a string of tomatoes. and take the distance from the particle. The current position is the zero point of the camera coordinate system. Return the pose matrix required for the robotic arm to move to the next pose: .

[0072] This invention primarily addresses the problem of insufficient semantic information provided by close-up perspectives in tomato picking scenarios. The described posture is not the posture of the picking point, but rather the camera's observation posture. The perspective transformation in this invention is not manually assigned, and there is only one active perspective transformation. Tomato identification and localization are completed through a single perspective, and reliable evidence for the next perspective transformation is then derived from image analysis.

[0073] For the specific implementation scheme of this embodiment, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0074] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0075] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0076] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0077] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0078] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0079] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0080] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0081] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0082] The present invention employs a method, apparatus, processor, and readable storage medium for estimating the close-up view of a bunch of tomatoes based on depth enhancement and geometric constraints. As a key bridge connecting distant detection and close-up harvesting, it provides robust and semantically rich view guidance, laying the foundation for subsequent harvesting operations and significantly improving the harvesting success rate and system stability.

[0083] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for estimating the close-up viewpoint of a string of tomatoes based on depth enhancement and geometric constraints, characterized in that, The method includes the following steps: (1) Using a depth camera to capture color and depth images of the distant view under the initial pose of the robotic arm, the images are processed by an adaptive image filter, and the TOLO model is used to detect all the bunches of tomatoes to be picked in the image and return the relevant coordinates of the detection box. (2) Input the relevant coordinates of the detection box and the color image, generate the segmentation mask of the tomato string, input the color image to generate the depth feature map and use the mask for segmentation, fuse the depth image of the distant scene captured by the depth camera and the depth feature map generated by the color image to obtain the tomato string point cloud with enhanced curvature features. (3) Input the enhanced string of tomato point cloud, combine the point cloud projection features, and estimate the optimal close-up view based on geometric constraints; Step (1) specifically includes the following steps: (1.1) Initialize the robotic arm's distant shooting pose, make the X and Z axes of the camera coordinate system parallel to the ground, and the Y axis perpendicular to the ground, mount the camera on the end effector of the robotic arm, and capture RGB-D images; (1.2) After filtering the background pixels and their depth using an adaptive image filter, the images are input into the TOLO model to identify and locate the tomato cluster; (1.3) The TOLO model infers all the detection boxes of the tomato clusters based on the input RGB-D image, and returns the coordinates of the top left vertex, the bottom right vertex, and the center point of each detection box in order from left to right and from top to bottom. Step (1.3) also includes the training steps of the TOLO model: (1.3.1) Replace the Stem Layer of the YOLOv11 model with the dual-modal fusion module CDFormer; (1.3.2) Replace Stage Layer 1 of the YOLOv11 model with the multimodal attention mechanism RGB-D Attention; (1.3.3) Replace the feature extraction module C3K2 of the YOLOv11 model with the FDW-C3K2 module; (1.3.4) Add a weighted occlusion loss function, Repulsion Loss, to the loss function of the YOLOv11 model to complete the design of the TOLO model; (1.3.5) Use the color image, depth image and corresponding labels processed by the adaptive image filter to train the TOLO model, update the network weight values ​​and obtain the trained TOLO model; (1.3.6) The trained TOLO model is used to detect the input color image and depth image after adaptive image filtering, and the detected tomato cluster detection boxes are returned. The dual-modal fusion module CDFormer outputs discriminative features that fuse RGB texture and depth geometric information; the FDW-C3K2 module fuses frequency dynamic convolution and depth separable convolution; and the RGB-D Attention module enhances the ability to extract key features between different modalities through the cascaded collaboration of Multi Channel self-Attention and spatial attention.

2. The method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to claim 1, characterized in that, The adaptive image filter in step (1.2) achieves an adaptive foreground focusing effect based on depth information, specifically including the following steps: (1.2.1) Filter out all valid depth values ​​greater than 0 from the depth map and flatten them into a one-dimensional array; (1.2.2) The K-means clustering algorithm was used to perform cluster analysis on the effective depth values, and three cluster centers were obtained, which correspond to the near, middle and far depth levels in the scene respectively; (1.2.3) Construct a full-map label matrix with the same size as the depth map, and map the cluster labels corresponding to the effective depth regions to the matrix so that each pixel in the full map is assigned a depth level label of near, middle, or far. (1.2.4) Generate a mask that retains only the near-field cluster label region, and filter the RGB-D image based on the mask.

3. The method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to claim 1, characterized in that, Step (2) specifically includes the following steps: (2.1) The coordinates of the upper left corner, the lower right corner and the center of each detection box are taken as input and fed into the Segment Anything model to obtain the tomato segmentation mask in the detection box area; (2.2) Input the original color image into the Depth Anything model to generate a high-precision depth feature map, and use the segmentation mask of each tomato to crop the depth feature map; (2.3) The original depth map is semantically cropped using the segmentation mask output by the Segment Anything model and fused with the depth feature map. Combined with camera parameters, a refined point cloud representation with enhanced curvature features is generated.

4. The method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints according to claim 3, characterized in that, Step (3) specifically includes the following steps: (3.1) Define the reference pose coordinate system for the string of tomatoes, where, The axis extends from top to bottom along the main stem of the fruit bunch. The axis is perpendicular to the direction of the fishbone-like structure on the surface of the tomato cluster. Shaft shaft and The cross product of the axes determines the direction; (3.2) Based on the tomato segmentation mask obtained in (2.1), fit its main downward direction due to gravity in the image space, and extract the slope of the line corresponding to this direction. A preliminary estimate of the orientation of the fruit clusters was made. (3.3) Enhance the point cloud along The axial direction is divided into 3 equally spaced segments, and the point cloud of each segment is projected onto... - For a plane, extract its outer contour edge points and fit a straight line to obtain multiple candidate linear slopes; calculate the projection length of all points on the plane along the slope direction, and denote the slope of the longest projected line segment as . The aforementioned axis, shaft and The axes are the camera coordinate system; (3.4) Enhance the point cloud along The axis is divided into 5 equally spaced segments, and each segment is projected onto... - For each set of projected points, fit the longest line segment on the plane and extract the corresponding slope as a candidate. Calculate the projected length of all points along this linear direction, and denote the slope of the longest projected line segment as... ; (3.5) Select the center of the enhanced point cloud as the mass of the bunch of tomatoes. Define the coordinate system on the tomato string. The direction vector is , The direction vector is Based on the three slopes , , Calculated , , , , and Calculate the pose that maximizes the visibility of fruits within a string of tomatoes. and take the distance from the particle. The position is the zero point of the camera coordinate system, and the pose matrix required for the robotic arm to move to the next pose is returned.

5. A device for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints as described in any one of claims 1 to 4.

6. A processor for estimating the close-up viewpoint of a string of tomatoes based on depth enhancement and geometric constraints, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method for estimating the close-up view of a string of tomatoes based on depth enhancement and geometric constraints as described in any one of claims 1 to 4.

Images