Fruit stem and calyx three-dimensional feature identification method based on monocular depth estimation

Abstract

The invention discloses a fruit stem and calyx three-dimensional feature identification method based on monocular depth estimation. RGB images of fruits and real depth images corresponding to the RGB images are collected, then all the RGB images are registered and marked, and a fruit RGB and depth pairing data set is obtained; constructing a monocular depth estimation model, and inputting the fruit RGB and depth pairing data set into the monocular depth estimation model for training to obtain a trained monocular depth estimation model; inputting the RGB image of the to-be-detected fruit into the trained monocular depth estimation model to obtain a predicted depth map of the to-be-detected fruit; and determining the stem and calyx areas of the to-be-detected fruit according to the predicted depth map of the to-be-detected fruit. According to the method, the inference speed is remarkably improved while the recognition precision of areas such as fruit stems and calyxes of fruits is ensured, a high-resolution texture branch and edge guide smoothing module is introduced, the three-dimensional feature boundary is effectively enhanced, and the integrity and definition of a segmentation result are ensured.

Description

Technical Field

[0001] This invention relates to the field of machine vision, specifically to a method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation. Background Technology

[0002] With the development of fruit and vegetable sorting and automated inspection technologies, accurately distinguishing defective areas on the fruit surface and avoiding misidentifying fruit stems and calyxes as defects has become a crucial aspect affecting fruit quality grading and commercialization. Traditional machine vision methods mainly rely on RGB color images for detection and segmentation (Lopez, JJ, Cobos, M. & Aguilera, E. Computer-based detection and classification of flaws in citrusfruits. Neural Comput & Applic 20). However, because fruit stems and calyxes are similar to some surface defect areas in grayscale distribution and color contrast, they are prone to false positives and false negatives, thus reducing the accuracy and robustness of the grading system.

[0003] Deep learning technology has been introduced into the detection of surface defects in fruits and vegetables and the recognition of fruit stalks and calyxes. Two-dimensional convolutional neural networks have achieved certain results in extracting features such as texture and edges, but due to the limitation of using only RGB color information, their ability to represent differences in three-dimensional geometric morphology is insufficient, especially under conditions of light changes, surface reflection, or partial occlusion of the fruit surface, where the recognition performance drops significantly (Liu, J.; Sun, J.; Wang, Y.; Liu, X.; Zhang, Y.; Fu, H. Non-Destructive Detection of Fruit Quality: Technologies, Applications and Prospects. Foods2025, 14, 2137.).

[0004] To overcome the limitations of two-dimensional vision, researchers have attempted to introduce three-dimensional feature extraction techniques, such as structured light, binocular stereo vision, and depth cameras, to obtain geometric structural information about fruit surfaces (Xia M, Zhu, et al. Stem and Calyx Identification of 3D Apples Using Multi-Threshold Segmentation and 2D Convex Hull. Photonics 2022, 9, 346). These methods can effectively characterize the morphological differences between fruit stalks and calyxes, improving recognition results. However, such systems are typically expensive, complex to calibrate, and bulky, and also suffer from insufficient real-time performance in scenarios such as high-speed conveyor belts, limiting their large-scale application in fruit production lines.

[0005] Methods based on monocular depth estimation (MDE) have developed rapidly. These methods rely solely on monocular RGB images, using deep neural networks to infer dense depth maps of the scene, thus acquiring 3D geometric features at low cost. However, directly applying monocular depth estimation to fruit stalk and calyx recognition still faces the following challenges: 1) In actual production, it is difficult to collect large-scale RGB-D paired data, which limits the training effect of deep networks; 2) Existing monocular depth estimation algorithms mostly use models with a large number of parameters and complex calculations, resulting in high training costs and slow inference speeds, which cannot meet the needs of high-throughput and real-time detection on fruit sorting production lines.

[0006] 3) Existing monocular depth estimation models are mostly trained based on outdoor data, and pre-trained models are difficult to extract high-precision three-dimensional features from fruits; 4) The fruit stalk and the edge of the calyx are small and have slight geometric differences, making it difficult to achieve accurate separation using traditional depth estimation methods. Summary of the Invention

[0007] To address the problems existing in the background technology, a three-dimensional feature recognition method for fruit pedicels and calyxes based on monocular depth estimation is proposed. This method can effectively utilize a large number of unlabeled samples through knowledge distillation and semi-supervised training, even with limited labeled depth data. It improves model lightweighting and real-time performance while maintaining accuracy, thus meeting the application requirements of high-throughput fruit production lines.

[0008] The technical solution of this invention is: The method of the present invention includes the following steps: S1. Collect RGB images of fruits and corresponding true depth images of RGB images, and then register and label all RGB images to obtain a fruit RGB and depth paired dataset. S2. Construct a monocular depth estimation model. Input the fruit RGB and depth pairing dataset into the monocular depth estimation model for training to obtain a trained monocular depth estimation model. S3. Input the RGB image of the fruit to be tested into the trained monocular depth estimation model to obtain the predicted depth map of the fruit to be tested. S4. Determine the fruit stalk and calyx regions of the fruit to be tested based on the predicted depth map of the fruit to be tested.

[0009] The monocular depth estimation model includes a fruit stalk and calyx depth recognition model and a trained depth guidance model. The fruit stalk and calyx depth recognition model serves as the student model, and the trained depth guidance model serves as the teacher model. The RGB images of the fruit are input into the fruit stalk and calyx depth recognition model and the depth guidance model, respectively. The fruit stalk and calyx depth recognition model outputs the student predicted depth map, and the depth guidance model outputs the teacher pseudo depth map. The student predicted depth map, the teacher pseudo depth map, and the real depth image corresponding to the RGB image are input into the loss function to calculate the distillation loss, and then backpropagation is used to update the fruit stalk and calyx depth recognition model in the monocular depth estimation model.

[0010] The fruit stalk and calyx depth recognition model includes three parallel branches, an edge-guided smoothing module, and a depth prediction head. The three parallel branches are an edge feature extraction branch, a high-resolution texture branch, and an encoder-decoder branch. The RGB image of the fruit is input into the edge feature extraction branch, the high-resolution texture branch, and the encoder-decoder branch, respectively, and then outputs the edge amplitude map, the high-resolution feature map, and the multi-scale decoding feature map, respectively. Subsequently, the high-resolution feature map and the multi-scale decoding feature map are fused to obtain a fused feature map. Then, the fused feature map and the edge amplitude map are input together into the edge guided smoothing module for feature smoothing processing, and output a smoothed feature map. Finally, the smoothed feature map is input into the depth prediction head, and the depth prediction head outputs the student predicted depth map.

[0011] The edge feature extraction branch includes a grayscale conversion operation, a Sobel convolution operation unit, and an edge magnitude calculation unit. After the RGB image of the fruit is input to the edge feature extraction branch, the RGB image is first converted to a grayscale image through a grayscale conversion operation. Then, the grayscale image is input to the Sobel convolution operation unit, and horizontal and vertical Sobel convolution operations are performed to obtain horizontal gradient maps and vertical gradient maps, respectively. Finally, the edge magnitude calculation unit calculates the edge magnitude map based on the horizontal gradient map and the vertical gradient map.

[0012] The encoder-decoder branch includes an encoder and a decoder. The RGB image of the fruit is input into the encoder for multi-scale feature extraction, and outputs feature maps of five different scales. The decoder includes four levels of upsampling blocks connected in sequence. The upsampling block upsamples the output of the previous level upsampling block and concatenates and fuses it with the feature map of the corresponding scale in the encoder. The fused feature is then processed by convolution and used as the output of the current level upsampling block. The output of the last level upsampling block is used as the multi-scale decoded feature map.

[0013] The edge-guided smoothing module processes the edge according to the following formula:

[0014]

[0015]

[0016] in, Represents a smooth feature map. Represents the fused feature map. Calculate the weighting coefficients for the gating mechanism. Represents the edge amplitude map. This represents a smoothed intermediate feature map. It is the sigmoid activation function. The preset mean convolution kernel channel weights, The preset mean convolution kernel, This represents the bias term in the gating mechanism.

[0017] The depth-guided model includes an image automatic processor, a Transformer-DPT feature extraction module, an automatic depth estimation module, and a bilinear upsampling module connected in sequence. The RGB image of the fruit is input to the image automatic processor for formatting and then sent to the Transformer-DPT feature extraction module for feature extraction to obtain an intermediate feature map. The intermediate feature map is then processed by the automatic depth estimation module for depth estimation and the bilinear upsampling module for size restoration to obtain a pseudo-depth map of the teacher.

[0018] The loss function of the monocular depth estimation model is mainly composed of a weighted sum of the BerHu loss function, SILog loss function, gradient consistency loss function, edge constraint loss function, and normal consistency loss function, and is specifically set according to the following formula:

[0019] in, Indicates depth of fruit stalk and calyx for depth recognition With true depth Anti-Huber losses Indicates depth of fruit stalk and calyx for depth recognition Compared to the scale-invariant log loss for depth-guided pseudo-depth, Represents the gradient consistency loss function. Represents the edge constraint loss function. Represents the normal consistency loss function. These are the weight hyperparameters for the BerHu loss function, SILog loss function, gradient consistency loss function, edge constraint loss function, and normal consistency loss function, respectively. The BerHu loss function is set according to the following formula:

[0020]

[0021]

[0022]

[0023]

[0024] in, Predicting depth maps for students i The absolute error of the depth value of each effective pixel. Predicting depth maps for students i The predicted depth value for each valid pixel. For the true depth image i The actual depth value of each effective pixel. Here, is the preset segmentation threshold, and is the correction coefficient. It is the set of valid pixels in the true depth image. For set The number of elements; The SILog loss function is set according to the following formula:

[0025]

[0026]

[0027]

[0028] in, For each pixel, predict the logarithmic difference between the depth and the target depth. The mean squared error is used to measure the variance or error strength of the overall prediction. The average error is used to measure the average deviation of the overall prediction. For the teacher pseudo-depth graph, the first i Depth value of each effective pixel, The preset numerical stability constant, This is the preset variance-mean compromise factor; The gradient consistency loss function is set according to the following formula:

[0029]

[0030]

[0031]

[0032]

[0033] in, and These represent the gradient consistency errors in the horizontal and vertical directions, respectively. and The student models predict depth values ​​for both horizontal and vertical directions, respectively. and These are the actual depth values ​​for the horizontal and vertical directions, respectively. and These are sets where adjacent pixels are valid both horizontally and vertically. and Sets and The number of pixel pairs in the middle These are the two-dimensional coordinates of pixels in the image. This indicates that the pixel is a valid pixel. For logical AND operator; The edge constraint loss function is set according to the following formula:

[0034] in, and These are the edge magnitudes calculated using the Sobel operator from the student's predicted depth map and the actual depth image, respectively. The normal consistency loss function is set according to the following formula:

[0035]

[0036] in, and Let be the predicted normal vector and the true normal vector at the i-th valid pixel, respectively. Let be the mask for the i-th valid pixel. Let be the norm of the vector.

[0037] The beneficial effects of this invention are: This invention utilizes a depth-guided, pedicel and calyx depth recognition distillation framework, which significantly improves inference speed while maintaining the recognition accuracy of fruit pedicels, calyxes, and other regions. High-resolution texture branches and edge-guided smoothing modules are introduced into the SC-DepthNet pedicel and calyx depth recognition model to effectively enhance 3D feature boundaries and ensure the integrity and clarity of the segmentation results. Simultaneously, a lightweight ResNet34 backbone and a 4-level upsampling block are employed to achieve efficient inference, meeting the high-throughput real-time detection requirements of fruit production lines, while balancing accuracy, speed, and ease of integration. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of the model of the present invention.

[0039] Figure 2 This is a schematic diagram of a fruit RGB and depth paired dataset.

[0040] Figure 3 This is a schematic diagram of the structure of the upsampling block, edge extraction branch, high-resolution branch, edge-guided smoothing module, and depth prediction head.

[0041] Figure 4 This is a schematic diagram of the loss function calculation process.

[0042] Figure 5 It is the depth map predicted by the student model.

[0043] Figure 6 It is the confusion matrix of the inference result accuracy. Detailed Implementation

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0046] The method of the present invention includes the following steps: S1. Collect RGB images of fruits and corresponding true depth images of RGB images, and then register and label all RGB images to obtain a fruit RGB and depth paired dataset. In this embodiment, an RGB and depth paired dataset of fruits was constructed using apples as the experimental subject. Image samples are as follows: Figure 2 As shown, the top row is the FRUIT-RGB color image, and the bottom row is the corresponding FRUIT-D depth image. The image samples are uniformly preprocessed. The size of the input image is normalized to 640×640 pixels, and the image data is enhanced, including random angle rotation, mirror flip, vertical flip, Gaussian blur, mean blur, median blur, adding Gaussian noise, and cropping.

[0047] This embodiment has two datasets, which are used for the deep guided model L-DepthNet and the fruit stalk and calyx depth recognition model SC-DepthNet, respectively.

[0048] The dataset used for the fruit stalk and calyx depth recognition model is a dataset of about 1,000 RGB-D apple images. The RGB and depth maps are acquired synchronously frame by frame. The dataset is divided into training set, validation set and test set in a 6:2:2 ratio and used as labeled data for supervised training. The dataset used by the deep guided model consists of approximately 9,000 RGB apple images. These images do not contain depth labels, but pseudo-labels are generated by the Depth-Anything-V2 pre-trained weights called by L-DepthNet during training to participate in semi-supervised learning.

[0049] S2. Construct a monocular depth estimation model. Input the fruit RGB and depth pairing dataset into the monocular depth estimation model for training to obtain a trained monocular depth estimation model. S3. Input the RGB image of the fruit to be tested into the trained monocular depth estimation model to obtain the predicted depth map corresponding to the RGB image of the fruit to be tested. S4. Determine the fruit stalk and calyx regions of the fruit based on the predicted depth map corresponding to the RGB image of the fruit to be tested. Specifically, step S4 involves performing binary classification on the generated depth map to obtain the classification results for the fruit stalk, calyx, and other regions, thereby obtaining the classification confusion matrix, as shown in the figure. Figure 6 As shown.

[0050] like Figure 1 As shown, the monocular depth estimation model includes a fruit stalk and calyx depth recognition model and a trained depth guidance model. The fruit stalk and calyx depth recognition model serves as the student model, which is used to output the student's predicted depth map. The trained depth guidance model serves as the teacher model, which is used to generate the teacher's pseudo depth map based on the RGB image of the fruit. The RGB images of the fruit are input into the fruit stalk and calyx depth recognition model and the depth guidance model, respectively. The fruit stalk and calyx depth recognition model outputs the student-predicted depth map, as shown in the image. Figure 5 As shown, the depth-guided model outputs a pseudo-depth map of the teacher. The student's predicted depth map, the pseudo-depth map of the teacher, and the real depth image corresponding to the RGB image are input into the loss function to calculate the distillation loss. Then, backpropagation is used to update the fruit stalk and calyx depth recognition model in the monocular depth estimation model.

[0051] The teacher model used in the training phase of step S2 is a pre-trained model. The training in step S2 involves using the teacher model to train the student model. Therefore, labeled RGB images (i.e., RGB images and their corresponding ground truth depth images) are needed to train the teacher model to obtain a trained teacher model. Then, the same RGB image is input into both the student model and the teacher model. The student model generates a predicted depth map, and the teacher model generates a pseudo-depth map, respectively. The loss is then calculated by combining the outputs of the two models and the ground truth depth image to optimize the student model. Therefore, the testing phase of step S3 only requires using the trained student model to generate the predicted depth map.

[0052] The SC-DepthNet model for recognizing the depth of fruit stalks and calyxes is based on an improved UNet framework. It employs a ResNet-34 encoder with a four-level upsampling block (Depth-Upblock), and fuses a high-resolution texture branch (HiRes Branch) at the top layer, while adding an edge feature extraction module (EdgeMagnitude, which combines Sobel edge detection with gated mean convolution). The specific implementation is as follows: The pedicel and calyx depth recognition model includes three parallel branches, an edge-guided smoothing module, and a depth prediction head. The three parallel branches are an edge feature extraction branch, a high-resolution texture branch, and an encoder-decoder branch. The RGB image of the fruit is input into the edge feature extraction branch, the high-resolution texture branch, and the encoder-decoder branch, respectively, which output the edge amplitude map, the high-resolution feature map, and the multi-scale decoding feature map, respectively. Then, the high-resolution feature map and the multi-scale decoding feature map are fused to obtain the fused feature map. The fused feature map and the edge amplitude map are then input into the edge guided smoothing module for feature smoothing processing, and the smoothed feature map is output. Finally, the smoothed feature map is input into the depth prediction head, and the depth prediction head outputs the student predicted depth map.

[0053] The edge feature extraction branch includes grayscale conversion, Sobel convolution operation unit, and edge magnitude calculation unit. After the RGB image of the fruit is input to the edge feature extraction branch, the RGB image is first converted to a grayscale image through grayscale conversion. Then the grayscale image is input to the Sobel convolution operation unit, and horizontal and vertical Sobel convolution operations are performed to obtain horizontal gradient map and vertical gradient map respectively. Finally, the edge magnitude calculation unit calculates the edge magnitude map based on the horizontal gradient map and vertical gradient map.

[0054] The high-resolution texture branch includes two convolutional layers, two normalization layers, two ReLU activation functions, and a bilinear interpolation module. After the RGB image of the fruit is input into the high-resolution texture branch, it first passes through the first convolutional layer, the first normalization layer, and the first ReLU activation function for feature extraction. Then, it passes through the second convolutional layer, the second normalization layer, and the second ReLU activation function to further extract high-resolution texture features. Finally, the output of the second ReLU activation function is input into the bilinear interpolation module to obtain a high-resolution feature map.

[0055] The High-Resolution Texture Branch (HiRes Branch) and the Edge Feature Extraction Branch extract texture information from the input raw image through two convolutional layers, outputting 32-channel feature maps. They also utilize the Sobel operator to compute edge magnitude maps of the input image, introducing explicit edge information into depth estimation. These feature maps are concatenated with the decoder's output feature map to enhance detail.

[0056] The encoder-decoder branch includes an encoder and a decoder. The encoder uses a ResNet34 network structure. The RGB image of the fruit is input into the encoder for multi-scale feature extraction, and the output is five feature maps at different scales. The decoder includes four levels of upsampling blocks connected in sequence. The first-level upsampling block upsamples the output of the highest layer of the encoder. The remaining upsampling blocks upsample the output of the previous level upsampling block and concatenate and fuse them with the feature maps of the corresponding scale in the encoder. The fused features are then processed by convolution and used as the output of the current level upsampling block. The output of the last level upsampling block is used as the multi-scale decoded feature map.

[0057] Specifically, the data-enhanced RGB-D apple image dataset is processed by the ResNet34 encoder to extract feature maps at different scales layer by layer: e0 (1 / 2 resolution, 64 channels), e1 (1 / 4 resolution, 64 channels), e2 (1 / 8 resolution, 128 channels), e3 (1 / 16 resolution, 256 channels), and e4 (1 / 32 resolution, 512 channels). Then, the decoder employs a four-level upsampling block structure, such as... Figure 3As shown, each level fuses high-level semantic features with low-level detail features of the encoder through skip connections. During the upsampling process, the number of channels is gradually reduced, and finally a 32-channel feature map is recovered.

[0058] ResNet34 serves as the backbone network, responsible for extracting multi-level features from the input image. The input image passes through the first convolutional layer, batch normalization, and activation function (ReLU) to generate the first feature map e0. Max pooling and the first residual module (layer 1) are then used to reduce the image size and extract deeper features e1. Subsequently, deeper residual modules (layer 2, layer 3, layer 4) are used to gradually extract higher-level features e2, e3, and e4, further reducing the image size.

[0059] The feature maps extracted by the network serve as input to the decoder for subsequent spatial resolution recovery. The decoder employs a UNet architecture, incorporating skip connections from the encoder to aid in image spatial resolution restoration. The decoder contains four main upsampling blocks, each comprising a convolutional layer and a batch normalization layer. Bilinear interpolation is used to upsample the feature maps, which are then concatenated with the encoder's feature maps. Skip connections combine low-level features e0 with high-level features e4, enhancing the decoder's ability to recover details.

[0060] The edge feature extraction branch calculates the edge magnitude map of the input image using the Sobel operator, introducing explicit edge information into depth estimation, such as... Figure 3 As shown. Then, it is fed together with the decoder's fused features into a gated smoothing unit, which suppresses noise interference while ensuring clear edges, improving the stability and boundary accuracy of depth prediction. Specifically:

[0061] in, For edge amplitude, It is the numerical stability constant. and These are obtained by performing horizontal and vertical Soble convolutions on the grayscale image, as shown below:

[0062]

[0063] in, The image is a grayscale image. , These are the horizontal and vertical Sobel convolution kernels, respectively.

[0064] High-resolution branch (HiRes Branch) Figure 3As shown, the module directly extracts 32-channel detail texture features from the input image and concatenates them with the backbone features at the top layer of the decoder to compensate for the loss of edge and texture information in deep networks. Its mathematical representation is as follows:

[0065] in, Indicates the input image. Represents the ReLU activation function. This indicates an instance normalization operation. The final high-resolution features are then bilinearly interpolated to... Size, and concatenate with the decoder output:

[0066] in, It is a high-resolution branch fusion feature map. It is the feature map output by the decoder.

[0067] After that, and The feature introduction of the edge-gated smoother module allows the network to accurately segment details such as fruit stalks and calyxes, for example... Figure 3 As shown, the module computes the Sobel edge map of the input image, smooths the feature map using depth-wise mean convolution, and uses a gating mechanism to determine whether to retain the original feature map or the smoothed feature map for each pixel, thereby preserving important edge information.

[0068] The edge-guided smoothing module processes edges according to the following formula:

[0069]

[0070]

[0071] in, Represents a smooth feature map. Represents the fused feature map. Calculate the weighting coefficients for the gating mechanism. Represents the edge amplitude map. This represents a smoothed intermediate feature map. It is the sigmoid activation function. The preset mean convolution kernel channel weights, The preset mean convolution kernel, , This represents the bias term in the gating mechanism.

[0072] Finally, output features After passing through the depth prediction head, such as Figure 3 The depth prediction result is then activated by the Softplus function and output with a resolution of 1×H×W. .

[0073] The depth prediction head consists of two convolutional layers, a ReLU activation function, a Softplus activation function, and a bilinear interpolation module. The smoothed feature map is processed sequentially by the third convolutional layer, the third ReLU activation function, the fourth convolutional layer, and the Softplus activation function, and then upsampled by the bilinear interpolation module to obtain the student's predicted depth map.

[0074] This invention adds an edge-gated smoothing module before the output head. This module is used by the network to accurately segment details such as fruit stalks and calyxes. It removes noise from the image while preserving edge information. The module calculates the Sobel edge map of the input image, smooths the feature map using depth-wise mean convolution, and uses a gating mechanism to determine whether to retain the original or smoothed feature map for each pixel, thus preserving important edge information.

[0075] The decoder output is passed through a 1x1 convolutional layer and activated using the Softplus activation function to ensure that the output depth map is positive. The depth map is then upsampled to restore it to the size of the input image, yielding the final depth estimate.

[0076] The depth-guided model consists of an Automatic Image Processor (APS), a Transformer-DPT feature extraction module, an Automatic Depth Estimation (AMFDE) module, and a Bilinear Upsampling module, connected in sequence. The RGB image of the fruit is input to the Automatic Image Processor for formatting and then sent to the Transformer-DPT feature extraction module for feature extraction to obtain an intermediate feature map. The intermediate feature map is then processed by the Automatic Depth Estimation module for depth estimation and by the Bilinear Upsampling module for size restoration to obtain a pseudo-depth map of the teacher.

[0077] Specifically, the image automatic processor in the deep guided model L-DepthNet converts the input FRUIT-RGB dataset images into a format acceptable to the model, calls functions to load the Transformer-DPT deep model DepthAnything-V2 pre-trained model, performs automatic depth estimation for depth inference on the FRUIT-RGB dataset images, and the bilinear upsampling module Depth-Upsample upsamples the low-resolution depth map of the depth inference to restore the image size.

[0078] For the RGB apple image dataset, the depth-guided model used is the pre-trained Depth-Anything-V2 model. This model is trained on a large-scale cross-domain dataset and has strong general depth estimation capabilities. During training, unlabeled RGB images are input into the depth-guided model to generate pseudo-depth maps. The deep-guided model keeps its weights frozen throughout the process, using them only for knowledge distillation and not participating in parameter updates, such as... Figure 1 As shown below.

[0079] like Figure 4 As shown, the loss function of the monocular depth estimation model mainly consists of the BerHu loss function, the SILog loss function, and the gradient consistency loss function. Marginal constraint loss function and normal consistency loss function The weighted summation is performed according to the following formula:

[0080] in, Indicates depth of fruit stalk and calyx for identification depth (i.e., the predicted depth map output by the student model) and the actual depth The inverse Huber loss (i.e., the true depth image corresponding to the RGB image) Indicates depth of fruit stalk and calyx for depth recognition (i.e., the predicted depth map output by the student model) and depth-guided pseudo-depth Scale-invariant log loss of (teacher pseudo-depth map), Represents the gradient consistency loss function. Represents the edge constraint loss function. Represents the normal consistency loss function. The weight hyperparameters for the BerHu loss function, SILog loss function, gradient consistency loss function, edge constraint loss function, and normal consistency loss function are determined by the upstream task. The BerHu loss function is set according to the following formula:

[0081]

[0082]

[0083]

[0084]

[0085] in, Predicting depth maps for students i The absolute error of the depth value of each effective pixel. Predicting depth maps for students i The predicted depth value for each valid pixel. For the true depth image i The actual depth value of each effective pixel. This is the preset segmentation threshold, and 𝜏 is the correction coefficient, which defaults to 0.2. It is the set of valid pixels in the true depth image. For set The number of elements, when the error Not exceeding the threshold When using paradigm 1, if the error exceeds The transition is smooth within the range of paradigm 2, avoiding excessive sensitivity to large errors; The SILog loss function is set according to the following formula:

[0086]

[0087]

[0088]

[0089] in, For each pixel, predict the logarithmic difference between the depth and the target depth. The mean squared error is used to measure the variance or error strength of the overall prediction. The average error is used to measure the average deviation of the overall prediction. For the teacher pseudo-depth graph, the first i Depth value of each effective pixel, This is the preset numerical stability constant, which is 10 by default. -6 , The preset variance-mean tradeoff coefficient is set to 0.85. Gradient consistency loss function Set it according to the following formula:

[0090]

[0091]

[0092]

[0093]

[0094] in, and These represent the gradient consistency errors in the horizontal and vertical directions, respectively. and The student models predict depth values ​​for both horizontal and vertical directions, respectively. and These are the actual depth values ​​for the horizontal and vertical directions, respectively. and These are sets where adjacent pixels are valid both horizontally and vertically. and Sets and The number of pixel pairs in the middle These are the two-dimensional coordinates of pixels in the image. This indicates that the pixel is a valid pixel. For the logical AND operator; Marginal constraint loss function Set it according to the following formula:

[0095] in, and These are the edge magnitudes calculated using the Sobel operator from the student's predicted depth map and the actual depth image, respectively. This represents the absolute difference between the edge magnitude of the predicted depth map and the edge magnitude of the true depth map at the i-th pixel. Normal consistency loss function Set it according to the following formula:

[0096]

[0097] in, and Let be the predicted normal vector and the true normal vector at the i-th valid pixel, respectively. Let be the mask for the i-th valid pixel. It is the norm of the vector, i.e., the length of the vector.

[0098] In summary, this invention combines monocular depth estimation and knowledge distillation techniques to identify the three-dimensional geometric features of fruit stalks, calyxes, and their surrounding areas. First, a fruit RGB-to-depth pairing dataset is constructed. Then, the depth-guided model L-DepthNet calls the pre-trained model DepthAnything-V2 to generate pseudo-depth labels for unlabeled images. These pseudo-labels, along with the real depth labels, are used to train the fruit stalk and calyx depth recognition model SC-DepthNet. During the training phase, multiple loss functions, including BerHu, gradient, normal, edge, and SILog distillation, are used for joint optimization, gradually updating the SC-DepthNet weights until convergence. Finally, the trained model can be used for depth estimation of fruit RGB images, achieving robust separation of fruit stalks, calyxes, and surface defects.

[0099] This embodiment uses the Adam optimizer for training on a GPU platform. During training, supervised batches (FRUIT-RGB-D) and unlabeled batches (FRUIT-RGB) are randomly sampled cyclically to ensure consistent batch size each time. The predicted depth is obtained from the FRUIT-RGB-D dataset using the fruit stalk and calyx depth recognition model. And calculate the difference from the true depth value. BerHu , and Loss value; simultaneously, pseudo-depth of the FRUIT-RGB dataset is generated by the depth-guided model. And calculate the predicted depth With pseudo-depth The SILog loss value is calculated; backpropagation is performed under the condition of weight freeze in the depth-guided model to update the weights of the pedicel and calyx depth recognition model until the error of the validation set converges, thereby ensuring the best performance of the model.

[0100] The RGB color image of the fruit to be tested is input into the trained model. The image is inferred according to the training weights to generate the corresponding depth map. Then, the generated depth map is classified into two categories to obtain the classification results of the fruit stalk, calyx and other regions, and the classification confusion matrix is ​​obtained.

[0101] This embodiment compares the inference accuracy and inference time of other existing monocular depth estimation models. As shown in Table 1, the FRUIT-DepthNet model proposed in this invention achieves better depth estimation results. While maintaining similar accuracy to the depth-guided model DepthAnything-V2, it significantly reduces the average inference time per image, thus achieving network lightweighting.

[0102] Table 1. Comparison of accuracy and inference time between the present invention and existing depth estimation models.

[0103] This invention utilizes a deep-guided model to perform knowledge distillation on the pre-trained weights of DepthAnything-V2, guiding the lightweight fruit stalk and calyx depth recognition model SC-DepthNet to learn deep features. Simultaneously, a high-resolution texture branch and an edge-guided smoothing module are introduced into the SC-DepthNet model to enhance the extraction and segmentation accuracy of detailed parts such as fruit stalks and calyxes. Compared with existing technologies, this invention has significant advantages such as lightweight network structure, fast inference speed, good edge detail preservation, and more accurate 3D feature representation. It achieves robust separation of fruit stalks, calyxes, and surface defects, and can be widely applied in intelligent fruit grading, defect detection, and automated harvesting, possessing significant promotional and application value.

[0104] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation, characterized in that, The method includes the following steps: S1. Collect RGB images of fruits and corresponding true depth images of RGB images, and then register and label all RGB images to obtain a fruit RGB and depth paired dataset. S2. Construct a monocular depth estimation model. Input the fruit RGB and depth pairing dataset into the monocular depth estimation model for training to obtain a trained monocular depth estimation model. S3. Input the RGB image of the fruit to be tested into the trained monocular depth estimation model to obtain the predicted depth map of the fruit to be tested. S4. Determine the fruit stalk and calyx regions of the fruit to be tested based on the predicted depth map of the fruit to be tested.

2. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 1, characterized in that: The monocular depth estimation model includes a fruit stalk and calyx depth recognition model and a trained depth guidance model. The fruit stalk and calyx depth recognition model serves as the student model, and the trained depth guidance model serves as the teacher model. The RGB images of the fruit are input into the fruit stalk and calyx depth recognition model and the depth guidance model, respectively. The fruit stalk and calyx depth recognition model outputs the student predicted depth map, and the depth guidance model outputs the teacher pseudo depth map. The student predicted depth map, the teacher pseudo depth map, and the real depth image corresponding to the RGB image are input into the loss function to calculate the distillation loss, and then backpropagation is used to update the fruit stalk and calyx depth recognition model in the monocular depth estimation model.

3. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 2, characterized in that: The fruit stalk and calyx depth recognition model includes three parallel branches, an edge-guided smoothing module, and a depth prediction head. The three parallel branches are an edge feature extraction branch, a high-resolution texture branch, and an encoder-decoder branch. The RGB image of the fruit is input into the edge feature extraction branch, the high-resolution texture branch, and the encoder-decoder branch, respectively, and then outputs the edge amplitude map, the high-resolution feature map, and the multi-scale decoding feature map, respectively. Subsequently, the high-resolution feature map and the multi-scale decoding feature map are fused to obtain a fused feature map. Then, the fused feature map and the edge amplitude map are input together into the edge guided smoothing module for feature smoothing processing, and output a smoothed feature map. Finally, the smoothed feature map is input into the depth prediction head, and the depth prediction head outputs the student predicted depth map.

4. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 3, characterized in that: The edge feature extraction branch includes a grayscale conversion operation, a Sobel convolution operation unit, and an edge magnitude calculation unit. After the RGB image of the fruit is input to the edge feature extraction branch, the RGB image is first converted to a grayscale image through a grayscale conversion operation. Then, the grayscale image is input to the Sobel convolution operation unit, and horizontal and vertical Sobel convolution operations are performed to obtain horizontal gradient maps and vertical gradient maps, respectively. Finally, the edge magnitude calculation unit calculates the edge magnitude map based on the horizontal gradient map and the vertical gradient map.

5. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 3, characterized in that: The encoder-decoder branch includes an encoder and a decoder. The RGB image of the fruit is input into the encoder for multi-scale feature extraction, and outputs feature maps of five different scales. The decoder includes four levels of upsampling blocks connected in sequence. The upsampling block upsamples the output of the previous level upsampling block and concatenates and fuses it with the feature map of the corresponding scale in the encoder. The fused feature is then processed by convolution and used as the output of the current level upsampling block. The output of the last level upsampling block is used as the multi-scale decoded feature map.

6. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 3, characterized in that: The edge-guided smoothing module processes the edge according to the following formula: ； ； ； in, Represents a smooth feature map. Represents the fused feature map. Calculate the weighting coefficients for the gating mechanism. Represents the edge amplitude map. This represents a smoothed intermediate feature map. It is the sigmoid activation function. The preset mean convolution kernel channel weights, The preset mean convolution kernel, This represents the bias term in the gating mechanism.

7. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 2, characterized in that: The depth-guided model includes an image automatic processor, a Transformer-DPT feature extraction module, an automatic depth estimation module, and a bilinear upsampling module connected in sequence. The RGB image of the fruit is input to the image automatic processor for formatting and then sent to the Transformer-DPT feature extraction module for feature extraction to obtain an intermediate feature map. The intermediate feature map is then processed by the automatic depth estimation module for depth estimation and the bilinear upsampling module for size restoration to obtain a pseudo-depth map of the teacher.

8. The method for recognizing three-dimensional features of fruit pedicels and calyxes based on monocular depth estimation according to claim 1, characterized in that: The loss function of the monocular depth estimation model is mainly composed of a weighted sum of the BerHu loss function, SILog loss function, gradient consistency loss function, edge constraint loss function, and normal consistency loss function, and is specifically set according to the following formula: ； in, Indicates depth of fruit stalk and calyx for depth recognition With true depth Anti-Huber losses Indicates depth of fruit stalk and calyx for depth recognition Compared to the scale-invariant log loss for depth-guided pseudo-depth, Represents the gradient consistency loss function. Represents the edge constraint loss function. Represents the normal consistency loss function. These are the weight hyperparameters for the BerHu loss function, SILog loss function, gradient consistency loss function, edge constraint loss function, and normal consistency loss function, respectively. The BerHu loss function is set according to the following formula: ； ； ； ； ； in, Predicting depth maps for students i The absolute error of the depth value of each effective pixel. Predicting depth maps for students i The predicted depth value for each valid pixel. For the true depth image i The actual depth value of each effective pixel. Here, is the preset segmentation threshold, and is the correction coefficient. It is the set of valid pixels in the true depth image. For set The number of elements; The SILog loss function is set according to the following formula: ； ； ； ； in, For each pixel, predict the logarithmic difference between the depth and the target depth. The mean squared error is used to measure the variance or error strength of the overall prediction. The average error is used to measure the average deviation of the overall prediction. For the teacher pseudo-depth graph, the first i Depth value of each effective pixel, The preset numerical stability constant, This is the preset variance-mean compromise factor; The gradient consistency loss function is set according to the following formula: ； ； ； ； ； in, and These represent the gradient consistency errors in the horizontal and vertical directions, respectively. and The student models predict depth values ​​for both horizontal and vertical directions, respectively. and These are the actual depth values ​​for the horizontal and vertical directions, respectively. and These are sets where adjacent pixels are valid both horizontally and vertically. and Sets and The number of pixel pairs in the middle These are the two-dimensional coordinates of pixels in the image. This indicates that the pixel is a valid pixel. For logical AND operator; The edge constraint loss function is set according to the following formula: ； in, and These are the edge magnitudes calculated using the Sobel operator from the student's predicted depth map and the actual depth image, respectively. The normal consistency loss function is set according to the following formula: ； ； in, and Let be the predicted normal vector and the true normal vector at the i-th valid pixel, respectively. Let be the mask for the i-th valid pixel. Let be the norm of the vector.

Images