Apple stem and calyx identification method based on binocular vision multi-modal fusion

Abstract

The invention discloses an apple stem and calyx identification method based on binocular vision multi-mode fusion, which comprises the following steps: synchronously acquiring RGB images of a left view angle and a right view angle of an apple, calculating a disparity map according to the RGB images of the left view angle and the right view angle and generating a depth image, and carrying out space calibration and pixel-level registration on the RGB images and the depth image; performing color normalization, reflection suppression and brightness compensation processing on the registered RGB image, and performing hole filling and filtering smoothing processing on the registered depth image; performing multi-scale fusion on the processed RGB image and the depth image, and constructing multi-modal input data; and inputting the multi-modal input data into a semantic segmentation network based on an encoder-decoder architecture, and outputting a pixel-level semantic segmentation mask of the stem and the calyx based on the semantic segmentation network to obtain an identification result. According to the method, the problem of confusion of fruit stems / calyx and fruit defects is effectively avoided, so that the recognition precision is further improved.

Description

Technical Field

[0001] This invention belongs to the field of agricultural intelligent sensing and computer vision technology, and in particular relates to a method for recognizing apple stems and calyxes based on binocular vision multimodal fusion. Background Technology

[0002] Apple stalks and calyxes are key morphological features of the fruit, and their accurate identification and localization are core technologies for agricultural intelligent equipment systems such as fruit posture estimation, automated harvesting, quality grading, and pest and disease detection. In recent years, with the development of computer vision technology, stalk and calyx recognition methods have evolved from traditional image processing to deep learning. Early methods mainly relied on manually designed color thresholds, edge gradients, texture features, and morphological operations to achieve region segmentation, while current mainstream technologies have shifted to semantic segmentation models based on convolutional neural networks, such as U-Net and the DeepLab series, achieving pixel-level prediction through encoder-decoder structures. Meanwhile, the introduction of 3D vision technology has provided a new way to overcome the limitations of single RGB modality. Multimodal perception schemes based on binocular vision or RGB-D cameras can simultaneously acquire the color texture and geometric depth information of the fruit, laying a data foundation for accurate recognition in complex scenes. Some studies have further explored multimodal feature fusion strategies and attempted to introduce channel attention or spatial attention mechanisms to enhance the saliency of target regions, initially improving segmentation accuracy.

[0003] Despite some progress in existing technologies, numerous challenges remain in practical agricultural applications. First, the fruit stalk and calyx regions occupy a very small portion of the overall image. Standard U-Net is prone to detail loss during feature extraction due to downsampling, causing foreground features to be overwhelmed by complex fruit peel textures and background information, resulting in blurred segmentation boundaries or false positives and false negatives. Second, existing RGB-D fusion methods are mostly limited to simple channel splicing or fixed weighting, failing to fully exploit the semantic correlation and complementarity between color and depth modalities. This leads to redundant feature representations and unstable fusion results. Furthermore, data gaps and noise caused by illumination reflection, stereo matching errors, and surface occlusion, which are prevalent in depth maps, will directly reduce network discrimination performance if not effectively corrected. In addition, the strong light reflection, uneven fruit color, local occlusion, and morphological differences between varieties in the natural environment of orchards make methods relying solely on visual or geometric information insufficiently robust, making it difficult to maintain stable recognition accuracy under complex lighting and varying postures. These problems severely restrict the practical deployment of existing technologies in real-time agricultural robot operations and high-precision sorting systems. Summary of the Invention

[0004] This invention aims to overcome the problems of low segmentation accuracy, sensitivity to illumination and surface reflection, insufficient feature discrimination, and difficulty in recognizing small targets in existing apple stem and calyx recognition methods. It provides an image segmentation method for apple stems and calyxes based on binocular vision multimodal fusion and an attention mechanism. This method fully utilizes the multimodal image information (RGB and depth maps) acquired by a binocular camera, combined with an improved deep learning network structure, to achieve high-precision segmentation of the apple stem and calyx regions. This provides an accurate visual foundation for subsequent tasks such as fruit pose recognition, defect detection, and automated sorting.

[0005] To achieve the above objectives, this invention provides a method for recognizing apple pedicels and calyxes based on binocular vision multimodal fusion, comprising:

[0006] Simultaneously acquire RGB images of the apple from the left and right perspectives, calculate the disparity map and generate a depth image based on the RGB images from the left and right perspectives, and perform spatial calibration and pixel-level registration on the RGB images and the depth image;

[0007] The registered RGB image is processed for color normalization, reflection suppression and brightness compensation, and the registered depth image is processed for hole filling and filtering smoothing.

[0008] The processed RGB image and depth image are fused at multiple scales to construct multimodal input data;

[0009] The multimodal input data is input into a semantic segmentation network based on an encoder-decoder architecture. The semantic segmentation network outputs pixel-level semantic segmentation masks of the pedicel and calyx to obtain the recognition result.

[0010] Preferably, the process of simultaneously acquiring RGB images of the apple from both left and right perspectives includes:

[0011] Simultaneously acquire left-view RGB images and right-view RGB images with a resolution of 1920×1080 using the ZED 2i binocular vision camera.

[0012] Preferably, the process of calculating the disparity map and generating the depth image based on the RGB images of the left and right viewpoints includes:

[0013] Based on the RGB images from the left and right perspectives, a disparity map is calculated using a stereo matching algorithm;

[0014] The disparity map is converted into a depth image according to the binocular geometric formula;

[0015] The depth image is smoothed by bilateral filtering, and holes in the depth image are filled by distance-weighted interpolation to obtain a filtered and smoothed depth image.

[0016] Preferably, the process of performing color normalization, reflection suppression, and brightness compensation on the registered RGB image includes:

[0017] The registered RGB image is subjected to Gamma correction and normalization. The mean and standard deviation of each channel are calculated for standardization. The brightness threshold is detected and the highlight area is interpolated based on the surrounding pixels to obtain the RGB image after color normalization, reflection suppression and brightness compensation.

[0018] Preferably, the process of constructing multimodal input data includes:

[0019] A multi-scale pyramid is constructed from the processed RGB image and depth image. RGB features and depth features are extracted at each scale. The RGB feature weights and depth feature weights are dynamically calculated by the modality adaptive fusion module. The RGB features and depth features are then weighted and fused according to the RGB feature weights and depth feature weights to obtain multimodal input data.

[0020] Preferably, the semantic segmentation network based on the encoder-decoder architecture is AttentionUNet, a deep segmentation network improved based on the U-Net structure;

[0021] The improvements include:

[0022] Channel attention, spatial attention, and feature gating mechanisms are introduced into the network to enhance the feature responses of the fruit stalk and calyx regions; and a hybrid loss function is used for network training.

[0023] Features are extracted through convolutional blocks during the encoding stage, and the resolution is restored through upsampling and convolutional blocks during the decoding stage. Features are passed through skip connections, and a pixel-level semantic segmentation mask for the fruit stalk and calyx is output.

[0024] Preferably, the channel attention mechanism obtains channel descriptors through global average pooling, calculates channel weights through fully connected layers and activation functions, and performs channel-level recalibration on the input features based on the channel weights to obtain features enhanced by channel attention.

[0025] Preferably, the spatial attention mechanism performs average pooling and max pooling along the channel dimension, concatenates the pooling results along the channel dimension, and generates a spatial attention map through convolution. The input features are then spatially weighted according to the spatial attention map to obtain features enhanced with spatial attention.

[0026] Preferably, the feature gating mechanism generates attention weights based on high-level features in the skip connection, selectively weights low-level features based on the attention weights, and fuses the weighted low-level features with the decoded features to obtain the fused features after feature gating.

[0027] Preferably, the hybrid loss function is obtained by weighting Dice loss and cross-entropy loss, calculating a weighted loss value based on the Dice loss and cross-entropy loss, optimizing the network parameters based on the weighted loss value, and outputting a pixel-level semantic segmentation mask for the fruit stalk and calyx.

[0028] Compared with the prior art, the present invention has the following advantages and technical effects:

[0029] The multimodal fusion and attention enhancement scheme proposed in this invention effectively solves the aforementioned technical problems. Through an improved AttentionUNet network with cascaded channel attention, spatial attention, and feature gating mechanisms at each encoding-decoding layer, the model can adaptively focus on small regions of the fruit stalk and calyx, significantly improving the feature representation ability of small targets and background suppression. A modal adaptive fusion module dynamically optimizes the weight allocation of RGB and depth features, and combined with depth map hole filling and smoothing filtering preprocessing, fully exploits the complementary advantages of color and geometric information, enhancing the stability of cross-modal semantic association. The introduction of a hybrid loss function of Dice and cross-entropy optimizes the training process, further improving boundary continuity. Experiments show that this method achieves high-precision pixel-level segmentation and 3D spatial localization of the fruit stalk and calyx under complex conditions such as strong light reflection, surface defects, and partial occlusion. It possesses excellent robustness and generalization ability, meeting the real-time computing requirements of embedded platforms and providing reliable technical support for automated apple harvesting and quality grading.

[0030] This invention proposes a method for recognizing apple stems and calyxes based on binocular vision multimodal fusion. It integrates color information (RGB), depth information, and geometric morphological features (concavity), with different information sources complementing each other. This effectively overcomes the shortcomings of single information sources, which are susceptible to changes in lighting, fruit occlusion, and interference from similar objects, enabling the detection system to maintain stable recognition performance even in complex environments. This invention utilizes a high-precision depth camera and 3D point cloud processing technology to achieve millimeter-level spatial positioning of the stems and calyxes. This method provides directly usable operational coordinates for robotic arms, significantly improving the accuracy of actions during automated harvesting and grading. Furthermore, this invention has good practicality and adaptability. Its overall method flow is clear and can be directly deployed on existing harvesting robot platforms and fruit and vegetable sorting lines, with a wide range of applications and broad market prospects. Combining the complementary characteristics of concavity analysis and semantic segmentation, this invention effectively avoids the confusion between stems / calyxes and fruit defects, thereby further improving recognition accuracy. Attached Figure Description

[0031] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the structure of the apple pedicel / calyx recognition model according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the SEBlock module according to an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the AttentionGate module in an embodiment of the present invention. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0038] like Figure 1 As shown, this embodiment provides a method for recognizing apple stems and calyxes based on binocular vision multimodal fusion, including:

[0039] Simultaneously acquire RGB images of the apple from the left and right perspectives, calculate the disparity map based on the RGB images from the left and right perspectives and generate a depth image, and perform spatial calibration and pixel-level registration on the RGB images and the depth image.

[0040] The registered RGB image is processed for color normalization, reflection suppression and brightness compensation, and the registered depth image is processed for hole filling and filtering smoothing.

[0041] The processed RGB image and depth image are fused at multiple scales to construct multimodal input data;

[0042] Multimodal input data is fed into a semantic segmentation network based on an encoder-decoder architecture. The semantic segmentation network outputs pixel-level semantic segmentation masks of the fruit stalk and calyx to obtain the recognition results.

[0043] Furthermore, the process of simultaneously acquiring RGB images of the apple from both left and right perspectives includes:

[0044] Simultaneously acquire left-view RGB images and right-view RGB images with a resolution of 1920×1080 using the ZED 2i binocular vision camera.

[0045] Furthermore, this embodiment uses a Zed 2i binocular vision camera to simultaneously acquire RGB images of the apple from the left and right eyes, denoted as follows: and The initial resolution of the acquired image is set to The image is then scaled to a suitable size as needed and used as network input. The camera's intrinsic parameter matrix is ​​obtained during the calibration phase. With external references Internal references are represented as:

[0046]

[0047] in, This is the focal length in pixels. The coordinates of the main point.

[0048] Furthermore, the process of calculating the disparity map and generating the depth image based on the RGB images from the left and right perspectives includes:

[0049] Based on the RGB images from the left and right perspectives, a disparity map is calculated using a stereo matching algorithm;

[0050] Convert the disparity map into a depth image based on the binocular geometric formula;

[0051] The depth image is smoothed by bilateral filtering, and the holes in the depth image are filled by distance-weighted interpolation to obtain the filtered and smoothed depth image.

[0052] Furthermore, this embodiment obtains a disparity map based on stereo matching of the left and right views. The disparity optimization algorithm built into ZEDSDK is used, and the relationship between disparity and depth satisfies the binocular geometry formula:

[0053]

[0054] To effectively suppress depth noise and holes, this embodiment first performs sub-pixel fitting on the disparity map, and then performs sub-pixel fitting on the original depth map. Bilateral filtering is performed, and its discrete implementation is as follows:

[0055]

[0056] Spatial weight Amplitude weight , This is the filtering window.

[0057] Then, a distance-weighted interpolation method is used to fill the holes in the pixels:

[0058]

[0059] in, The effective depth pixel geometry within the window. This is the attenuation coefficient.

[0060] Furthermore, the process of performing color normalization, reflection suppression, and brightness compensation on the registered RGB image includes:

[0061] The registered RGB image is subjected to Gamma correction and normalization. The mean and standard deviation of each channel are calculated for standardization. The brightness threshold is detected and the highlight area is interpolated based on the surrounding pixels to obtain the RGB image after color normalization, reflection suppression and brightness compensation.

[0062] Furthermore, this embodiment performs color normalization, reflective area suppression, and brightness compensation on the acquired RGB images; and performs hole filling, edge smoothing, and noise filtering on the depth images. The processed RGB images and depth images are then fused at multiple scales to form a four-channel multimodal input (R, G, B, D) to fully preserve the fruit surface texture and spatial structure features, thereby obtaining a multimodal image dataset containing color and geometric information.

[0063] Furthermore, this embodiment preprocesses the RGB image and depth image, first performing Gamma correction and normalization to reduce the effect of highlights:

[0064]

[0065] Then the mean difference for each channel is calculated. with standard deviation Standardize:

[0066]

[0067] To remove highlights, this embodiment detects a brightness threshold. The three-channel RGB and single-channel depth are then interpolated for these pixels based on the differences between surrounding pixels. After processing, the three-channel RGB and single-channel depth are stacked channel-wise to form a four-channel input tensor. .

[0068] Furthermore, the process of constructing multimodal input data includes:

[0069] A multi-scale pyramid is constructed from the processed RGB and depth images. RGB and depth features are extracted at each scale. The RGB and depth feature weights are dynamically calculated by the modality adaptive fusion module. The RGB and depth features are then weighted and fused according to the RGB and depth feature weights to obtain multimodal input data.

[0070] Furthermore, this embodiment employs a multi-scale fusion strategy and an improved AttentionUNet model.

[0071] Specifically, a scale pyramid is constructed from the input image. Each scale enters the corresponding encoder branch. The encoder consists of several ConvBlocks, and each ConvBlock can be represented as:

[0072]

[0073] in For convolution kernel, Indicates batch normalization, Represents ReLU. The number of base channels is... Starting with 64, 128, 256, and 512 layers, the increments are multiplied layer by layer (64, 128, 256, 512). Each layer in the encoding stage can perform separate convolutions on the RGB and depth channels in parallel to extract features. Then, at each layer, the mixture is fused using a Modality Adaptive Fusion Module (MAFM):

[0074]

[0075]

[0076]

[0077] in, Indicates global average pooling. These are the parameters for the fully connected layer. The activation function is ReLU. For the sigmoid function. MAFM is designed to allow the model to dynamically adjust the fusion weight of RGB and depth at each scale, thus adaptively relying on a more reliable modality in scenes with varying lighting or depth noise.

[0078] Furthermore, the semantic segmentation network based on the encoder-decoder architecture is AttentionUNet, a deep segmentation network improved based on the U-Net structure;

[0079] like Figure 2 As shown, the improvements include:

[0080] Channel attention, spatial attention, and feature gating mechanisms are introduced into the network to enhance the feature responses of the fruit stalk and calyx regions; and a hybrid loss function is used for network training.

[0081] Features are extracted through convolutional blocks during the encoding stage, and the resolution is restored through upsampling and convolutional blocks during the decoding stage. Features are passed through skip connections, and a pixel-level semantic segmentation mask for the fruit stalk and calyx is output.

[0082] Furthermore, in this embodiment, the RGB and Depth images are fused at multiple scales and then input into the improved AttentionUNet semantic segmentation network;

[0083] Channel attention module, spatial attention module and feature gating module are introduced into the network to enhance the salient features of the fruit stalk and calyx regions;

[0084] A hybrid loss function is used for training, and the semantic segmentation mask of the fruit stalk and calyx is output.

[0085] Specifically, such as Figure 3 and Figure 4 As shown, this embodiment uses an improved deep segmentation network, AttentionUNet, based on the U-Net structure to perform pixel-level semantic segmentation of the apple stem and calyx regions. This network, based on the U-Net encoder-decoder structure, introduces the following features at each layer:

[0086] (1) Channel attention mechanism (SE module), which improves the response of salient features through adaptive channel weighting;

[0087] (2) Spatial attention mechanism (CBAM module), which suppresses background noise by enhancing local salient regions;

[0088] (3) Feature gating mechanism (Attention Gate): In skip connections, low-level features are selectively filtered based on high-level features to ensure the effectiveness of feature fusion. The above improvements enable the network to focus on the fruit stalk and calyx regions, reduce background interference, and improve the model's ability to perceive small targets.

[0089] Furthermore, the channel attention mechanism obtains channel descriptors through global average pooling, calculates channel weights through fully connected layers and activation functions, and performs channel-level recalibration on the input features based on the channel weights to obtain features enhanced by channel attention.

[0090] Furthermore, the channel attention module described in this embodiment calculates channel weights using global average pooling. And normalized using the Sigmoid function:

[0091]

[0092] in As input features, For global average pooling, for function.

[0093] Furthermore, the spatial attention mechanism performs average pooling and max pooling along the channel dimension, concatenates the pooling results along the channel dimension, and generates a spatial attention map through convolution. The input features are then spatially weighted based on the spatial attention map to obtain the spatial attention-enhanced features.

[0094] Furthermore, the spatial attention module described in this embodiment generates a salient weight map through convolution operations:

[0095]

[0096] in for Convolution kernel, and For averaging and max pooling operations.

[0097] Furthermore, the feature gating mechanism generates attention weights based on high-level features in the skip connection, selectively weights low-level features based on the attention weights, and fuses the weighted low-level features with the decoded features to obtain the fused features after feature gating.

[0098] Furthermore, the Attention Gate module described in this embodiment, in skip connections, is based on high-level features... For low-level features Weighting:

[0099]

[0100]

[0101] in, This is the attention weight coefficient. This is element-wise multiplication.

[0102] As an additional implementation, this embodiment introduces multiple attention modules at skip connections and in the decoder block of the AttentionUNet constructed based on fused features. Channel attention uses the SE module, assuming the fused features of a certain layer are... The channel descriptor is The update channel features are Spatial attention employs the spatial branching method in CBAM, first taking the average and maximum projection along the channel dimension to obtain two-dimensional features. splicing them together in the channel dimension and using convolution Generate a spatial attention map:

[0103]

[0104] The final feature is Attention gates are used for selective propagation of skip links, assuming stratigraphic features. With decoding guidance features go through After the convolution transformation is performed to obtain the same number of channels, the gating weights are calculated as follows:

[0105]

[0106] final The multiple attention modules mentioned above can be jointly optimized through backpropagation during training, enabling the model to optimize target-related responses in a learnable manner across the channel and spatial dimensions, while suppressing misleading background responses such as fruit peel texture.

[0107] The model decoder uses a combination of upsampling (transposed convolution) and ConvBlock. Let the upsampling operation be... Then a certain decoding layer can be described as:

[0108]

[0109] in This indicates channel splicing. The final output layer uses... Convolution maps the number of channels to the number of classes (3 classes in this example), and generates a pixel-level probability map using Softmax. Softmax is defined as:

[0110]

[0111] Furthermore, the hybrid loss function is obtained by weighting the Dice loss and cross-entropy loss. The weighted loss value is calculated based on the Dice loss and cross-entropy loss, and the network parameters are optimized based on the weighted loss value to output a pixel-level semantic segmentation mask for the fruit stalk and calyx.

[0112] The hybrid loss function is composed of a weighted average of the Dice loss and the cross-entropy loss, and its specific expression is as follows:

[0113]

[0114] in,

[0115]

[0116] Training employs a hybrid loss function to balance overall accuracy and small target recognition capability. The loss function takes the following form:

[0117]

[0118] The cross-entropy loss is:

[0119]

[0120] Dice loss takes the following form:

[0121]

[0122] This embodiment uses experimental parameter tuning to obtain... At that time, it significantly improved IoU and Recall for small targets on the fruit stalk. The optimizer used was Adam, and the initial learning rate was... Weight decay The training batch size is Training rounds .

[0123] The learning rate is adjusted using a cosine annealing strategy to achieve better convergence.

[0124]

[0125] in This represents the current iteration step. This represents the total number of steps.

[0126] After model training is completed, the probability graph is first thresholded during the inference phase:

[0127]

[0128] Subsequently Perform morphological closing operations to fill small holes and remove noise:

[0129]

[0130] structural elements Pick .

[0131] To improve the smoothness of the mask edges, this embodiment uses a post-processing step based on Conditional Random Field (CRF) to further refine the boundaries. The energy minimization form of CRF is as follows:

[0132]

[0133] in The negative logarithmic probability is at the pixel level. This is the smoothing term for pixel pairs.

[0134] This embodiment uses segmentation metrics including Intersection over Union (IoU) and Accuracy to evaluate the performance of the method. The formula for IoU is:

[0135]

[0136] The output of the method is a mask segmentation map, which distinguishes the fruit stalk region and the calyx region by different colors, and is used for subsequent fruit position detection and fruit shape analysis.

[0137] As an additional implementation method, the method in this embodiment is lightweight and computationally efficient, capable of running in real time on embedded platforms. It is suitable for agricultural intelligent equipment systems such as automatic detection of apple pedicels and calyxes, fruit position recognition, intelligent harvesting, and quality grading. Specifically, it is applied to automated vision systems for fruit grading, fruit surface defect detection, and fruit three-dimensional pose estimation.

[0138] This embodiment proposes a multimodal apple stem and calyx segmentation method combining binocular vision and deep learning attention mechanisms. This method utilizes a ZED 2i binocular camera to acquire synchronized RGB and depth map information, and employs an improved AttentionUNet network for feature fusion and segmentation. It comprehensively utilizes channel, spatial, and hierarchical attention mechanisms to achieve significant region enhancement. The learning process for small targets is optimized using a hybrid loss function of Dice and cross-entropy, enabling the model to maintain excellent segmentation accuracy and robustness under complex lighting, surface reflection, and partial occlusion conditions. Compared to existing single-modal or shallow fusion methods, this embodiment is more targeted and practical in its structural design, providing a high-precision, real-time visual solution for intelligent fruit detection, automatic harvesting, and quality assessment.

[0139] To further optimize the technical solution, two sets of comparative experiments were conducted on actual sample data using the apple stalk and calyx segmentation method based on binocular vision multimodal fusion proposed in this embodiment. These experiments comprehensively verified the superiority of the network structure and multimodal fusion strategy of this invention. One set of experiments used only RGB images as input to train the segmentation network, while the other set used both RGB and depth images for joint training to evaluate the gain effect of depth information in extracting salient regions of the stalk and calyx.

[0140] The quantitative indicators of the comparative experiments further demonstrate the technical advantages of this invention: The RGB training results are shown in Table 1. When using only RGB images for segmentation network testing, the improved AttentionUNet network achieves an mIoU of 0.8726, outperforming other networks. The RGB+Depth training results are shown in Table 2. After adding depth information, the geometric constraints of the Depth image and the texture information of the RGB image complement each other in the multi-scale fusion module. This allows the network to more accurately capture the shape features and local concave structures of the fruit stalk and calyx under the synergistic effect of channel attention, spatial attention, and feature gating mechanisms, significantly improving the feature salience and edge consistency of the target region. The RGB+Depth input model shows significant improvements over the RGB single-modal model in key indicators such as mIoU and accuracy. Especially in samples with small fruit stalk and calyx areas, the Depth information effectively alleviates the training instability caused by the weakening of small target features. These visualization results demonstrate that the proposed AttentionUNet improved structure can fully utilize depth information to compensate for illumination variations and color interference in RGB images, thereby achieving more stable and robust segmentation performance of fruit stalks and calyxes.

[0141] Table 1

[0142] mIoU Pixel Acc SegNet (RGB) 0.8441 0.9986 DeepLabV3+ (RGB) 0.8394 0.9984 Unet (RGB) 0.8599 0.9986 AttentionUNet (RGB) 0.8726 0.9985

[0143] Table 2

[0144] mIoU Pixel Acc SegNet (RGB+Depth) 0.8513 0.9984 DeepLabV3+ (RGB+Depth) 0.8458 0.9984 Unet (RGB+Depth) 0.8667 0.9985 AttentionUNet (RGB+Depth) 0.8745 0.9986

[0145] Therefore, this embodiment not only improves the network's perception of target areas through structural improvements and attention enhancement mechanisms, but also further improves the model's recognition accuracy in complex scenes by introducing binocular depth information. The experimental results and visualization mask fully verify the technological advancement and practical value of this invention in the intelligent segmentation of agricultural fruits.

[0146] The method described in this embodiment has significant advantages in multimodal information fusion: by fusing RGB and depth map features, it effectively overcomes the problem of single-modality sensitivity to changes in illumination, reflection, and surface texture, and enhances the robustness of the model in complex natural environments.

[0147] The method described in this embodiment significantly improves the small target segmentation performance: by introducing a dual attention mechanism, the model can more accurately focus on the fruit stalk and calyx regions, reducing missed detections and false detections.

[0148] The method described in this embodiment has a reasonable fusion structure design: the modal adaptive fusion module can dynamically adjust the feature weights of different modalities, avoid feature redundancy and information conflict, and ensure the effectiveness of fused features.

[0149] The method described in this embodiment optimizes the loss function to improve boundary accuracy: the hybrid loss function of Dice and cross-entropy significantly improves the continuity and integrity of the boundary contour while ensuring the overall segmentation accuracy.

[0150] The method described in this embodiment is highly versatile: it does not rely on specific hardware devices and can achieve high-precision segmentation based solely on image data, making it suitable for identifying the pedicels and calyxes of other fruits (such as pears, peaches, and citrus).

[0151] In summary, the apple pedicel and calyx segmentation method based on binocular vision multimodal fusion and attention mechanism proposed in this embodiment can achieve high robustness and high accuracy in image segmentation under complex environments, and has good scalability and practical application value.

[0152] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying apple pedicel and calyx based on binocular vision multi-modal fusion, characterized in that, The method comprises the following steps: Synchronously collecting RGB images of apples from left and right perspectives, calculating a parallax map from the RGB images of the left and right perspectives, and generating a depth image, spatially calibrating and pixel-level registering the RGB images and the depth image; Performing color normalization, reflection suppression, and brightness compensation processing on the registered RGB images, and performing hole filling and filter smoothing processing on the registered depth image; Fusing the processed RGB images and depth image at multiple scales to construct multi-modal input data; Inputting the multi-modal input data into a semantic segmentation network based on an encoder-decoder architecture, outputting a pixel-level semantic segmentation mask of pedicel and calyx based on the semantic segmentation network, and obtaining a recognition result.

2. The method of claim 1, wherein the process of synchronously collecting RGB images of apples from left and right perspectives comprises: Synchronously collecting left-perspective and right-perspective RGB images with a resolution of 1920x1080 through a ZED 2i binocular vision camera.

3. The method of claim 1, wherein the process of calculating a parallax map from the RGB images of the left and right perspectives and generating a depth image comprises: Calculating a parallax map from the RGB images of the left and right perspectives through a stereo matching algorithm; Converting the parallax map into a depth image according to a binocular geometry formula; Performing smoothing processing on the depth image through bilateral filtering, and filling holes in the depth image through distance-weighted interpolation, to obtain a depth image that has been smoothed and filtered.

4. The method of claim 1, wherein the process of performing color normalization, reflection suppression, and brightness compensation processing on the registered RGB images comprises: Performing Gamma correction and normalization processing on the registered RGB images, normalizing each channel by calculating the mean and standard deviation, detecting a brightness threshold, and performing interpolation compensation on highlight areas based on surrounding pixels, to obtain RGB images that have been color-normalized, reflection-suppressed, and brightness-compensated.

5. The method of claim 1, wherein the process of constructing multi-modal input data comprises: Constructing a multi-scale pyramid for the processed RGB images and depth image, extracting RGB features and depth features at each scale, dynamically calculating RGB feature weights and depth feature weights through a modal adaptive fusion module, and performing weighted fusion of the RGB features and depth features according to the RGB feature weights and depth feature weights, to obtain multi-modal input data.

6. The method of claim 1, wherein the semantic segmentation network based on an encoder-decoder architecture is a deep segmentation network AttentionUNet improved based on a U-Net structure; The improvements include: Introducing a channel attention mechanism, a spatial attention mechanism, and a feature gating mechanism into the network to enhance the feature response of the pedicel and calyx regions; and using a hybrid loss function for network training. ​ ​ ​ ​ ​ In the encoding stage, the features are extracted by convolution blocks, in the decoding stage, the resolution is recovered by upsampling and convolution blocks, the features are transmitted in the skip connection, and the pixel-level semantic segmentation mask of the fruit stalk and calyx is output.

7. The method of claim 6, wherein, The channel attention mechanism obtains channel descriptors through global average pooling, calculates channel weights through a fully connected layer and an activation function, performs channel-level re-labeling on the input features according to the channel weights, and obtains channel attention enhanced features.

8. The method of claim 6, wherein, The spatial attention mechanism performs average pooling and maximum pooling along the channel dimension, concatenates the pooled results in the channel dimension, generates a spatial attention map through convolution operation, performs spatial weighting on the input features according to the spatial attention map, and obtains spatial attention enhanced features.

9. The method of claim 6, wherein, The feature gating mechanism generates attention weights according to high-level features in the skip connection, selectively weights low-level features according to the attention weights, fuses the weighted low-level features with the decoding features, and obtains feature-gated fusion features.

10. The method of claim 6, wherein, The hybrid loss function is obtained by weighting the Dice loss and the cross-entropy loss, the weighted loss value is calculated according to the Dice loss and the cross-entropy loss, the network parameters are optimized according to the weighted loss value, and the pixel-level semantic segmentation mask of the fruit stalk and calyx is output.

Images