Dual-task-driven fruit stem identification and picking point positioning correction method and system

Abstract

The invention relates to a double-task-driven fruit stem identification and picking point positioning correction method and system. The method comprises the following steps: constructing a fruit stem identification and positioning correction model based on a double-task mechanism; in the first-stage task, an input image is processed through another fruit stem identification network StemNet, and fruit stem and fruit area identification and picking key point positioning are carried out; in the second-stage task, fine segmentation of the fruit stem boundary is carried out on the fruit stem area output by the first-stage task through StemNet, and an accurate fruit stem segmentation area is obtained; then, combining the picking point position information obtained by the first-stage task and the accurate fruit stem segmentation area obtained by the second-stage task to check and correct the spatial position of the picking point; the StemNet takes YOLOv12 as a basic model, a Conv module is replaced by an HAWConv module, and an A2C2f module is replaced by a SteAttn module; and performing model training for image detection to obtain a fruit target and accurate picking point position information. According to the invention, the accuracy and robustness of fruit picking point positioning can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent agricultural technology, specifically to a dual-task driven method and system for fruit stem recognition and picking point positioning correction suitable for automated harvesting robots. Background Technology

[0002] The automation of chili pepper harvesting is of great significance for improving the level of agricultural intelligence. However, in actual harvesting, the precise positioning of the harvesting point is challenged due to factors such as the slender shape of chili pepper stems, overlap and occlusion between fruits and stems, and interference from adjacent stems. Currently used harvesting point positioning models are mostly based on object detection and keypoint prediction algorithms, using feature extraction to infer the stem region and predict the harvesting location. In complex environments, stems are often occluded, and while posture detection-based methods can still infer the relative position of the harvesting point from the occluded object, the slender shape of the stems, coupled with their drooping posture under gravity, results in high tolerance for errors in the longitudinal direction of the stem prediction, but extremely low tolerance for errors in the lateral direction. Therefore, although existing posture models perform well in overall accuracy, incorrect harvesting point estimation can lead to harvesting failure by the robotic arm. The harvesting point is still prone to deviating laterally from the stem, causing the robotic arm to perform harvesting actions based on incorrect depth information, ultimately leading to harvesting failure. To address the aforementioned issues and achieve accurate localization of the picking point on the fruit stem, it is necessary to explore a new method that integrates object detection boxes, instance segmentation regions, and pose prediction models to effectively improve the accuracy and robustness of picking point localization. Summary of the Invention

[0003] The purpose of this invention is to provide a dual-task driven method and system for fruit stem identification and picking point positioning correction, so as to overcome the shortcomings of the prior art and improve the accuracy and robustness of fruit picking point positioning.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a dual-task driven method for fruit stem identification and picking point positioning correction, comprising:

[0005] A fruit stem recognition and localization correction model based on a dual-task mechanism is constructed. In the first stage task, the model processes the input image of the fruit to be picked using a fruit stem recognition network, StemNet, to identify the fruit stem and fruit region and locate the key picking point, outputting the fruit region, fruit stem region, and picking point location information. In the second stage task, the model further refines the fruit stem boundary of the fruit stem region output from the first stage task using another fruit stem recognition network, StemNet, to obtain a precise fruit stem segmentation region. Then, the spatial location verification and correction of the picking point is performed by combining the picking point location information obtained from the first stage task and the precise fruit stem segmentation region obtained from the second stage task. The StemNet fruit stem recognition network is based on YOLOv12, replacing the Conv module with a hybrid attention weighted convolution module HAWConv that combines channel attention and spatial attention mechanisms, and replacing the A2C2f module with a stepped attention feature fusion module SteAttn ​​that integrates multi-level attention mechanisms.

[0006] The fruit stalk recognition and localization correction model was trained using the dataset to obtain a well-trained fruit stalk recognition and localization correction model.

[0007] The image to be detected is input into the trained fruit stem recognition and localization correction model to obtain the fruit target and accurate picking point location information.

[0008] Furthermore, the HAWConv module includes the Group Conv module, Norm+ReLU module, Downsample module, GlobalAvgPool module, ChannelAttentionConv module, first Sigmoid module, AttentionWeighting module, AvgPool module, MaxPool module, Concat module, SpatialAttentionConv module, second Sigmoid module, Re-weight module, and Conv module;

[0009] After the input feature map is fed into the HAWConv module, it undergoes feature extraction sequentially through the Group Conv module, Norm+ReLU module, Downsample module, GlobalAvgPool module, ChannelAttentionConv module, first Sigmoid module, and AttentionWeighting module. The input feature map is first processed by group convolution in the Group Conv module, then standardized and activated by the Norm+ReLU module, and then downsampled by the Downsample module to reduce the spatial size of the feature map. Next, the feature map undergoes global average pooling in the GlobalAvgPool module, then the ChannelAttentionConv module calculates the attention weight for each channel, and then passes through the first Sigmoid module to map the attention weights to the range [0,1] using the Sigmoid activation function, and outputs the attention weights to the AttentionWeighting module. The AttentionWeighting module uses the calculated attention weights to weight each channel in the feature map. The AttentionWeighting module outputs features in three paths, with the first path outputting to Re... The first-weight module performs average pooling through the AvgPool module in the second path and max pooling through the MaxPool module in the third path to capture different spatial information. Then, the Concat module concatenates the output features of the AvgPool and MaxPool modules along the channel dimension to form a comprehensive spatial feature map, which is then output to the SpatialAttentionConv module. The SpatialAttentionConv module uses the concatenated spatial feature map to calculate spatial attention, and then applies the second Sigmoid module to generate spatial attention weights, which are then output to the Re-weight module. The Re-weight module uses the generated spatial attention weights to re-weight the channel-weighted feature map. The re-weighted feature map is then subjected to final convolution processing by the Conv module to output the final feature representation.

[0010] Furthermore, the SteAttn ​​module includes a LayerNorm module, a first LSKAttention module, a first StepConv module, a second LSKAttention module, a second StepConv module, a 1x1 Conv module, a GeLU module, and an UpSample module;

[0011] The input feature map is fed into the SteAttn ​​module and then splits into two paths: one path is fed into the LayerNorm module, and the other path is connected to the output of the UpSample module via a skip connection. The LayerNorm module performs layer normalization on the input feature map, and then sequentially passes through the first LSKAttention module, the first StepConv module, the second LSKAttention module, and the second StepConv module for feature extraction. The output of the first LSKAttention module also leads to a skip connection with the output of the second StepConv module. The LSKAttention module applies an attention mechanism to weight the feature map. The StepConv module is a staircase convolution module that combines depthwise separable convolution and dilated convolution for feature extraction. After the output of the second StepConv module is connected to the output of the first LSKAttention module via a skip connection, it undergoes a 1x1 convolution through the 1x1 Conv module to adjust the number of channels in the feature map. Then, the GeLU activation function is applied to the output of the 1x1 convolution through the GeLU module, and finally, the UpSample module performs upsampling to restore the feature map to a suitable spatial resolution. After the upsampled output is connected to the input feature map via a skip connection, the final feature representation is output.

[0012] Furthermore, the StepConv module includes five branches. The first branch includes a 3×3 dilated convolution module, a first 5×5 depthwise separable convolution module, a first 3×3 depthwise separable convolution module, and a first 1×1 depthwise separable convolution module. The second branch includes a second 5×5 depthwise separable convolution module, a second 3×3 depthwise separable convolution module, and a second 1×1 depthwise separable convolution module. The third branch includes a third 3×3 depthwise separable convolution module and a third 1×1 depthwise separable convolution module. The fourth branch includes a third 1×1 depthwise separable convolution module. The fifth branch does not perform any operation, thus forming a stair-shaped feature extraction network. After the input feature map is input into the StepConv module, it is input into each branch respectively to extract features in different ways. Then, the outputs of the five branches are connected to obtain the final feature representation.

[0013] Furthermore, combining the picking point location information obtained from the first phase of the task and the precise fruit stem segmentation area obtained from the second phase of the task, the spatial location of the picking points is verified and corrected, specifically as follows:

[0014] Step A1: Determine whether the vertical coordinate of the picking point is above the target box of the fruit area. If yes, proceed to step A2; otherwise, ignore the picking point and calculate the corrected picking point based on the highest point and geometric center point of the precise fruit stem segmentation area.

[0015] Step A2: Determine whether the picking point is located within the precise fruit stem segmentation area. If yes, determine that the picking point is accurately positioned and output it. Otherwise, calculate the corrected picking point based on the highest point and geometric center point of the precise fruit stem segmentation area.

[0016] Furthermore, when the precisely segmented fruit stalk region is missing, depth information is used for correction, specifically:

[0017] Step B1: Obtain the depth value dp of the picking point and the depth value df of the center point of the target box of the fruit region;

[0018] Step B2: If df < dp < df + r, then adjust the depth value of the picking point to dp' = dp + k, where r is the average radius of the fruit, k is the preset correction value, and 0 < k < r; otherwise, the picking point positioning is determined to have failed.

[0019] Furthermore, the StemNet stem recognition network replaces the UpSample module in the YOLOv12 model with the dynamic sampling module DySample.

[0020] Furthermore, the StemNet fruit stem recognition network includes a backbone network, a neck network, and a head network connected in sequence. The head network includes a segment head for outputting the fruit stem and fruit region and a pose head for outputting the picking point position.

[0021] The present invention also provides a dual-task driven fruit stem recognition and picking point positioning correction system, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, it can implement the above-mentioned method.

[0022] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the above-described method.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. High accuracy: Through the dual-task driven mechanism of "preliminary identification - fine segmentation - spatial correction", the precise spatial constraints provided by fruit stem segmentation are combined with key point prediction, which effectively corrects the problem of lateral deviation of picking point caused by occlusion and misidentification, and significantly improves positioning accuracy.

[0025] 2. Strong robustness: The StemNet network and its core modules (HAWConv, StepAttn, StepConv) are innovatively proposed, which enhances the network's feature extraction and attention focusing capabilities under complex backgrounds, occlusions, and multi-scale targets.

[0026] 3. Balancing efficiency and practicality: The second stage only performs fine segmentation on small fruit stalk regions, which significantly reduces the computational load compared to processing the entire high-resolution image, ensuring the real-time performance of the system and making it more suitable for automated agricultural harvesting scenarios. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the implementation of the dual-task driven fruit stem recognition and picking point positioning correction method provided in this embodiment of the invention.

[0028] Figure 2 This is an architecture diagram of StemNet, the fruit stem recognition network, in an embodiment of the present invention.

[0029] Figure 3 This is a structural diagram of the HAWConv module in an embodiment of the present invention;

[0030] Figure 4 This is a structural diagram of the SteAttn ​​module in an embodiment of the present invention;

[0031] Figure 5 This is a structural diagram of the StepConv module in an embodiment of the present invention;

[0032] Figure 6 This is a rendering of the fruit area, fruit stalk area, and picking point location obtained after the first stage of the task in this embodiment of the invention.

[0033] Figure 7 This is a precise fruit stalk segmentation region map obtained after performing the second stage task in this embodiment of the invention;

[0034] Figure 8 This is a diagram showing the effect of fruit stem identification and picking point positioning after performing two-stage tasks in an embodiment of the present invention.

[0035] Figure 9 This is a diagram showing the corrected fruit stalk identification and picking point positioning effect in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0038] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0039] like Figure 1 As shown, this embodiment provides a dual-task driven method for fruit stem recognition and picking point localization correction, characterized by the following steps:

[0040] S1. Construct a fruit stalk recognition and localization correction model based on a dual-task mechanism. In the first stage task, the model processes the input image of the fruit to be picked using a fruit stalk recognition network (StemNet), identifying the fruit stalk and fruit region and locating key picking points, outputting the fruit region, fruit stalk region, and picking point location information. In the second stage task, the model uses another fruit stalk recognition network (StemNet) to refine the fruit stalk boundary segmentation of the fruit stalk region output from the first stage task, obtaining a precise fruit stalk segmentation region. Then, combining the picking point location information obtained from the first stage task with the precise fruit stalk segmentation region obtained from the second stage task, the picking point is determined. Spatial location verification and correction; StemNet, the stem recognition network, is based on YOLOv12. The Conv module is replaced with HAWConv, a hybrid attention-weighted convolutional module that combines channel attention and spatial attention mechanisms. The A2C2f module is replaced with SteAttn, a step-attention feature fusion module that integrates multi-level attention mechanisms. The UpSample module is replaced with DySample, a dynamic sampling module. StemNet consists of a backbone network, a neck network, and a head network connected in sequence. The head network includes a segmentation head (Segment) for outputting the stem and fruit region and a pose head (Pose) for outputting the picking point position.

[0041] S2. Train the fruit stem recognition and localization correction model using the dataset to obtain the trained fruit stem recognition and localization correction model.

[0042] S3. Input the image to be detected into the trained fruit stem recognition and localization correction model to obtain the fruit target and accurate picking point location information.

[0043] 1. StemNet (a network for recognizing fruit stems)

[0044] like Figure 2As shown, this invention designs a multi-dimensional attention-based stem recognition network (StemNet) to achieve stem segmentation and stem picking point localization tasks, which can simultaneously solve object detection, image segmentation, and keypoint regression problems. To efficiently achieve these tasks, this invention innovatively proposes the HAWConv (Hybrid Attention WeightedConv) module, the StepAttn (step attention feature fusion) module, and the StepConv convolution to implement the aforementioned stem detection network, achieving picking point localization and stem segmentation tasks, effectively improving the accuracy and robustness of fruit picking point localization.

[0045] StemNet is an innovative neural network architecture based on the YOLOv12 model. By combining multi-task learning, attention mechanisms, and dynamic sampling techniques, it successfully achieves dual-task processing of fruit picking point localization and fruit stem segmentation. This network structure not only improves the accuracy of picking point localization but also ensures the accuracy of fruit stem region segmentation, making it suitable for agricultural automation and intelligent harvesting systems, and possessing broad application prospects.

[0046] The StemNet network (fruit stem network) can be divided into three main parts: Backbone, Neck, and Head. This network achieves efficient feature extraction, attention weighting, and multi-task output through innovative module combinations. The HAWConv module utilizes hybrid attention feature convolutions to effectively capture key features while reducing computational complexity; the StepAttn module combines multi-layered attention mechanisms to further enhance the network's ability to focus on the fruit picking point and stem region; and the StepConv module enhances its ability to perceive different scales and long-range dependencies in the image through multi-scale convolutions and dilated convolutions. In StemNet, StepAttn and StepConv work closely together to process features at different scales and spatial regions, ensuring that the network can extract and fuse information from multiple angles to accurately locate the picking point and segment the stem region. Finally, the network outputs the stem region and the picking point location on the stem through the Segment and Pose branches, respectively.

[0047] (1) Backbone

[0048] The backbone network is responsible for extracting features from the input image and providing necessary information for subsequent tasks.

[0049] The StemNet backbone network mainly consists of the following modules:

[0050] HAWConv: Hybrid attention-weighted convolution module, which combines convolution operations and attention mechanisms, can enhance the network's attention to important features and improve the network's ability to perceive key features with lower computational overhead.

[0051] C3k2: This is an existing convolutional module used for feature extraction (a module present in the original YOLOv12 model). Structurally, C3k2 represents three convolutional layers, each using a 3x3 kernel for feature extraction. It is crucial in the early layers of the entire network, effectively capturing local features of the input image.

[0052] StepAttn: A ladder-like attention feature fusion module that learns spatial information by performing convolutional processing at multiple scales, with a particular focus on important image regions.

[0053] The backbone network's task is to extract multi-level features from the input image and provide the necessary low-level information for downstream tasks such as segmentation and localization.

[0054] (2) Neck network

[0055] The neck network is primarily responsible for further processing the features obtained from the backbone and passing them to the final output stage. The StemNet neck network mainly consists of the following modules:

[0056] StepAttn: Here, the StepAttn module further optimizes the representation of the feature map, enhancing attention to important regions through an attention mechanism. By stacking multiple StepAttn modules, the network can effectively learn the features of key regions in the image and provide effective input for segmentation and localization tasks.

[0057] DySample: This module is used for dynamic upsampling of feature maps. Through this module, the network can flexibly adjust according to features at different levels, thereby improving sampling accuracy. Especially when precise localization is required in an image, dynamic sampling helps improve the final localization accuracy.

[0058] The main function of the neck network is to fuse and adjust the multi-scale features extracted from the backbone to optimize the quality of the feature maps.

[0059] (3) Head (output header)

[0060] The head network is responsible for performing the final Pose (picking point location) and Segment (fruit stem segmentation) tasks.

[0061] Segment: The network segments the fruit stem region using the segment header. This segmentation task extracts the fruit stem region from the image, providing a reference for the accurate location of the picking point.

[0062] Pose: The network uses the Pose header to locate the fruit picking point. This task accurately predicts the fruit picking point location by combining the results of the Segment task and the features learned by the StepAttn module.

[0063] 2. HAWConv module

[0064] like Figure 3 As shown, the HAWConv module includes a Group Conv module, a Norm+ReLU module, a Downsample module, a GlobalAvgPool module, a ChannelAttentionConv module, a first Sigmoid module, an AttentionWeighting module, an AvgPool module, a MaxPool module, a Concat module, a SpatialAttentionConv module, a second Sigmoid module, a Re-weight module, and a Conv module. This invention improves the convolutional feature extraction capability by introducing the HAWConv module, particularly enhancing the network's ability to perceive fruit features in complex environments.

[0065] The HAWConv module's processing mainly includes initial convolution and normalization, channel attention calculation, spatial feature extraction and concatenation, spatial attention calculation, and final convolution. After the input feature map is input into the HAWConv module, it undergoes feature extraction sequentially through the Group Conv module, Norm+ReLU module, Downsample module, GlobalAvgPool module, ChannelAttentionConv module, first Sigmoid module, and AttentionWeighting module. The input feature map is first processed through the Group... The Conv module performs grouped convolutions to improve computational efficiency; then, the Norm+ReLU module performs standardization and activation (applying the ReLU activation function), followed by downsampling to reduce the spatial size of the feature map; next, the feature map undergoes global average pooling through the GlobalAvgPool module to capture global information; then, the ChannelAttentionConv module (channel attention convolution module) calculates the attention weights for each channel to enhance the representation of important features; finally, it passes through the first Sigmoid module to map the attention weights to the range [0,1] using the Sigmoid activation function, and outputs the attention weights to the AttentionWeighting module; the AttentionWeighting module uses the calculated attention weights to weight each channel in the feature map; the AttentionWeighting module outputs... The features are processed in three paths: the first path outputs to the Re-weight module, the second path performs average pooling through the AvgPool module, and the third path performs max pooling through the MaxPool module to capture different spatial information. Then, the Concat module concatenates the features output from the AvgPool and MaxPool modules along the channel dimension to form a comprehensive spatial feature map, which is then output to the SpatialAttentionConv module (spatial attention convolution module). The SpatialAttentionConv module calculates spatial attention using the concatenated spatial feature map, and then applies a second Sigmoid module (applying the Sigmoid activation function) to generate spatial attention weights, which are then output to the Re-weight module. The Re-weight module uses the generated spatial attention weights to reweight the channel-weighted feature map. The reweighted feature map is then subjected to final convolution processing through the Conv module to output the final feature representation.

[0066] The HAWConv module acquires global information through the GlobalAvgPool and then generates weight coefficients for each channel via convolution. These weights reflect the importance of each channel. The network then uses these weights to weight the feature maps of each channel. After applying channel attention, spatial attention is also applied. The spatial attention mechanism captures spatial information through AvgPool and MaxPool, and then combines them to generate a spatial attention map used to weight features at specific spatial locations. The HAWConv module combines channel attention and spatial attention mechanisms, learning features through convolutional operations and enhancing the weight of important features through attention weighting, thereby improving network performance. The combination of channel attention and spatial attention enhances the network's understanding of feature maps at different levels, thus improving the localization and segmentation of targets.

[0067] The technical advantages of the HAWConv module are:

[0068] The HAWConv module adaptively assigns different weights to each feature map by weighting each channel. Fruit stems and fruits typically have different visual features, and channel attention automatically identifies which features are more important and suppresses irrelevant channels. This is crucial for fruit stem segmentation and localization, as the features of the stem may be unevenly represented in different image regions, especially in complex scenes where background noise can affect the recognition results. Through this mechanism, the network can make more accurate distinctions between the stem and fruit regions, thereby improving segmentation accuracy.

[0069] Spatial attention mechanisms capture spatial information in images by combining average pooling (AvgPool) and max pooling (MaxPool). For fruit stem segmentation tasks, spatial accuracy is crucial, especially when the stem may intersect with the background or other objects. Spatial attention helps the network focus on important spatial regions related to the stem, improving the accuracy of stem localization in complex backgrounds. By incorporating spatial attention, the network can pay more attention to the spatial location of the stem, contributing to improved accuracy in stem localization.

[0070] The HAWConv module extracts global information through the GlobalAvgPool, thereby obtaining the contextual information of the entire image. This is crucial for fruit stem segmentation and localization, as the spatial distribution of fruit stems and the background are often complex, and the network can better understand the overall layout through global context.

[0071] Through attention mechanisms (especially the combination of channel and spatial attention), HAWConv can enhance the network's understanding of features, particularly in target localization and segmentation, helping the model to more effectively identify important regions and improve localization accuracy.

[0072] In the task of locating the picking point on the fruit stalk, this fine feature weighting can help the model accurately determine the location of the picking point on the fruit stalk, reducing misjudgments and omissions.

[0073] 3. SteAttn ​​module

[0074] This invention innovatively proposes a StepAttn module. Through the StepAttn module, the network can dynamically select regions of interest on feature maps at different levels, enhancing the network's spatial understanding ability and improving segmentation and localization accuracy. Figure 4 As shown, the SteAttn ​​module includes a LayerNorm module, a first LSKAttention module, a first StepConv module, a second LSKAttention module, a second StepConv module, a 1x1Conv module, a GeLU module, and an UpSample module. The input feature map is fed into the SteAttn ​​module and then splits into two paths: one path inputs to the LayerNorm (layer normalization) module, and the other path jumps to the output of the UpSample module. The LayerNorm module performs layer normalization on the input feature map, normalizing the mean and variance of each input feature to make the feature distribution more stable during training, thus accelerating model convergence. Then, the output of the LayerNorm module sequentially passes through the first LSKAttention module, the first StepConv module, the second LSKAttention module, and the second StepConv module for feature extraction. The output of the first LSKAttention module also has a jump connection to the output of the second StepConv module. The LSKAttention module applies an attention mechanism to weight the feature map. The StepConv module is a stair-step convolution module that combines depthwise separable convolution and dilated convolution for feature extraction. After the output of the second StepConv module is skipped and connected to the output of the first LSKAttention module, a 1x1 convolution is performed through the 1x1Conv module to adjust the number of channels in the feature map. The 1x1 convolution effectively compresses or expands channels and increases the network's non-linear expressive power. Then, a smooth GeLU activation function is applied to the output of the 1x1 convolution through the GeLU module, followed by upsampling through the UpSample module to restore the feature map to a suitable spatial resolution. The upsampled output is then skipped and connected to the input feature map to output the final feature representation. In the SteAttn ​​module, two skip connections help avoid information loss, accelerate network convergence, and facilitate gradient flow.

[0075] The SteAttn ​​module utilizes the attention mechanism provided by LSKAttention, enabling the network to effectively focus on important features. It extracts multi-scale information through StepConv while optimizing computational efficiency. Nonlinear transformations and channel adjustments are performed using the GeLU activation function and 1x1 convolutions, and finally, UpSample restores the output resolution. Furthermore, this attention structure avoids information loss through two skip connections, aiding gradient flow and accelerating network convergence. This allows the network to focus on learning more complex representations instead of relearning the same features. This design reduces training time and enables the network to achieve good performance in fewer training epochs. This module is well-suited for image processing and multi-task learning, especially in tasks requiring enhanced feature representation and computational efficiency.

[0076] 4. StepConv module

[0077] This invention innovatively proposes a StepConv module, which, compared to traditional convolutional networks, can better handle target features at different scales, effectively improving detection accuracy. For example... Figure 5 As shown, the StepConv module includes five branches. The first branch includes a 3×3 dilated convolution module, a first 5×5 depthwise separable convolution module, a first 3×3 depthwise separable convolution module, and a first 1×1 depthwise separable convolution module. The second branch includes a second 5×5 depthwise separable convolution module, a second 3×3 depthwise separable convolution module, and a second 1×1 depthwise separable convolution module. The third branch includes a third 3×3 depthwise separable convolution module and a third 1×1 depthwise separable convolution module. The fourth branch includes a third 1×1 depthwise separable convolution module. The fifth branch does not perform any operation, thus forming a staircase-like feature extraction network. After the input feature map is input into the StepConv module, it is input into each branch respectively to extract features in different ways. Then, the outputs of the five branches are connected to obtain the final feature representation.

[0078] The StepConv module is a composite convolutional module that employs multiple depthwise convolutions with different kernels (5×5, 3×3, 1×1, etc.) and a 3×3 dilated convolution to form a stepped layout. Each convolutional layer processes different parts of the input feature map and extracts features in different ways, combining the outputs of different paths through a single connection operation. This structure reduces computational cost, can extract multi-scale features, and enhances the perception of diverse features through multi-path parallel feature extraction and fusion, thereby improving model performance.

[0079] It's important to note that the reason for choosing Dilated 3×3 instead of Dilated 7×7 over Depthwise 5×5 is that the core idea of ​​dilated convolution is to expand the receptive field by inserting gaps (i.e., holes) into the convolution kernel without increasing the actual size of the kernel. This method can expand the receptive range without increasing computational cost, thus capturing a wider range of contextual information. For example, when the dilation rate of a 3×3 dilated convolution is 3, its computational cost is equivalent to 3×3, but its receptive range is equivalent to an actual 9×9 convolution kernel. However, the convolution computation does not perform a 9×9 operation on each pixel, effectively reducing computational complexity. Dilated convolution is particularly effective in capturing global information because it can perceive a larger range of contextual information, especially in modeling sparse or long-range relationships. If a depthwise 7×7 convolutional kernel is used, its computational cost and receptive range are both 7×7 pixels. However, unlike dilated convolution, depthwise separable convolution does not expand the receptive field by inserting holes. Instead, it performs local operations directly on the input feature map. Therefore, each convolutional kernel does not introduce long-distance contextual relationships when perceiving information.

[0080] The technical advantages of the StepConv module are:

[0081] (a) Advantages of multi-scale feature extraction. StepConv employs convolutional kernels of different sizes (e.g., 1×1, 3×3, 5×5) and dilated convolution. These different kernels can extract features at different scales, especially in complex scenes where fruit stems and fruits may have different sizes and shapes. Through multi-scale processing, StepConv can effectively detect and segment fruit stems at different scales, especially when there are significant differences between the fruit and the background, capturing more detailed local features. Dilated convolution, by expanding the receptive field, can capture a wider range of contextual information, helping to identify the relationship between the fruit stem and the background in a larger area and improving segmentation accuracy, especially in complex backgrounds.

[0082] (b) Advantages of reduced computational cost. StepConv significantly reduces computational cost by using depthwise convolution. Each convolutional kernel operates on only a single channel of the input feature map, rather than computing on all channels simultaneously as in traditional convolution, making the model more efficient. Reduced computational cost is particularly important for real-time fruit stem segmentation and picking point localization tasks, as these tasks typically require high processing power, especially in agricultural production environments where large amounts of image data need to be processed.

[0083] (c) Advantages of Parallel Feature Extraction and Fusion. StepConv extracts different feature information by parallelizing multiple convolutional paths. Each convolutional path processes a different part of the input data, and the outputs of these paths are fused together through an addition operation. This design helps the network better handle diverse features in complex scenes, especially since the position and shape of the fruit stem may vary in different backgrounds. Parallel feature extraction of multiple paths enhances the model's ability to perceive fruit stems in different scenes.

[0084] 5. Verification and correction of the spatial location of the picking site

[0085] This invention uses StemNet for fruit stem identification and picking point localization (Task 1), and then uses StemNet for fruit stem segmentation (Task 2). The results of the two tasks are then combined, that is, the picking point location information obtained from the first task and the precise fruit stem segmentation area obtained from the second task are combined to verify and correct the spatial position of the picking point, thereby significantly improving the positioning accuracy of the picking point. Especially when the fruit stem is obscured or the fruit is partially visible, it can achieve accurate positioning, providing information support for the active vision system of the robotic arm during the picking process.

[0086] The specific method for verifying and correcting the spatial location of the picking point is as follows:

[0087] Step A1: Determine whether the vertical coordinate of the picking point is above the target box of the fruit area. If yes, proceed to step A2; otherwise, ignore the picking point and calculate the corrected picking point based on the highest point and geometric center point of the precise fruit stem segmentation area.

[0088] Step A2: Determine whether the picking point is located within the precise fruit stem segmentation area. If yes, determine that the picking point is accurately positioned and output it. Otherwise, calculate the corrected picking point based on the highest point and geometric center point of the precise fruit stem segmentation area.

[0089] When the precisely segmented fruit stalk region is missing, depth information is used for correction, specifically:

[0090] Step B1: Obtain the depth value dp of the picking point and the depth value df of the center point of the target box of the fruit region;

[0091] Step B2: If df < dp < df + r, then adjust the depth value of the picking point to dp' = dp + k, where r is the average radius of the fruit, k is the preset correction value, and 0 < k < r; otherwise, the picking point positioning is determined to have failed.

[0092] The following section provides a further explanation of the specific implementation of the method for verifying and correcting the spatial location of picking points.

[0093] This method, by combining pose detection, multi-angle judgment of the target bounding box and segmented regions, depth map information, and picking point correction, can accurately predict the location of the picking point under different environments. Especially when the fruit stem is obscured or there is no segmented region, the depth map information is compared with the depth of the target bounding box center point to effectively correct the depth position of the picking point, ensuring its accuracy. This method has broad application prospects in automated fruit picking systems, and can improve the efficiency and accuracy of automated picking.

[0094] Let B represent the target bounding box of the fruit, defined as B=[x min ,y min ,x max ,y max ], where (x min ,y min ) and (x max ,y max Let P be the coordinates of the top left and bottom right corners of the box. p =(x p , y p ) represents the predicted picking point coordinates, S represents the segmented region of the fruit stalk (a set of pixels), and C s =(x c ,y c ) represents the geometric center point of the segmented region S, Hs=(x h , y h Let represent the highest point of the segmented region S (i.e., the point with the smallest y-coordinate, assuming the image origin is at the top left corner). Furthermore, when depth information is available, let d... p For picking point P p The depth value, d f The target center point of the fruit (x) f ,y f The depth value of ) and r is the average radius of the fruit (which can be obtained through prior knowledge or dataset statistics).

[0095] Step 1: Verify the position of the picking point relative to the target box

[0096] Based on the prior knowledge that the fruit stalk is usually located above the fruit, the picking point should be located in the area above the target bounding box. If the picking point is located inside or below the box, it may lead to positioning errors due to occlusion or misidentification. Therefore, this point should be ignored and the calculation should rely on the segmented region.

[0097] If y p <y min (B) Then the picking point P p If the initial verification passes, proceed to step 2; otherwise, ignore P. p And the picking point calculation is based on segmented regions (see step 3). Here, y min(B) is the minimum y-coordinate of the target box B.

[0098] Step 2: Verification of the location of the picking point and the divided area

[0099] The fruit stalk segmentation region provides a precise spatial distribution of the fruit stalks, and the picking point should be located within this region. The rationality of the picking point can be verified by a point-in-polygon test.

[0100] If P p If ∈S (i.e., the picking point is located inside the segmented region), then P p If the result is considered correct, the final picking point will be output directly. Otherwise, proceed to step 3 for correction.

[0101] Point-in-region testing can be performed using ray casting or contour inclusion algorithms.

[0102] Step 3: Picking Point Correction Calculation

[0103] When the picking point is above the target box but not within the segmented region, it indicates a local error in the prediction. By interpolating using the geometric features of the segmented region (the highest point and the center point), a reasonable picking point can be generated on the fruit stalk, ensuring that the point is within the critical area of ​​the fruit stalk.

[0104] First, calculate the highest point H of the segmented region S. s and center point C s The highest point H s It can be accessed via y h =min{y|(x,y)∈S} is obtained, and the center point C is obtained. s It can be calculated using the average of pixel coordinates:

[0105] ,

[0106] Then, define the correction point P. c For H s and C s Midpoint between:

[0107]

[0108] To ensure P c Located inside the segmented region S, perform internal point verification: if P c ∉S, then P will be projected. c Adjust it to the inside of S. Specifically, find the entry point from P. c The shortest direction to the boundary of S is P c Move along this direction to the nearest point P inside S. c ′:

[0109]

[0110] Among them, ||·|| represents the Euclidean distance. Finally, the output is used as the corrected picking point.

[0111] Step 4: Depth correction when the segmentation area is missing

[0112] When the fruit stalk is blocked by leaves or the like, resulting in a missing segmentation area, the picking point may be mispositioned on the blocker. The depth information can be used to verify the relative spatial relationship between the picking point and the fruit. By comparing and adjusting the depth, the picking point is corrected to a reasonable depth range to simulate the actual position of the fruit stalk.

[0113] If the fruit stalk segmentation area is not detected (i.e., S = ∅), but there is a picking point P p , then depth verification is performed.

[0114] First, check the depth condition: if df < dp < df + r, it indicates that the picking point may be on the blocker in front of the fruit, and the depth value needs to be adjusted.

[0115] The adjustment formula is:

[0116] dp' = dp + k

[0117] Among them, k is the estimated distance between the leaf and the fruit stalk, and 0 < k < r (k can be set as a fixed value according to the dataset or experiment, such as k = r / 2). Finally, the three-dimensional coordinates of the picking point are updated to (xp, yp, dp′). If the depth condition is not satisfied (i.e., dp ≤ df or dp ≥ df + r), the picking point is ignored and a positioning failure is prompted.

[0118] In this embodiment, the dual-task-driven fruit stalk recognition and picking point positioning and correction method proposed by the present invention is experimentally verified. Figure 6 is the effect diagram of the fruit area, fruit stalk area and picking point position obtained after the first-stage task in this embodiment. Figure 7 is the accurate fruit stalk segmentation area diagram obtained after the second-stage task in this embodiment. Figure 8 is the effect diagram of fruit stalk recognition and picking point positioning obtained after the two-stage tasks in this embodiment. In the figure, the red dots are the predicted picking points, and the yellow is the reference point (i.e., the fruit top key point). After passing through the stage tasks, the predicted picking points may have deviations. As Figure 8 in the left 3 picking points, they deviate from the fruit stalk area. On this basis, combining the picking point position information obtained from the first-stage task and the accurate fruit stalk segmentation area obtained from the second-stage task, the spatial position verification and correction of the picking point are carried out. Figure 9 is the effect diagram of fruit stalk recognition and picking point positioning after correction in this embodiment. As Figure 9 As shown, after correction, the three picking points on the left have returned to the fruit stalks.

[0119] This embodiment also provides a dual-task driven fruit stem recognition and picking point positioning correction system, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method.

[0120] This embodiment also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method.

[0121] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A dual-task driven method for fruit stem recognition and picking point localization correction, characterized in that, include: A fruit stem recognition and localization correction model based on a dual-task mechanism is constructed. In the first stage task, the model processes the input image of the fruit to be picked using a fruit stem recognition network, StemNet, to identify the fruit stem and fruit region and locate the key picking point, outputting the fruit region, fruit stem region, and picking point location information. In the second stage task, the model further refines the fruit stem boundary of the fruit stem region output from the first stage task using another fruit stem recognition network, StemNet, to obtain a precise fruit stem segmentation region. Then, the spatial location verification and correction of the picking point is performed by combining the picking point location information obtained from the first stage task and the precise fruit stem segmentation region obtained from the second stage task. The StemNet fruit stem recognition network is based on YOLOv12, replacing the Conv module with a hybrid attention weighted convolution module HAWConv that combines channel attention and spatial attention mechanisms, and replacing the A2C2f module with a stepped attention feature fusion module SteAttn ​​that integrates multi-level attention mechanisms. The fruit stalk recognition and localization correction model was trained using the dataset to obtain a well-trained fruit stalk recognition and localization correction model. The image to be detected is input into the trained fruit stem recognition and localization correction model to obtain the fruit target and accurate picking point location information.

2. The dual-task driven fruit stem recognition and picking point positioning correction method according to claim 1, characterized in that, The HAWConv module includes the Group Conv module, Norm+ReLU module, Downsample module, GlobalAvgPool module, ChannelAttentionConv module, First Sigmoid module, AttentionWeighting module, AvgPool module, MaxPool module, Concat module, SpatialAttentionConv module, Second Sigmoid module, Re-weight module, and Conv module; After the input feature map is fed into the HAWConv module, it undergoes feature extraction sequentially through the Group Conv module, Norm+ReLU module, Downsample module, GlobalAvgPool module, ChannelAttentionConv module, first Sigmoid module, and AttentionWeighting module. The input feature map is first processed by group convolution in the Group Conv module, then standardized and activated by the Norm+ReLU module, and then downsampled by the Downsample module to reduce the spatial size of the feature map. Next, the feature map undergoes global average pooling in the GlobalAvgPool module, then the ChannelAttentionConv module calculates the attention weight for each channel, and then passes through the first Sigmoid module to map the attention weights to the range [0,1] using the Sigmoid activation function, and outputs the attention weights to the AttentionWeighting module. The AttentionWeighting module uses the calculated attention weights to weight each channel in the feature map. The AttentionWeighting module outputs features in three paths, with the first path outputting to Re... The first-weight module performs average pooling through the AvgPool module in the second path and max pooling through the MaxPool module in the third path to capture different spatial information. Then, the Concat module concatenates the output features of the AvgPool and MaxPool modules along the channel dimension to form a comprehensive spatial feature map, which is then output to the SpatialAttentionConv module. The SpatialAttentionConv module uses the concatenated spatial feature map to calculate spatial attention, and then applies the second Sigmoid module to generate spatial attention weights, which are then output to the Re-weight module. The Re-weight module uses the generated spatial attention weights to re-weight the channel-weighted feature map. The re-weighted feature map is then subjected to final convolution processing by the Conv module to output the final feature representation.

3. The dual-task driven fruit stalk recognition and picking point positioning correction method according to claim 1, characterized in that, The SteAttn ​​module includes the LayerNorm module, the first LSKAttention module, the first StepConv module, the second LSKAttention module, the second StepConv module, the 1x1 Conv module, the GeLU module, and the UpSample module; The input feature map is fed into the SteAttn ​​module and then splits into two paths: one path is fed into the LayerNorm module, and the other path is connected to the output of the UpSample module via a skip connection. The LayerNorm module performs layer normalization on the input feature map, and then sequentially passes through the first LSKAttention module, the first StepConv module, the second LSKAttention module, and the second StepConv module for feature extraction. The output of the first LSKAttention module also leads to a skip connection with the output of the second StepConv module. The LSKAttention module applies an attention mechanism to weight the feature map. The StepConv module is a staircase convolution module that combines depthwise separable convolution and dilated convolution for feature extraction. After the output of the second StepConv module is connected to the output of the first LSKAttention module via a skip connection, it undergoes a 1x1 convolution through the 1x1 Conv module to adjust the number of channels in the feature map. Then, the GeLU activation function is applied to the output of the 1x1 convolution through the GeLU module, and finally, the UpSample module performs upsampling to restore the feature map to a suitable spatial resolution. After the upsampled output is connected to the input feature map via a skip connection, the final feature representation is output.

4. The dual-task driven fruit stalk recognition and picking point positioning correction method according to claim 3, characterized in that, The StepConv module includes five branches. The first branch includes a 3×3 dilated convolution module, a first 5×5 depthwise separable convolution module, a first 3×3 depthwise separable convolution module, and a first 1×1 depthwise separable convolution module. The second branch includes a second 5×5 depthwise separable convolution module, a second 3×3 depthwise separable convolution module, and a second 1×1 depthwise separable convolution module. The third branch includes a third 3×3 depthwise separable convolution module and a third 1×1 depthwise separable convolution module. The fourth branch includes a third 1×1 depthwise separable convolution module. The fifth branch does not perform any operation, thus forming a stair-shaped feature extraction network. After inputting the feature map into the StepConv module, each branch is input separately to extract features in different ways. The outputs of the five branches are then connected to obtain the final feature representation.

5. The dual-task driven fruit stem recognition and picking point positioning correction method according to claim 1, characterized in that, Combining the picking point location information obtained from the first phase of the task with the precise fruit stalk segmentation area obtained from the second phase of the task, the spatial location of the picking points is verified and corrected, specifically as follows: Step A1: Determine whether the vertical coordinate of the picking point is above the target box of the fruit area. If yes, proceed to step A2; otherwise, ignore the picking point and calculate the corrected picking point based on the highest point and geometric center point of the precise fruit stem segmentation area. Step A2: Determine whether the picking point is located within the precise fruit stem segmentation area. If yes, determine that the picking point is accurately positioned and output it. Otherwise, calculate the corrected picking point based on the highest point and geometric center point of the precise fruit stem segmentation area.

6. The dual-task driven fruit stem recognition and picking point positioning correction method according to claim 5, characterized in that, When the precisely segmented fruit stalk region is missing, depth information is used for correction, specifically: Step B1: Obtain the depth value dp of the picking point and the depth value df of the center point of the target box of the fruit region; Step B2: If df < dp < df + r, then adjust the depth value of the picking point to dp' = dp + k, where r is the average radius of the fruit, k is the preset correction value, and 0 < k < r; otherwise, the picking point positioning is determined to have failed.

7. The dual-task driven fruit stalk recognition and picking point positioning correction method according to claim 1, characterized in that, The StemNet pedicel recognition network also replaces the UpSample module in the YOLOv12 model with the dynamic sampling module DySample.

8. The dual-task driven method for fruit stem identification and picking point localization correction according to claim 1, characterized in that, The StemNet fruit stalk recognition network includes a backbone network, a neck network, and a head network connected in sequence. The head network includes a segment head for outputting the fruit stalk and fruit region and a pose head for outputting the picking point position.

9. A dual-task driven fruit stalk recognition and picking point positioning correction system, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, wherein when the processor executes the computer program instructions, it can implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method described in any one of claims 1-8 is implemented.

Images