Detection system, method and equipment for automatically picking pears based on SRSMama

Abstract

The invention discloses an automatic pear picking detection system based on SRSMama. According to the system, three-dimensional selective scanning SS3D is provided, the original scanning mechanism of Vama is improved, the scanning range is expanded from two dimensions to three dimensions, semantic information between channels is efficiently processed, and the capability of a model in a complex orchard environment is enhanced; in order to eliminate redundant background information in the pear garden, the system provides a reward and punishment mechanism (RPM), enhances useful information, weakens redundant information and retains important feature information in a neural network; the invention further provides a stacked feature pyramid network (SFPN), the scale information of the feature fusion section is enhanced, the features of different scales are effectively integrated, and the capability of positioning the dense pear clusters is enhanced. The invention further discloses detection equipment for automatic pear picking.

Description

Technical Field

[0001] The invention belongs to the technical field of automatic fruit picking, and in particular relates to a detection system, method and equipment for automatic pear picking based on SRSMamba. Background Art

[0002] Pears are one of the top five fruits in the world, prized for their economic importance, rich nutritional value, and versatility in both fresh and processed products, winning the favor of growers and consumers. Due to their strong adaptability, early maturity, high productivity, and long economic life, pears are cultivated in many countries, making a significant contribution to the improvement of rural economies and increasing farmers' incomes over a long period of time. Pear picking, a key link in the production chain, has traditionally been a laborious task that is still mainly manual in many areas and is characterized by low automation, intensive labor demand, low efficiency, and high costs. In addition, with industrial and economic growth, the labor force has migrated from rural to urban areas, and changing demographic trends have exacerbated agricultural labor shortages, highlighting the urgent need to integrate automation into fruit picking to improve efficiency and enhance the competitiveness of agricultural products.

[0003] Vision systems have become a key element in the field of robotic harvesting. They play a vital role in identifying and accurately locating the fruit, stem, and calyx, providing important data to the robot's central processor to coordinate and execute the picking operation. Whether wild or cultivated, the complex growth environment of pears, coupled with issues such as occlusion, illumination variation, and the diversity of pear cultivars' appearance, pose significant obstacles to traditional object detection frameworks and limit the development of robotic pickers. In recent years, progress has been made in fruit recognition and detection, achieving high-precision recognition and localization in multi-faceted orchard environments, which can significantly improve the adaptability and robustness of vision systems and the accuracy of neural networks. This progress is key to the widespread application of pear picking robots, solving the contradiction between labor shortages and production growth, controlling production costs, and improving picking efficiency. Traditionally, robotic fruit picking systems rely on sensor-based visual recognition, which, although effective in simple scenarios, has limited anti-interference capabilities. Sensor performance can drop dramatically with environmental changes (such as weather fluctuations), and the specificity of certain fruit types further limits the applicability of sensors. Given the wide variety of pear varieties, each with unique physical characteristics, developing customized systems for different environments requires significant research and production expenses, resulting in a waste of resources.

[0004] As a result, traditional sensor-driven fruit recognition systems have limited application and impact in agricultural operations. The emergence of large datasets and complex model architectures through deep learning has paved the way for innovative solutions. Robotic systems that combine deep learning algorithms with robotic arms for object acquisition have demonstrated their utility in a wide range of fields. Unlike traditional machine learning techniques that require manual feature selection, deep learning models can autonomously discern the best features from large amounts of training data, showing remarkable adaptability and generalization across different objects and environments. This approach not only surpasses sensor-based methods in versatility, but also has the potential to significantly improve pear detection and handling under different conditions, making deep learning the preferred method for achieving efficient and automated pear picking.

[0005] In the automated picking of pears, the accuracy of recognition and positioning is the key to accurate picking by the robotic arm and preventing fruit damage, while the speed of recognition and positioning affects production efficiency. Therefore, it is urgent to design a high-precision and high-efficiency object detection method. In the field of object detection, the Transformer method [Literature: Li Y, Mao H, Girshick R, et al. Exploring plain vision transformer backbones for object detection [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 280-296.] can generally achieve a wider receptive field than CNN, thereby achieving higher accuracy, but the Transformer is limited by the quadratic complexity of the self-attention mechanism, resulting in relatively low efficiency.Mamba network [Literature: Liu Y, Tian Y, Zhao Y, et al.VMamba:Visual State Space Model[J].arXiv preprint arXiv:2401.10166,2024.] Compared with YOLO[Literature: Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). Youonly look once: Unified, real-time object detection. In Proceedings of the IEEEconference on computer vision and pattern recognition (pp.779-788).], FasterRCNN [Literature: Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.] and DETR [Reference: Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020, August). End-to-end object detection with transformers. In European conference on computer vision (pp. 213-229). Cham: Springer International Publishing.] show clear advantages. This advantage, coupled with its success in handling challenging scenarios such as small or occluded object detection in complex environments, makes Vmamba, a variant of the Mamba network designed for agricultural applications, an ideal choice for the difficult task of fruit detection in automated agricultural systems. First, the lightweight design of the Mamba network translates into Vmamba's excellent processing speed, which is critical for agricultural automation applications that require real-time processing of large numbers of images. This real-time capability ensures effective recognition and classification of various fruits, even in dynamic environments such as moving conveyor belts. In terms of detection accuracy, both the Mamba network and Vmamba performed well.By integrating deep separable convolutions and attention mechanisms, Vmamba effectively enhances fruit recognition capabilities in complex backgrounds, breaking through YOLO's limitations in detecting small or partially occluded fruits, which are common challenges in fruit detection scenarios. While Faster R-CNN has a high 93% accuracy, its computational requirements limit its applicability in resource-limited scenarios, and Vmamba overcomes this limitation. Although DETR solves some problems such as class imbalance and frame alignment, its long training time and high computational requirements make it less suitable for rapid deployment in evolving agricultural environments. In contrast, Vmamba's streamlined training process minimizes the need for complex pre- and post-processing, facilitating rapid deployment and updates, which is critical when fruit types and maturity change frequently. While YOLOv7 may have a slight advantage in raw processing speed, this advantage is often offset by its lower accuracy in detecting small or occluded objects, which is a key factor in fruit detection tasks. Vmamba prioritizes the balance between speed and accuracy, making it a more suitable choice for real-world agricultural applications, as both factors are critical. Therefore, Vmamba inherits the advantages of the Mamba network in terms of speed and accuracy, and its enhanced ability to handle small or occluded objects makes it particularly suitable for complex fruit detection. Its efficient processing speed, excellent recognition accuracy (especially for small objects and occluded fruits), and optimized training process make it an ideal choice for image recognition tasks in modern agricultural technology and automation systems. However, in the task of automatic pear picking, Vmamba still faces the following urgent problems (such as Figure 1 (1) The application scenario of automatic pear picking is complex, and the trees may be blocked by irrelevant backgrounds such as leaves and trunks. It is difficult to extract features in complex scenarios, and it is necessary to enhance Vmamba's feature extraction capabilities in complex scenarios; (2) In an environment where pear trees grow lushly, such as orchards and farmlands, complex backgrounds will interfere with detection redundancy, and filtering out complex background information is an important issue; (3) Fruit trees usually produce multiple pears in a season, and the pears are densely stacked or even blocked. Detecting pears in dense scenes is an important issue because non-specially designed networks may cause large-scale missed detections and false positives due to the dense distribution of pears. Summary of the invention

[0006] The present invention aims to provide a detection system for automatic pear picking, which adopts three-dimensional selective scanning SS3D to enhance the feature extraction capability of Vmamba in complex scenes, introduces a reward and punishment mechanism RPM to filter complex background information, and uses a stacked feature pyramid network SFPN to generate a more robust and adaptable model, so as to effectively cope with various challenges in the automated pear harvesting environment.

[0007] In order to achieve the above object, the technical scheme adopted by the present invention is: a detection system for automatic pear picking, wherein the detection system adopts Faster-RCNN as a target detection model for automatic detection of pear tree images; the target detection model adopts SRSMamba as a backbone network for feature extraction of the pear tree image to generate feature maps of different scales; the target detection model adopts a stacked feature pyramid network SFPN to fuse the feature maps of different scales to generate a multi-scale feature map;

[0008] The SRSMamba backbone network includes a Stem block and several VSS blocks; the Stem block generates a three-dimensional feature map from the pear tree image; the VSS block uses a three-dimensional selective scan SS3D to cross-scan the three-dimensional feature map in six directions, and expands the three-dimensional feature map into six sequences; the three-dimensional selective scan SS3D includes a selective scanning space state sequence block (S6 block), and the six sequences are sent to the selective scanning space state sequence block for processing to extract key features in each sequence; the six sequences processed by the selective scanning space state sequence block are merged to obtain a three-dimensional feature map containing global information;

[0009] The VSS block also includes a reward and punishment mechanism RPM, which is used to extract important features of the three-dimensional feature map containing global information and delete redundant features.

[0010] The target detection model Faster-RCNN has certain limitations when processing multi-scale targets. In order to overcome this limitation, Faster-RCNN integrates the feature pyramid network FPN structure to generate multi-scale feature maps, which is the basis for subsequent detection of targets of different sizes. The stacked feature pyramid network SFPN proposed by the detection system enhances the FPN function. The detection system uses RPM and three-dimensional selective scanning SS3D together for feature extraction to generate feature maps of different scales, that is, through several VSS blocks, each layer gradually reduces the resolution of the feature map to generate feature maps of different scales and resolutions. The above are all the preliminary processing processes of the target detection stage of automatic pear picking.

[0011] According to the embodiments of the present invention, the present invention can be further optimized. The following is the technical solution formed after optimization:

[0012] In one preferred embodiment, the six directions are from the upper left corner to the lower right corner, from the upper right corner to the lower left corner, from the lower left corner to the upper right corner, from the lower right corner to the upper left corner, from front to back and from back to front.

[0013] 3D selective scanning SS3D is a further extension of 2D selective scanning SS2D, which increases the four-directional scanning of 2D selective scanning SS2D to six-directional scanning, so as to capture the spatial characteristics of the data more comprehensively. 3D selective scanning SS3D improves the processing effect by increasing the number of variables and adjusting the scanning function.

[0014] In one of the preferred embodiments, the selective scan space state sequence block processes the six sequences through a selective_scan function.

[0015] The selective_scan function is responsible for processing data scans in six directions. This function extracts key features by selectively activating and processing data in specific directions. The selective_scan function is the key function to implement the selective scanning mechanism in the S6 block of the selective scanning space state sequence. It selectively processes the data in the input sequence to reduce computational complexity and improve model performance.

[0016] In one of the preferred embodiments, the reward and punishment mechanism RPM includes a Swish activation function, a Tanh function and a reward and punishment operator;

[0017] The first branch of the reward and punishment mechanism RPM maps each feature point in the three-dimensional feature map containing global information to (0, 1) through the Swish activation function, and the calculation formula is as follows:

[0018]

[0019] in, Output feature map for the first branch, is the feature point of the i-th row and j-th column of the three-dimensional feature map containing global information, and o is the index of the three-dimensional feature map containing global information;

[0020] The second branch of the reward and punishment mechanism RPM uses the three-dimensional feature map containing global information through the Tanh function, and the calculation formula is as follows:

[0021]

[0022] in, Output feature map for the second branch, is the tanh function expression;

[0023] The first branch output feature map and the second branch output feature map are rewarded or punished by a reward or punishment operator, and the calculation formula is as follows:

[0024]

[0025] Among them, T is the reward and punishment threshold, 0≤T≤1, The Hadamard product of the first branch output feature map and the second branch output feature map, that is, the feature merging of the output feature map, This is the feature map after the reward and punishment mechanism is processed.

[0026] In one preferred embodiment, the specific process of generating the multi-scale feature map includes:

[0027] The stacked feature pyramid network SFPN replaces the 1*1 convolution in the feature pyramid network FPN with 3*3 convolution and 1*1 convolution in parallel. The expression is as follows:

[0028]

[0029] Among them, w is the kernel weight of 3x3 convolution, w' is the real-valued kernel weight of 1x1 convolution, and x' in The 1x1 convolution corresponds to the center pixel in the 3x3 convolution. is vector multiplication, x in is the input feature map, x out is the output feature map.

[0030] Based on the same concept, the present invention also provides a method for automatically picking pears using the above detection system, comprising the following steps:

[0031] Step 1), using a professional camera to collect images of pear trees;

[0032] Step 2), using LabelImg to label the pear tree image, obtaining a labeled pear tree image, and using the labeled pear tree image to create a data set;

[0033] Step 3), sending the data set in step 2) to the SRSMamba model for training to obtain the global features of the pear tree image, and obtaining a training feature map after removing redundant information;

[0034] Step 4), sending the training feature map to the stacked feature pyramid network SFPN to generate a multi-scale feature map for training;

[0035] Step 5), sending the training multi-scale feature map to the target detection model for processing, training the target detection model to more accurately detect target pears of different categories, and obtaining a trained target detection model;

[0036] Step 6), select the remaining images in the pear tree image as the pear tree images for testing, and send them into the SRSMamba model to generate a three-dimensional feature map for testing; send the test feature map into the stacked feature pyramid network SFPN to generate a multi-scale feature map for testing; send the multi-scale feature map for testing into the trained target detection model to detect target pears of different categories.

[0037] In one preferred embodiment, the step 2) comprises the following steps:

[0038] S21, preprocessing the pear tree image;

[0039] S22, labeling the pear tree image using LabelImg to obtain a labeled pear tree image in VOC format;

[0040] S23, converting the labeled pear tree image in the VOC format into the COCO format to obtain a data set.

[0041] In one preferred embodiment, the step 3) comprises the following steps:

[0042] S31, selecting part of the pear tree image as a training pear tree image, sending the images to a Stem block, and generating a three-dimensional feature map for training;

[0043] S32, the VSS block uses three-dimensional selective scanning SS3D to cross-scan the training three-dimensional feature map along six directions, and expands the training three-dimensional feature map into six training sequences; the six training sequences are sent to the S6 block of the VSS block for processing, and the key features in each training sequence are extracted; the six training sequences processed by the S6 block are merged to obtain a training three-dimensional feature map containing global information;

[0044] S33, using the reward and punishment mechanism RPM to extract important features of the training three-dimensional feature map containing global information and delete redundant features, to obtain the training feature map processed by the reward and punishment mechanism.

[0045] In one preferred embodiment, the step 6) comprises the following steps:

[0046] The remaining images in the pear tree image are selected as test pear tree images, and sent to the Stem block to generate a three-dimensional feature map for testing; the VSS block uses three-dimensional selective scanning SS3D to cross-scan the three-dimensional feature map for testing in six directions, and expands the three-dimensional feature map for testing into six test sequences; the six test sequences are sent to the selective scanning space state sequence block for processing, and the key features in each test sequence are extracted; the six test sequences processed by the selective scanning space state sequence block are merged to obtain a three-dimensional feature map for testing containing global information; the reward and punishment mechanism RPM is used to extract important features of the three-dimensional feature map for testing containing global information and delete redundant features at the same time, so as to obtain a test feature map after the reward and punishment mechanism is processed; the test feature map after the reward and punishment mechanism is processed is sent to the stacked feature pyramid network SFPN to generate a multi-scale feature map for testing; the multi-scale feature map for testing is sent to a trained target detection model for processing, and the target detection model detects target pears of different sizes.

[0047] Based on the same concept, the present invention also provides a detection device for automatically picking pears, which includes a memory and one or more processors; one or more programs are stored on the memory, and when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-mentioned detection method for automatically picking pears.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1) The present invention provides an automatic pear picking object detection system based on SRSMamba, which can be integrated into a visual robot as a visual system for automatic picking and positioning;

[0050] 2) The SRSMamba model provided by the present invention enhances the model's feature processing capability in complex backgrounds and detection capability in dense scenes by introducing three-dimensional selective scanning SS3D, reward and punishment mechanism RPM and stacked feature pyramid network SFPN, significantly improves the accuracy of object detection in automatic pear picking scenes, and reduces the loss of fruit during automatic picking;

[0051] 3) The SRSMamba model provided by the present invention is the first implementation of the new visual architecture Vmamba in the field of agricultural target detection, opening up a new path for agricultural automation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 These are three challenge scenarios for automatic pear picking based on the Vmamba model;

[0053] Figure 2 This is an overall architecture diagram of SRSMamba according to an embodiment of the present invention;

[0054] Figure 3 is a data sample diagram of a self-made data set according to another embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram of automatic picking of SRSMamba pear detection system based on the Internet of Things according to the third embodiment of the present invention;

[0056] Figure 5 4 is a model performance comparison chart of the third embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0058] Example 1

[0059] Vmamba (Visual State Space Model) is a visual representation learning model based on the state space model SSM (Structured Space Model). VMamba integrates the state space model with the selective scanning mechanism to process visual data.

[0060] The VMamba architecture first divides the input image into blocks, maintains its two-dimensional structure to form feature maps, and then processes it through four VSS blocks (Vision State Space blocks), which form the basic building blocks of the VMamba model. The VMamba model also uses a hierarchical representation, similar to the CNN model and Vision Transformers (ViTs), downsampling the feature maps at different stages to create hierarchical representations of different resolutions, and outputting feature maps of the corresponding scale at each stage. The feature maps of multiple scales output by Vmamba differ in spatial resolution and semantic information richness. The low-level feature maps have high spatial resolution but less semantic information, while the high-level feature maps have the opposite.

[0061] The main innovation of VMamba is that it combines global receptive field and dynamic weights while achieving linear computational complexity. This is in stark contrast to the quadratic complexity of the traditional Transformer. VMamba introduces the Cross-Scan Module (CSM) to address the problem of direction sensitivity in visual data processing. Unlike the Transformer that uses the attention mechanism, the CSM adopts a four-way scanning strategy, scanning from the four corners of the image feature map to the opposite position, ensuring that each element in the feature map can integrate information from all other positions from different directions. This approach achieves a global receptive field without increasing linear computational complexity, which is very different from the traditional Transformer model. CSM allows for a comprehensive understanding of spatial relationships in the image, making it a key difference from traditional models that use attention mechanisms.

[0062] Faster R-CNN (Faster Region-based Convolutional Neural Network) is a region-based convolutional neural network for object detection tasks. It extends the capabilities of its predecessors R-CNN and Fast R-CNN by introducing the Region Proposal Network (RPN). The RPN network autonomously proposes potential object regions from feature maps generated by the Convolutional Neural Network (CNN), greatly improving efficiency and accuracy. The integration of RPN with the Fast R-CNN detector enables the sharing of convolutional features between the proposal and detection networks, simplifying the process. In addition, Faster R-CNN uses RoI (Region of Interest) pooling to extract fixed-size feature vectors from the object region of each proposal for classification and bounding box regression. The Faster R-CNN model sets new standards in object detection, especially in terms of speed and accuracy, making it a key development in computer vision research.

[0063] The embodiment of the present invention provides a detection system for automatic pear picking. The detection system selects Faster R-CNN as a detector for automatic detection of pear tree images, and the loss function adopts Faster R-CNN. The smooth L1 loss of R-CNN, the detection system adopts the SRSMamba model improved based on the Vmamba model as the backbone network for feature extraction of the pear tree image, and generates feature maps of different scales; the SRSMamba backbone network includes a Stem block and several VSS blocks; the Stem block generates a three-dimensional feature map from the pear tree input image; the VSS block uses a three-dimensional selective scan to cross-scan the three-dimensional feature map in six directions, and expands the three-dimensional feature map into six sequences; the three-dimensional selective scan includes a selective scanning space state sequence block, and the six sequences are sent to the selective scanning space state sequence S6 block for processing to extract the key features in each sequence; the six sequences processed by the selective scanning space state sequence S6 block are merged to obtain a three-dimensional feature map containing global information; the VSS block also includes a reward and punishment mechanism RPM, which is used to extract important features of the three-dimensional feature map containing global information and delete redundant features; the target detection model also uses a stacked feature pyramid network SFPN to fuse the feature maps of different scales to generate a multi-scale feature map.

[0064] The Vmamba model divides the input image into tiles through the Stem block to generate a two-dimensional feature map. The Vmamba model also includes multiple stages, each of which consists of a VSS block, which gradually reduces the resolution of the feature map. The VSS block uses a two-dimensional selective scan SS2D module to process the data to achieve a global receptive field with linear complexity. The two-dimensional selective scan SS2D module contains scan expansion, S6 blocks, and scan merge. The S6 model (Selective ScanSpace State Sequential Model) was originally used to solve natural language processing (NLP) tasks. It effectively reduces the computational complexity to linear through the combination of selective scanning and state space model (SSM). However, S6 processes input data based on causal relationships, which means that it can only capture partial information of the data scan, making S6 consistent with NLP tasks involving temporal data, but it poses significant challenges when adapting to non-causal data types such as images, graphs, and collections. Vmamba introduces a cross-scan module that chooses to expand image blocks into sequences along rows and columns (scan expansion), scanning along four different directions: from the upper left corner to the lower right corner, from the lower right corner to the upper left corner, from the upper right corner to the lower left corner, and from the lower left corner to the upper right corner. In this way, any pixel can integrate information from all other pixels in different directions. Each sequence is then reintegrated into a single image, and all sequences are combined into a new sequence.

[0065] 3D Selective Scanning SS3D

[0066] Based on the Vmamba model, this embodiment proposes three-dimensional selective scanning SS3D, adds two different directions of scanning data, forward and backward, compensates the receptive field of the data without increasing the computational complexity, extracts and utilizes the intrinsic spatial and channel relationships in high-dimensional data sets, and reveals the horizontal and vertical spatial continuity, the similarity of adjacent channel features, and the distinction between different categories occupying continuous spatial positions. Three-dimensional selective scanning is particularly suitable for applications with complex spatial features, such as pear detection tasks. Three-dimensional selective scanning SS3D effectively reveals the horizontal and vertical spatial continuity in the data and the feature similarity between different channels through sophisticated multi-dimensional scanning and analysis.

[0067] The three-dimensional selective scanning SS3D increases the four-directional scanning of the two-dimensional selective scanning SS2D to six-directional scanning, so that the spatial characteristics of the data can be captured more comprehensively. The three-dimensional selective scanning SS3D improves the processing effect by increasing the number of variables and adjusting the scanning function. In terms of feature processing, feature normalization and element-by-element product operations, as well as subsequent linear layer processing, are introduced to optimize and fuse feature representation. In the SS3D method, the scanning variables in six directions are first defined. These variables may include but are not limited to forward and reverse X-axis, Y-axis, and Z-axis scanning; the variable initialization part will be set according to the dimension and specific needs of the input data to ensure that the scanning in each direction can be performed correctly. The selective scanning spatial state sequence S6 block processes the data scanning in six directions through the selective_scan function. This function extracts key features by selectively activating and processing data in specific directions. After completing the data scanning in six directions, the obtained features are normalized and then merged with the features processed by the previous layer of convolutional network by element-by-element product. The element-by-element product merging starts from the second layer of convolutional network. SS3D processes image data efficiently through steps such as initialization, scanning, feature processing, and linear layer fusion. The selective_scan function is one of the key functions that implement the selective scanning mechanism in the S6 block. It selectively processes the elements in the input sequence to reduce the computational complexity and improve the model performance. It plays an important role in optimizing the feature extraction and information processing process in the deep learning model.

[0068] In the application of pear detection, 3D selective scanning SS3D first scans the image data along different spatial dimensions, namely horizontal, vertical and channel dimensions, to capture key features such as the shape, size and texture of the pear. Then, 3D selective scanning SS3D uses the feature comparison mechanism between channels to analyze the information captured by each channel and identify the subtle differences between different pears. For example, by comparing color differences and texture patterns in adjacent channels, SS3D can identify pears of different maturity or distinguish healthy fruits from damaged fruits. In addition, 3D selective scanning SS3D also introduces a classification algorithm that combines differential feature extraction and spatial position information to accurately identify and classify different types of pears in continuous spatial regions. It can recognize and utilize subtle differences between objects located in adjacent spatial positions but belonging to different categories.

[0069] (1) Multidimensional feature extraction

[0070] SS3D captures the spatial continuity and channel similarity in the image through multi-dimensional feature extraction. Assume there is a three-dimensional image data I (x, y, c), where x and y are the first and second coordinates of the spatial coordinate system, and c is the channel coordinate. SS3D applies convolution operations in multiple dimensions to extract features, expressed as:

[0071]

[0072] Among them, F xy represents the features obtained by horizontal and vertical scanning, w ij is the convolution kernel weight, k is the convolution kernel size, i is the offset of x, j is the offset of y, -k≤i≤k, -k≤j≤k.

[0073] (2) Inter-channel similarity evaluation

[0074] To evaluate the similarity between channels, SS3D calculates the correlation between adjacent channels. Assume there are two channels c m and c n , their correlation R(c m ,c n ) is calculated as follows:

[0075]

[0076] Among them, R(c m ,c n ) is channel c m and channel c n The relevance of Indicates channel c m The average strength of Indicates channel c n The average intensity of , m is a positive integer, n is a positive integer.

[0077] (3) Category classification formula

[0078] Using spatial continuity and channel features, SS3D uses differential feature extraction when identifying different categories (such as healthy and damaged pears). The differential feature D can be calculated as:

[0079]

[0080] Among them, N pos is the number of positive samples, N neg is the number of negative samples, F(p) is the probability function of positive samples, and F(q) is the probability function of negative samples; C pos is the set of positive feature vectors, C neg is the set of negative feature vectors.

[0081] SS3D not only enhances the model's sensitivity to pear structure and texture patterns, but also provides a more detailed and precise interpretation of the data through in-depth analysis of channel features. This comprehensive data analysis approach significantly improves the performance of machine learning models, enabling them to perform tasks such as object detection, scene classification, and anomaly detection with higher accuracy and reliability. Through this technology, the accuracy of pear detection has been significantly improved, which has important practical significance for agricultural production and quality control.

[0082] Reward and punishment mechanism (RPM)

[0083] Feature points in farmland and pear orchard images usually contain relevant semantic information and redundant information. Too much redundant information may interfere with other feature points. Previous deep learning methods often ignore the contextual relationship between pixels when processing images, resulting in potential false positives and missed detection of targets of interest (objects with highly similar background features). Introducing methods to reduce redundant and irrelevant information in the model can alleviate this phenomenon. The key to reducing redundant information is to extract important feature information such as texture, shape, and semantics, and find the accurate contextual relationship between pixels while ignoring the useless parts of the feature map. Therefore, removing unnecessary features while retaining important features is crucial to effectively enhance the anti-interference ability of the model.

[0084] In order to reduce the impact of redundant information on target detection performance and make full use of the contextual relationship between pixels, this embodiment provides a reward and punishment mechanism, which consists of a Swish activation function, a Tanh function and a reward and punishment operator.

[0085] Since the Swish activation function is very suitable for local response normalization, the first branch of RPM uses the Swish activation function to map each feature point of the convolutional feature map to the (0,1) range. The calculation formula is as follows:

[0086]

[0087] in, Output feature map for the first branch, is the feature point of the i-th row and j-th column of the three-dimensional feature map containing global information, and o is the index of the three-dimensional feature map containing global information.

[0088] Considering the feature points of hyperspectral images, even feature points with negative values ​​contain semantic information after convolution. Therefore, the ReLU function is not used to eliminate negative values ​​from the feature map. The sigmoid activation function is not centered on zero, which will lead to slow network convergence and is not considered. Therefore, the second branch of RPM uses the Tanh function to process the feature map after convolution. The calculation formula is as follows:

[0089]

[0090] in, Output feature map for the second branch, is the tanh function expression.

[0091] Then, the Hadamard product of the two branches is used to merge the features of the output feature map. The fusion of the above output feature map can be expressed as Next, the output feature map will be rewarded and punished, and the core of the operation revolves around setting the threshold T. For features below the threshold T, 0 is assigned to eliminate the feature, while features above the threshold T remain unchanged. This aims to remove redundant information that is prevalent in images and retain important feature information, thereby reducing computational load and improving accuracy. By setting the threshold T, the model achieves selective retention of features. A lower threshold helps filter out signals that are not strong enough, which may be noise or non-critical features, and only important feature information needs to be retained. The effectiveness of RPM can be demonstrated by analogy with threshold processing in image compression technology. And RPM reduces the amount of data by setting a threshold to delete some data. The formula for the reward and punishment operator is as follows:

[0092]

[0093] Among them, T is the reward and punishment threshold, 0≤T≤1, The Hadamard product of the first branch output feature map and the second branch output feature map, that is, the feature merging of the output feature map, This is the feature map after the reward and punishment mechanism is processed.

[0094] The mathematical principle of removing redundant information can be explained from the perspective of information compression and feature selection. RPM effectively screens and optimizes features by combining different activation functions and threshold operations, thereby achieving the purpose of reducing redundant information in the data.

[0095] Stacked Feature Pyramid Network (SFPN)

[0096] In the field of automatic fruit harvesting, the processing of scale information plays a vital role in the accuracy and efficiency of target detection. Since fruits vary in size, shape, and maturity in natural environments, the variation in fruit size in images poses a significant challenge to automatic harvesting systems. Effective use and processing of scale information is essential to improving the accuracy and adaptability of fruit detection. For example, the size of pears varies significantly during their natural growth, which means that their sizes can vary greatly at different stages of maturity. Therefore, the automated harvesting system must be able to accurately identify and process these scale variations to ensure effective detection of fruits of various sizes. This not only improves the applicability of the system, but also ensures the efficiency and accuracy of the harvesting process.

[0097] Effective processing of scale information is also crucial for distinguishing fruits from other objects in the surrounding environment. In complex natural environments, pears often blend in with leaves, branches, and other environmental elements. Through multi-scale analysis, the automatic harvesting system can better distinguish between fruits and backgrounds and improve recognition accuracy. In addition, effective use of scale information helps improve the operational accuracy of the harvesting robot. In the actual harvesting process, the robot needs to adjust the grasping action according to the size and position of the fruit. Accurate recognition and processing of scale information enables embedded deep learning in the robot to locate the fruit more accurately, thereby improving the success rate and efficiency of harvesting.

[0098] Based on the background of using scale information in automated fruit harvesting, this embodiment increases the ability of feature pyramid network (FPN) to capture multi-scale information through convolution [Literature: Lin TY, Dollár P, Girshick R, et al. Feature pyramid networks for object detection [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2117-2125.]. Further, based on the Inception module and Res2Net [Literature: Gao SH, Cheng MM, Zhao K, et al. Res2net: A new multi-scale backbone architecture [J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 43 (2): 652-662.10.1109 / TPAMI.2019.2938758.], this embodiment proposes a scale fusion convolution (SFC) of FPN, i.e., a stacked feature pyramid network (SFPN). SFPN consists of standard 3x3 convolutions paired with 1x1 convolutions. The 3x3 convolutions effectively process spatial information and capture features related to the scale and shape of various fruits, while the 1x1 convolutions serve as a mechanism to integrate these features, ensuring that finer details are not lost during the feature fusion process. The dual convolutional approach of SFPN promotes the flow of information through multiple local receptive fields during feature fusion, enhancing the network's ability to detect fruits of different sizes and maturity with higher accuracy. The 3x3 convolutional layers capture a wider range of backgrounds and more obvious scale variations, which are critical for distinguishing fruits from complex backgrounds such as leaves and branches. On the other hand, the 1x1 convolutional layers focus on finer details and textures, which are critical for identifying smaller fruits or fruits that are partially occluded by leaves and other environmental elements. In addition, the stacked 3x3 and 1x1 convolutions are designed to operate seamlessly within the FPN architecture, ensuring that multi-scale feature extraction does not affect the efficiency of the network. Integrating stacked 3x3 and 1x1 convolutions into FPN results in a more robust and adaptable model that can cope with various challenges that arise in the context of automated fruit harvesting, such as varying lighting conditions, fruit orientations, and occlusions. In summary, the introduction of stacked 3x3 and 1x1 convolutions into Feature Pyramid Networks (FPNs) marks a significant advancement in the field of automated fruit harvesting.By leveraging the advantages of the Inception module and Res2Net, SFC enriches FPN's capabilities in multi-scale information processing, paving the way for more accurate and efficient fruit detection, and thus paving the way for more effective automated harvesting solutions. The stacked feature pyramid network SFPN replaces the 1*1 convolution in the feature pyramid network FPN with 3*3 convolution and 1*1 convolution in parallel, as shown below:.

[0099]

[0100] Among them, w is the kernel weight of 3x3 convolution, w' is the real-valued kernel weight of 1x1 convolution, and x' in The 1x1 convolution corresponds to the center pixel in the 3x3 convolution. is vector multiplication, x in is the input feature map, x out is the output feature map.

[0101] Example 2

[0102] This embodiment provides a detection method for automatically picking pears, comprising the following steps:

[0103] Step 1), using a professional camera to collect images of pear trees;

[0104] Step 2), using LabelImg to label the pear tree image, obtaining a labeled pear tree image, and using the labeled pear tree image to create a data set; the specific steps are as follows:

[0105] S21, preprocessing the pear tree image;

[0106] S22, labeling the pear tree image using LabelImg to obtain a labeled pear tree image in VOC format;

[0107] S23, converting the labeled pear tree image in the VOC format into the COCO format to obtain a data set;

[0108] Step 3), sending the data set in step 2) to the SRSMamba model for training to obtain the global features of the pear tree image, and obtaining a training feature map after removing redundant information; the specific steps are as follows:

[0109] S30, improving the Vmamba model to obtain the SRSMamba model, specifically including: replacing the two-dimensional selective scanning SS2D in the VSS block with the three-dimensional selective scanning SS3D; adding a reward and punishment mechanism RPM in the VSS block;

[0110] S31, selecting part of the pear tree image as a training pear tree image, sending the images to a Stem block, and generating a three-dimensional feature map for training;

[0111] S32, the VSS block uses three-dimensional selective scanning SS3D to cross-scan the training three-dimensional feature map along six directions, and expands the training three-dimensional feature map into six training sequences; the six training sequences are sent to the S6 block of the VSS block for processing, and the key features in each training sequence are extracted; the six training sequences processed by the S6 block are merged to obtain a training three-dimensional feature map containing global information;

[0112] S33, using a reward and punishment mechanism RPM to extract important features of the training three-dimensional feature map containing global information and delete redundant features, to obtain a training feature map processed by the reward and punishment mechanism;

[0113] Step 4), sending the training feature map to the stacked feature pyramid network SFPN to generate a multi-scale feature map for training;

[0114] Step 5), sending the training multi-scale feature map to the region proposal network RPN for processing, training the target detection model to more accurately detect target pears of different categories, and obtaining a trained target detection model;

[0115] Step 6), select the remaining images in the pear tree image as the pear tree image for testing, send them into the SRSMamba model to generate a three-dimensional feature map for testing; send the test feature map into the stacked feature pyramid network SFPN to generate a multi-scale feature map for testing; send the multi-scale feature map for testing into the trained target detection model to detect target pears of different categories, the specific steps are as follows:

[0116] The remaining images in the pear tree image are selected as test pear tree images and sent to the Stem block to generate a three-dimensional feature map for testing; the VSS block uses three-dimensional selective scanning SS3D to cross-scan the three-dimensional feature map for testing in six directions, and expands the three-dimensional feature map for testing into six test sequences; the six test sequences are sent to the S6 block for processing to extract the key features in each test sequence; the six test sequences processed by the S6 block are merged to obtain a three-dimensional feature map for testing containing global information; the reward and punishment mechanism RPM is used to extract important features of the three-dimensional feature map for testing containing global information and delete redundant features at the same time to obtain a test feature map processed by the reward and punishment mechanism; the test feature map processed by the reward and punishment mechanism is sent to the stacked feature pyramid network SFPN to generate a multi-scale feature map for testing; the multi-scale feature map for testing is sent to the region proposal network RPN for processing, and the target detection model detects target pears of different sizes.

[0117] Dataset acquisition

[0118] In the field of automated pear picking, existing datasets are very scarce, and there is currently no public dataset for automatic detection of pear tree objects, making it difficult for the model to achieve accurate recognition and detection. Therefore, before implementing this embodiment, it is necessary to collect and create a dataset. First, a Canon professional camera is used to capture images, and then low-quality images caused by camera shake and other factors are removed, resulting in a total of 7,366 images of pear trees in farmland and orchards. Subsequently, the collected images are converted into a format suitable for model training and reasoning, including selection, cropping, and standardization. The LabelImg software is used to annotate the pear tree objects (including pear body, pear stalk, and pear calyx) in the collected images, and the annotation information includes location coordinates, length, height, etc. After the operation of the LabelImg annotation software, a dataset in the VOC format is obtained. Due to the limitations of the code and model framework, the dataset is further converted to the COCO format. An example of a homemade automatic pear picking dataset is shown below. Figure 3 As shown in Figure 2, the dataset is rich in content, highlighting the richness of pear varieties in the automated pear picking scenario.

[0119] Experimental Results

[0120] An experimental evaluation of the object detection capability of the automatic pear picking detection method based on SRSMamba provided in an embodiment of the present invention is performed, and a comparison with similar models on the same data set is performed.

[0121] As shown in Table 1, SRSMamba outperforms other detectors in the test of the self-made automatic pear picking dataset, outperforming all other detectors in detection accuracy and achieving competitive results with many single-stage detectors in real time. The 7366 self-assembled image data are divided into training set, validation set and test set in a ratio of 7:1:2. SRSMamba achieves 94.8% mAP50 and 26.9 FPS, surpassing other state-of-the-art object detection methods, proving the effectiveness of the SRSMamba model.

[0122] Table 1

[0123]

[0124]

[0125] In the field of computer vision and deep learning, GFLOPs (Giga Floating Point Operations Per Second) refers to one billion floating point operations per second, which is used to measure the computational complexity of the model; mAP (mean Average Precision) refers to the average accuracy of the model across all categories, while mAP50 and mAP75 refer to the average accuracy when the IoU (intersection over union) threshold is 0.50 and 0.75 respectively, both of which are key indicators for evaluating the performance of target detection models; FPS (Frames Per Second) is used to measure the frame rate of the model processing the video stream, reflecting the real-time performance of the algorithm. The above indicators together help evaluate and compare the effectiveness and applicability of different models.

[0126] Experimental results show that the number of parameters of SRSMamba is as low as 21.1M, GFLOPs is 50.4, mAP reaches 72.0%, mAP50 reaches 94.8%, mAP75 reaches 68.1%, and FPS reaches 26.9. Compared with Vmamba, SRSMamba improves mAP by 2.9. The effectiveness of the SRSMamba model on the self-made automatic pear picking dataset COCO2017 is proved, and its good performance in pear detection tasks and its potential in practical applications are demonstrated.

[0127] Example 3

[0128] In order to verify the generalization ability of the model, 300 real orchard images of a pear orchard in a village were sorted and photographed. In order to establish a complete system to facilitate actual detection operations, this embodiment establishes an Internet of Things system with a camera as the image input interface, such as Figure 4 As shown. On the one hand, the IoT system is easily disturbed by the poor signal quality in the orchard, which affects the transmission of the image and the final detection performance of the model. On the other hand, a good pear detection model needs to be tested not only under ideal lighting conditions, but also in a real, complex orchard environment. Because the actual application environment may involve more common environmental interferences, such as shadows, changing lighting, and obstacles.

[0129] exist Figure 4In the paper, edge computing and server computing are handled in the "network processing" block and the "host computing" block, respectively. Specifically, the "network processing" block is the computing tasks performed on the edge device, such as the initial stages of image preprocessing and feature extraction. These tasks are designed to be lightweight and efficient, leveraging the computing resources available on the edge device to reduce the amount of data transmitted to the server. The "host computing" block is the computing tasks performed on the server, which usually involves more complex and computationally intensive operations, such as the final stages of object detection and classification. The server uses its powerful processing power to analyze the preprocessed data received from the edge device and generate the final detection results. The task division between edge computing and server computing aims to optimize the overall system performance by balancing the trade-offs between latency, bandwidth usage, and computing resources. This distributed computing architecture enables efficient and real-time object detection in the environment of pear fruit picking.

[0130] 300 real pear tree images taken in a village pear orchard were divided into three categories for evaluation. Category 1 contains close-range targets with unique features, Category 2 covers medium-range targets with less obvious features, and Category 3 includes long-range targets with difficult-to-distinguish features. Figure 5 The performance comparison of Vmamba and SRSMamba in three different pear tree detection scenarios is shown. The data shows that Vmamba and SRSMamba have an accuracy of 99% and 92% in the first category of pear trees, 91% and 73% in the second category of pear trees, and 78% and 51% in the third category of pear trees.

[0131] To address the challenges of poor signal quality and environmental interference in the pear orchard, the IoT system uses highly sensitive cameras optimized for variable lighting conditions. These cameras dynamically adjust to minimize the effects of shadows and sunlight fluctuations, which is critical in outdoor agricultural environments. The system uses a mesh network topology to enhance the reliability of data transmission. This approach enables each device to connect with multiple nodes, ensuring that data is rerouted in the event of node failure or signal interference, thereby maintaining continuous network performance throughout the orchard. Edge computing capabilities are integrated to process data locally at the camera site, reducing the amount of data sent to the central server and speeding up overall processing time. This local processing includes preliminary image analysis, which is critical for efficient data management and rapid decision-making. Security measures are strictly implemented, utilizing encrypted communication protocols and secure storage to prevent data leakage and ensure the integrity of transmitted information. In addition, the system's scalable design allows for the integration of new technologies and additional sensors, providing flexibility and future growth potential. This scalability ensures that the IoT infrastructure can adapt to evolving technological advances and changing agricultural practices. The system's enhanced capabilities not only enhance the efficacy of pear detection and health monitoring, but also strengthen the infrastructure to address common problems in agricultural IoT applications.

[0132] Example 4

[0133] The present embodiment provides a detection device for automatically picking pears, the detection device comprising a memory and one or more processors; one or more programs are stored in the memory, and when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the detection method described in Example 2 of the present invention.

[0134] The contents explained in the above embodiments should be understood as these embodiments are only used to more clearly illustrate the present invention, and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the embodiments made by those skilled in the art fall within the scope defined by the claims attached to the present invention.

Claims

1. A detection system for automatic pear picking, wherein the detection system uses Faster-RCNN as a target detection model for automatic detection of pear tree images; characterized in that: The target detection model uses SRSMamba as the backbone network for feature extraction of the pear tree image to generate feature maps of different scales; the target detection model uses a stacked feature pyramid network SFPN to fuse the feature maps of different scales to generate a multi-scale feature map; The SRSMamba backbone network includes the Stem block and several VSS blocks; The Stem block generates a three-dimensional feature map from the pear tree image; The VSS block uses three-dimensional selective scanning SS3D to cross-scan the three-dimensional feature map along six directions to expand the three-dimensional feature map into six sequences; The three-dimensional selective scanning SS3D includes a selective scanning space state sequence block, the six sequences are sent to the selective scanning space state sequence block for processing, and are used to extract key features in each sequence; the six sequences processed by the selective scanning space state sequence block are merged to obtain a three-dimensional feature map containing global information; The VSS block also includes a reward and punishment mechanism RPM, which is used to extract important features of the three-dimensional feature map containing global information and delete redundant features.

2. The detection system for automatic pear picking according to claim 1, characterized in that: The six directions are from the upper left corner to the lower right corner, from the upper right corner to the lower left corner, from the lower left corner to the upper right corner, from the lower right corner to the upper left corner, from front to back, and from back to front.

3. The detection system for automatic pear picking according to claim 1, characterized in that: The selective scan space state sequence block processes the six sequences through the selective_scan function.

4. The detection system for automatic pear picking according to claim 1, characterized in that: The reward and punishment mechanism RPM includes a Swish activation function, a Tanh function and a reward and punishment operator; The first branch of the reward and punishment mechanism RPM maps each feature point in the three-dimensional feature map containing global information to (0, 1) through the Swish activation function, and the calculation formula is as follows: in, Output feature map for the first branch, is the feature point of the i-th row and j-th column of the three-dimensional feature map containing global information, and o is the index of the three-dimensional feature map containing global information; The second branch of the reward and punishment mechanism RPM uses the three-dimensional feature map containing global information through the Tanh function, and the calculation formula is as follows: in, Output feature map for the second branch, is the tanh function expression; The first branch output feature map and the second branch output feature map are rewarded or punished by a reward or punishment operator, and the calculation formula is as follows: Among them, T is the reward and punishment threshold, 0≤T≤1, The Hadamard product of the first branch output feature map and the second branch output feature map, that is, the feature merging of the output feature map, This is the feature map after the reward and punishment mechanism is processed.

5. The detection system for automatic pear picking according to claim 1, characterized in that: The specific generation process of the multi-scale feature map includes: The stacked feature pyramid network SFPN replaces the 1*1 convolution in the feature pyramid network FPN with 3*3 convolution and 1*1 convolution in parallel. The expression is as follows: Among them, w is the kernel weight of 3x3 convolution, w' is the real-valued kernel weight of 1x1 convolution, and x' in The 1x1 convolution corresponds to the center pixel in the 3x3 convolution. is vector multiplication, x in is the input feature map, x out is the output feature map.

6. A method for automatically picking pears using the detection system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Step 1), collecting pear tree images; Step 2), using LabelImg to label the pear tree image, obtaining a labeled pear tree image, and using the labeled pear tree image to create a data set; Step 3), sending the data set in step 2) to the SRSMamba model for training to obtain the global features of the pear tree image, and obtaining a training feature map after removing redundant information; Step 4), sending the training feature map to the stacked feature pyramid network SFPN to generate a multi-scale feature map for training; Step 5), sending the training multi-scale feature map to the target detection model for processing, training the target detection model, and obtaining a trained target detection model; Step 6), select the remaining images in the pear tree image as the pear tree images for testing, and send them into the SRSMamba model to generate a three-dimensional feature map for testing; send the test feature map into the stacked feature pyramid network SFPN to generate a multi-scale feature map for testing; send the multi-scale feature map for testing into the trained target detection model to detect target pears of different categories.

7. The detection method for automatic pear picking according to claim 6, characterized in that: The step 2) comprises the following steps: S21, preprocessing the pear tree image; S22, labeling the pear tree image using LabelImg to obtain a labeled pear tree image in VOC format; S23, converting the labeled pear tree image in the VOC format into the COCO format to obtain a data set.

8. The detection method for automatic pear picking according to claim 6, characterized in that: The step 3) comprises the following steps: S31, selecting part of the pear tree image as a training pear tree image, sending the images to a Stem block, and generating a three-dimensional feature map for training; S32, the VSS block uses three-dimensional selective scanning SS3D to cross-scan the training three-dimensional feature map along six directions, and expands the training three-dimensional feature map into six training sequences; the six training sequences are sent to the selective scanning space state sequence block for processing, and the key features in each training sequence are extracted; the six training sequences processed by the selective scanning space state sequence block are merged to obtain a training three-dimensional feature map containing global information; S33, using the reward and punishment mechanism RPM to extract important features of the training three-dimensional feature map containing global information and delete redundant features, to obtain the training feature map processed by the reward and punishment mechanism.

9. The detection method for automatic pear picking according to claim 6, characterized in that: The step 6) comprises the following steps: The remaining images in the pear tree image are selected as test pear tree images and sent to the Stem block to generate a three-dimensional feature map for testing; the VSS block uses three-dimensional selective scanning SS3D to cross-scan the three-dimensional feature map for testing in six directions, and expands the three-dimensional feature map for testing into six test sequences; the six test sequences are sent to the selective scanning space state sequence block for processing to extract the key features in each test sequence; the six test sequences processed by the selective scanning space state sequence block are merged to obtain a three-dimensional feature map for testing containing global information; the reward and punishment mechanism RPM is used to extract important features of the three-dimensional feature map for testing containing global information and delete redundant features at the same time to obtain a test feature map processed by the reward and punishment mechanism; the test feature map processed by the reward and punishment mechanism is sent to the stacked feature pyramid network SFPN to generate a multi-scale feature map for testing; the multi-scale feature map for testing is sent to the region proposal network RPN for processing, and the target detection model detects target pears of different sizes.

10. A detection device for automatically picking pears, characterized in that: The detection device includes a memory and one or more processors; one or more programs are stored in the memory, and when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the method described in one of claims 6 to 9.

Images