Deep learning-based strawberry fruit detection and picking key point positioning method and model

Abstract

The invention relates to the technical field of agricultural picking robot visual perception, and discloses a strawberry fruit detection and picking key point positioning method and model based on deep learning. The method comprises the following steps: acquiring strawberry RGB-D images to construct a data set; on the basis of a YOLOv8-Pose architecture, global feature extraction is enhanced by constructing a C2f-MAMBA module of a fusion state space model, an HWD module is introduced to retain detail features, and a Faster Net Block is used for weight reduction, so that an STRAW-MAMBA model is constructed; outputting coordinates of five key points such as a fruit bounding box and a shear point by the model, and performing geometric compensation on a shielding point; and finally, calculating three-dimensional space coordinates of the key points in combination with depth information. According to the method, a C2f-MAMBA module based on Mama is innovatively introduced, global feature association is effectively modeled, a multi-scale attention mechanism is combined, the defect that long-range dependence is difficult to capture in a traditional CNN is overcome, and the detection and positioning precision of strawberry fruits and picking key points thereof in a complex background is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of visual perception technology for agricultural harvesting robots, specifically to a method and model for strawberry fruit detection and key point localization during harvesting based on deep learning. Background Technology

[0002] Strawberries, a perennial herbaceous plant belonging to the genus *Fragaria* of the Rosaceae family, are an important berry widely cultivated globally. Traditional manual harvesting methods are labor-intensive, inefficient, and costly, making them unsuitable for modern large-scale cultivation. With the rapid development of artificial intelligence technology, utilizing automated harvesting robots to improve harvesting efficiency has become a crucial research direction in smart agriculture, significantly contributing to reducing labor costs and increasing operational efficiency.

[0003] However, automating strawberry harvesting faces multiple technical challenges. On one hand, strawberries typically grow in dense clusters, exhibiting diverse shapes and easily obscured or overlapping by leaves and vines, making machine vision inspection difficult. On the other hand, strawberries are soft and easily damaged, requiring precise harvesting methods such as pulling; instead, end effectors must accurately locate the stem cutting point and gently grasp the fruit surface to achieve damage-free harvesting. Furthermore, existing harvesting models typically have high computational costs, leading to high robot system costs and limiting large-scale deployment in real-world strawberry farm environments. Therefore, developing a lightweight, high-precision model for strawberry fruit detection and key point localization during harvesting has become crucial for promoting the practical application of harvesting robots.

[0004] In existing research, key point localization for early fruit picking often employs image processing combined with geometric mechanism analysis. This involves segmenting the fruit based on color, shape, and texture features, and then using geometric calculations to estimate the picking point. While this method is effective in controlled environments, it lacks robustness in complex orchard environments and struggles to adapt stably to real-world conditions such as changes in light intensity, shading, and the diversity of fruit morphology.

[0005] With the development of deep learning technology, methods for locating key points in fruit picking have gradually evolved. Early mainstream approaches combined deep learning with geometric computation, using data-driven deep models for fruit detection and segmentation, and then deriving key points through geometric calculations. While this approach improved robustness in fruit recognition, its applicability and effectiveness were limited because key point localization still relied on geometric assumptions. Related research includes Mask-RCNN with geometric computation proposed by Yu, Zhang, Yang, & Zhang in 2019, and a method based on rotated YOLO bounding boxes with geometric computation proposed by Yu, Zhang, Liu, Yang, & Zhang in 2020.

[0006] In recent years, end-to-end multi-task learning models have provided new solutions for this field. These models can uniformly perform fruit detection, instance segmentation, and keypoint localization for picking, requiring only a single frame of RGB image input to simultaneously output target bounding boxes, segmentation masks, and keypoint coordinates, significantly simplifying the visual processing workflow. Representative models include the standard YOLOv8-pose model and the STRAW-YOLO model proposed by Ma Zenghong et al. in 2025 based on YOLOv8-Pose. However, existing models are mostly built on convolutional neural networks, whose feature extraction range is limited to local receptive fields, making it difficult to effectively capture long-range global dependencies, thus restricting further improvements in model accuracy. Furthermore, the complex structure and large number of parameters in most models also lead to high computational costs, limiting their deployment and application in practical picking systems. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and model for strawberry fruit detection and key point localization based on deep learning, in order to solve the technical problems mentioned in the background.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A deep learning-based method for strawberry fruit detection and key point localization during picking includes the following steps: S1. Data Acquisition and Preprocessing: RGB images and depth images of strawberries are acquired simultaneously using a depth camera. The RGB images are labeled and data augmented to construct a training dataset. S2. Constructing the STRAW-MAMBA model: Based on the YOLOv8-Pose model architecture, the C2f-MAMBA module is constructed to enhance global and multi-scale feature extraction, the HWD module is introduced to retain high-frequency detail features, and FasterNetBlock is used for lightweight design to form the STRAW-MAMBA model. S3. Model Training and Output: Input the training dataset into the STRAW-MAMBA model for training, so that the model can output the detection bounding box of strawberry fruit and the coordinates of five picking key points; S4. Key point localization reasoning and information fusion: The real-time acquired strawberry RGB image is input into the trained STRAW-MAMBA model to obtain the strawberry detection result and the initial two-dimensional pixel coordinates of five picking key points; for occluded key points, based on the preset geometric constraints between the five picking key points, their two-dimensional pixel coordinates are recovered by calculation, and the depth values ​​corresponding to all key points are obtained from the synchronously acquired depth image. S5. Spatial coordinate calculation: Combining the two-dimensional pixel coordinates of the key points, the corresponding depth values, and the camera parameters, calculate the three-dimensional coordinates of the strawberry center and each picking key point in the camera coordinate system and the world coordinate system.

[0009] As a further aspect of the present invention, in step S1, the depth camera is a depth camera based on structured light or time-of-flight ranging.

[0010] As a further aspect of the present invention, in step S1, the annotation adopts a five-point annotation method, and the annotation content includes the strawberry bounding box, category, credibility, and key picking points; The categories include two types: those that can be picked and those that cannot be picked. The key points for picking are the cutting point, the upper vertex, the lower vertex, the left gripping point, and the right gripping point.

[0011] As a further aspect of the present invention, in step S2, constructing the STRAW-MAMBA model includes: S21. Construct and deploy the C2f-MAMBA module to enhance global and multi-scale feature extraction: Based on the EfficientVim model, a multi-scale convolutional attention module is introduced to achieve multi-scale feature extraction, and the LayerScale mechanism is used to dynamically weight the residual connections to construct the Straw-Vim module. The constructed Straw-Vim module is used to replace the Bottleneck component in the C2f module of the YOLOv8-Pose model, forming the C2f-MAMBA module. Replace the first two C2f modules in the backbone network and the last C2f module in the neck network of the YOLOv8-Pose model with the C2f-MAMBA module. S22. Introducing an HWD module to preserve high-frequency detail features: Replace the standard convolutions in the backbone and neck networks of the YOLOv8-Pose model, except for the first layer, with HWD modules; S23. Introduce FasterNet Block to achieve model lightweighting: Replace the last two C2f modules in the backbone network of the YOLOv8-Pose model with FasterNet Blocks to complete the model construction.

[0012] As a further aspect of the present invention, the algorithm flow of the C2f-MAMBA module includes the following steps: a) Sequence modeling of input features based on the State-Space Model (SSM) to capture long-range dependencies; b) Optimize the gating mechanism and output projection of features processed by SSM using the Hidden State Mixer (HSM); c) Introduce the Multi-Scale Convolutional Attention (MSCA) module to perform multi-scale convolutional attention calculations on the optimized features to improve the accuracy of feature fusion.

[0013] As a further aspect of the present invention, the HWD module decomposes and reconstructs the feature map through Haar wavelet downsampling, thereby preserving high-frequency detail features of the image and reducing the loss of detail information while achieving feature downsampling.

[0014] As a further aspect of the present invention, in step S4, the specific method for restoring the two-dimensional pixel coordinates of the occluded key points based on geometric relationships is as follows: When the shearing point is not visible, the coordinates of the shearing point are calculated based on the coordinates of the upper and lower vertices and the vector from the lower vertex to the upper vertex. When neither the cutting point nor the top vertex is visible, the coordinates of the cutting point and the top vertex are derived and calculated using the coordinates of the midpoint of the line segment connecting the left and right grab points: When the cut-off point, top vertex, and one grab point are all invisible, first calculate the coordinates of the missing grab point using the coordinates of the visible key points, and then deduce the coordinates of the remaining occluded points in the same way as in the case where the cut-off point and top vertex are both invisible.

[0015] As a further aspect of the present invention, step S5, the spatial position calculation specifically includes: S51. Based on the strawberry depth image and camera intrinsic parameters, calculate the three-dimensional coordinates of the pixels in the camera coordinate system: X camera =(u-cx)*depth(u,v) / fx; Y camera =(v-cy)*depth(u,v) / fy; Z camera =depth(u,v); Where (u,v) are pixel coordinates, depth(u,v) is the depth value at that pixel, (cx,cy) are the coordinates of the camera optical center, and (fx,fy) is the camera focal length. S52. Using the camera's extrinsic parameter matrix, the three-dimensional point P in the camera coordinate system... camera Transform to world coordinate system: P world =R -1 ×(P camera -t); Where R and t are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, respectively, R-1 is the inverse of the rotation matrix R, Pcamera is the 3D coordinate vector in the camera coordinate system, and Pworld is the 3D coordinate vector in the transformed world coordinate system.

[0016] As a further aspect of the present invention, it also includes S6, robot deployment and execution: the STRAW-MAMBA model is deployed to the strawberry picking robot platform, and the robot completes the automated picking operation based on the key point positions in the world coordinate system.

[0017] A deep learning-based model for strawberry fruit detection and key point localization during picking is proposed, which is the STRAW-MAMBA model constructed using the method described above.

[0018] Compared with the prior art, the beneficial technical effects of the present invention are as follows: 1. This invention innovatively introduces a C2f-MAMBA module based on Mamba, which effectively models global feature associations and combines a multi-scale attention mechanism to overcome the shortcomings of traditional CNNs in capturing long-range dependencies, thus significantly improving the detection and localization accuracy of strawberry fruits and key picking points in complex backgrounds. 2. By retaining more high-frequency detail features through the HWD module, the problem of information loss during downsampling is effectively alleviated; at the same time, the occlusion key point recovery mechanism based on preset geometric constraints enhances the robustness of the model in complex scenarios such as occlusion by branches and leaves, ensuring the complete output of the picking point. 3. By replacing some complex modules with lightweight FasterNet Blocks and optimizing the HWD module, the number of model parameters and computational complexity are greatly reduced while maintaining high performance, achieving real-time inference with low parameter count and high frame rate, which is easy to deploy on embedded devices. 4. Based on the pre-defined geometric constraints between the five points, the occluded key picking points are effectively recovered through logical calculations, which enhances the robustness and practicality of the system in complex real-world environments and reduces the dependence of the picking task on the complete visibility of the fruit. 5. This invention provides a complete solution from two-dimensional image perception to three-dimensional spatial coordinate calculation, and seamlessly connects the model output with the robot actuator to form an end-to-end automated picking system, which greatly improves the working efficiency and intelligence level of the strawberry picking robot. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention. Figure 2 This is a schematic diagram of the camera pose during the data acquisition process of this invention; Figure 3This invention provides a comparison of the three-point and five-point marking methods for strawberries. Figure 4 This is a schematic diagram of the STRAW-MAMBA model structure of the present invention; Figure 5 This is a schematic diagram of the C2f-MAMBA module structure of the present invention; Figure 6 This is a schematic diagram of the HSM-SSD module structure of the present invention; Figure 7 This is a schematic diagram of the MSCA module structure of the present invention; Figure 8 This is a schematic diagram of the PConv and FasterNet Block module structure of the present invention; Figure 9 This is a schematic diagram of the HWD module structure of the present invention; Figure 10 The bar chart shows the comparative experiment of the STRAW-MAMBA of this invention with mainstream harvesting key point positioning models; Figure 11 This is a comparison chart of the effects of the present invention STRAW-MAMBA with mainstream harvesting key point positioning models; Figure 12 This is a comparison chart showing the effective receptive field visualization results of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0021] like Figure 1 As shown in the figure, this embodiment provides a method for strawberry fruit detection and key point localization based on deep learning. Its core is to construct and apply the STRAW-MAMBA deep learning model. The specific implementation steps of this method are as follows: S1. Data Acquisition and Preprocessing In this embodiment, an Intel® RealSense™ Depth Camera D455 (based on structured light principles) was used to acquire RGB images and corresponding depth images of a strawberry orchard under natural light conditions. The camera's posture during the acquisition process was as follows: Figure 2 As shown, the image resolution is set to 1280×720; The acquired RGB images are annotated. This embodiment uses a five-point annotation method, and the annotation content includes four parts: Strawberry bounding box: A rectangular frame that tightly surrounds the strawberry fruit, such as... Figure 3 As shown in (c); Category: Divided into two categories: "harvestable" and "non-harvestable," which are determined by the labelers based on the ripeness and integrity of the fruit; Confidence: Characterizes the confidence level of the annotation results; Five key harvesting points: These include the cutting point (where the fruit stem connects to the fruit and needs to be cut), the top point (the very top of the fruit), the bottom point (the very bottom of the fruit), the left grasping point, and the right grasping point (the optimal contact points between the robotic arm and the fruit). The locations of these key points are as follows: Figure 3 As shown in (b).

[0022] and Figure 3 Compared to the three-point annotation method shown in (a) (which only includes the cutting point, top vertex, and bottom vertex), the core advantage of the five-point annotation method lies in the introduction of left and right grasping points, which, together with the top and bottom vertices, define the geometric central axis of the strawberry. This achieves a higher level of understanding, moving from isolated point annotation to a holistic understanding of the fruit's posture and space. This geometric prior not only provides the robot with a clear picking orientation but, more importantly, empowers the system to intelligently recover occluded points (such as the cutting point) in occluded scenarios by using the geometric relationships between visible key points. This fundamentally solves the problems of poor adaptability and insufficient robustness of the three-point method in complex environments.

[0023] To increase the robustness of the model, data augmentation was performed on the labeled dataset, including random rotation (±30°), brightness and contrast adjustment, and addition of Gaussian noise, to construct the final training dataset.

[0024] S2. Construct the STRAW-MAMBA model The STRAW-MAMBA model in this embodiment is an improvement on the YOLOv8-Pose architecture, and its overall structure is as follows: Figure 4 As shown, the core improvements lie in the backbone network and the neck network, and the specific construction process is as follows: S21. Construct and deploy the C2f-MAMBA module to enhance global and multi-scale feature extraction. like Figure 5 As shown, the C2f-MAMBA module is one of the core innovations of this invention, and its construction and deployment process is as follows: 1. Building the Straw-Vim module: Based on the efficient EfficientVim model, it introduces... Figure 7The multi-scale convolutional attention module (MSCA) shown is used to achieve multi-scale feature extraction, and the LayerScale mechanism is used to dynamically weight the residual connections to construct the Straw-Vim module; 2. Forming the C2f-MAMBA module: Replace the Bottleneck component in the C2f module of the YOLOv8-Pose model with the Straw-Vim module obtained above; 3. Deployment Module: The resulting C2f-MAMBA module will replace the first two C2f modules in the backbone network and the last C2f module in the neck network of the YOLOv8-Pose model. This move aims to leverage the network's powerful global modeling capabilities to better handle occlusion and complex background issues in the shallow and key stages of feature fusion. S22. Introducing an HWD module to preserve high-frequency detail features. To address the detail loss issue caused by traditional convolutional downsampling, this embodiment replaces all standard convolutions (Conv) in the backbone and neck network of the YOLOv8-Pose model (except for the first layer) with HWD (Haar Wavelet Downsampling) modules; such as Figure 9 As shown, the HWD module uses Haar wavelet transform to decompose the feature map into four sub-bands: LL (low frequency), LH (horizontal high frequency), HL (vertical high frequency), and HH (diagonal high frequency). While achieving downsampling, it preserves the high-frequency details of the image (such as edges and corners) to the greatest extent, which is crucial for accurately locating key picking points. S23. Introduce FasterNet Block to achieve model lightweighting. To further reduce the number of model parameters and computational cost, the last two C2f modules in the original backbone network of the YOLOv8-Pose model were replaced with FasterNet Blocks, such as... Figure 8 As shown, FasterNet Block uses partial convolution (PConv), which greatly improves computational efficiency while maintaining excellent feature extraction capabilities; at this point, the STRAW-MAMBA model is complete.

[0025] The algorithm flow of the C2f-MAMBA module is as follows (see reference). Figure 5 , Figure 6 The following are in order: a) Sequence modeling: Convert the input feature map into a sequence and capture long-range dependencies based on the state-space model (SSM).

[0026] b) Gating and Projection Optimization: The output of the SSM is gated and linearly projected using a Hidden State Mixer (HSM).

[0027] c) Multi-scale attention fusion: Introducing the Multi-scale Convolutional Attention (MSCA) module to perform multi-scale convolutional attention calculations on features, refining features and improving fusion accuracy.

[0028] S3, Model Training and Output The training dataset constructed in step S1 is input into the STRAW-MAMBA model for training. The AdamW optimizer is used for training, with the initial learning rate set to 1e-3 and adjusted using a cosine annealing strategy. After the model is trained, for the input strawberry RGB image, it can directly output the detection bounding box (center point coordinates, width and height), class confidence, and the two-dimensional pixel coordinates and visibility status of five picking key points of the strawberry fruit.

[0029] S4. Key Point Localization Reasoning and Information Fusion The RGB images captured in real time by the strawberry picking robot are input into the trained STRAW-MAMBA model to obtain the strawberry detection results and the initial two-dimensional pixel coordinates of five picking key points. The strawberry detection results include the strawberry bounding box, category, confidence level and picking key points. The center coordinates of the strawberry can be obtained by the pixel coordinates corresponding to the intersection of the two diagonals of the strawberry bounding box. For occluded key points, recovery calculations are performed based on the preset geometric constraints between the five harvesting key points, specifically including the following three scenarios: Scenario 1: The shear point is not visible Assuming it is located on the axis of the fruit stalk, with the upper vertex P known... top With lower vertex P bottom In the case of vectors Direction, according to formula P cut =P top +k×(P top -P bottom ), calculate the coordinates of the shear point, where the scaling factor k is obtained through statistical analysis of the labeled data; Scenario 2: Neither the shear point nor the upper vertex is visible. First, based on the left and right grab points P left P right Calculate the midpoint P of the connecting line. mid =(P lef t+P right ) / 2; Combined with vertex P bottom Based on the assumption of the direction of the fruit stalk, the upper vertex is calculated sequentially. P top =P mid +k1×(P mid -P bottom and shear point P cut =Ptop +k2×(P mid -P bottom The scaling factors k1 and k2 are determined through data fitting. Scenario 3: The clipping point, the top vertex, and any grab point are not visible. First, the fruit stem axis is estimated based on the visible grasp points and the lower vertex. The missing grasp points are then recovered through symmetric mapping. The problem is then transformed into case two, and finally the coordinates of all occluded key points are recovered. Simultaneously, the depth values ​​corresponding to the coordinates of all key points are obtained from the synchronously acquired depth images.

[0030] S5. Spatial Coordinate Calculation Combining the 2D pixel coordinates and depth values ​​of the keypoints obtained from S4, we perform 3D spatial coordinate calculations: S51. Calculate camera coordinates: Use camera intrinsic parameters to transform pixel coordinates to camera coordinates.

[0031] X camera =(u-cx)*depth(u,v) / fx; Y camera =(v-cy)*depth(u,v) / fy; Z camera =depth(u,v); Where (u,v) are pixel coordinates, depth(u,v) is the depth value at that pixel, (cx,cy) are the coordinates of the camera optical center, and (fx,fy) is the camera focal length. S51. Calculate the world coordinate system coordinates: Using the camera's extrinsic parameters (obtained through hand-eye calibration), locate point P in the camera coordinate system. camera Transform to world coordinate system: P world =R -1 ×(P camera -t); Where R and t are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, respectively. -1 Let P be the inverse of the rotation matrix R. camera P is a 3D coordinate vector in the camera coordinate system. world This is the three-dimensional coordinate vector in the transformed world coordinate system.

[0032] S6, Robot Deployment and Execution The trained STRAW-MAMBA model is deployed to the strawberry picking robot computing platform (such as Jetson AGXOrin). The robot control system receives the three-dimensional coordinates of the strawberry center and key picking points in the world coordinate system calculated by S5. The path planning module plans the movement trajectory of the robotic arm accordingly, and finally controls the end effector (such as gripper and scissors) to complete the automated picking operation.

[0033] A deep learning-based model for strawberry fruit detection and key point localization during picking is proposed, which is the STRAW-MAMBA model constructed in the above method.

[0034] Experiment and effect verification To verify the effectiveness of the present invention, ablation experiments and comparative experiments were conducted. Ablation Experiment Analysis: To verify the specific contributions of each module in this invention, we designed a systematic ablation experiment, as shown in Tables 1 and 2. The experiment used the standard YOLOv8-Pose model as the baseline and gradually introduced the C2f-MAMBA, FasterNetBlock, and HWD modules for evaluation. The results show that with the sequential addition of modules, the model's mAP@0.5 performance on keypoint localization and fruit detection tasks continuously improved, while the number of model parameters significantly decreased to 1.9M. The final combined model improved the mAP@0.5 performance on keypoint localization by 2.3 percentage points and on detection tasks by 2.1 percentage points compared to the baseline. This result fully verifies the effectiveness and synergistic effect of each innovative module in improving model performance and achieving lightweight design.

[0035] Table 1: Ablation Experiment Results for Key Point Location in Harvesting Task

[0036] Table 2: Ablation Experiment Results in Strawberry Fruit Detection Task

[0037] Comparative Experiment Results: To comprehensively evaluate the advancement of the model of this invention, we conducted comparative experiments with current mainstream models; such as... Figure 10 and Figure 11 As shown, the STRAW-MAMBA model achieves an mAP@0.5 of 84.9% on the keypoint localization task, significantly outperforming the standard YOLOv8-Pose (82.6%) and the latest STRAW-YOLO and other comparable models. The visualization results further demonstrate that the model of this invention is more accurate and stable in locating strawberry fruits and key points for picking in complex scenes, proving its comprehensive advantages in global feature extraction and detail preservation.

[0038] Effective visualization of the receptive field: such as Figure 12As shown, compared with the original YOLOv8-Pose model, the STRAW-MAMBA model of this invention has a wider effective receptive field and is more focused on the target itself, demonstrating its advantage in global feature extraction capabilities.

[0039] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for strawberry fruit detection and key point localization based on deep learning, characterized in that, Includes the following steps: S1. Data Acquisition and Preprocessing: RGB images and depth images of strawberries are acquired simultaneously using a depth camera. The RGB images are labeled and data augmented to construct a training dataset. S2. Constructing the STRAW-MAMBA model: Based on the YOLOv8-Pose model architecture, the C2f-MAMBA module is constructed to enhance global and multi-scale feature extraction, the HWD module is introduced to retain high-frequency detail features, and the FasterNet Block is used for lightweight design to form the STRAW-MAMBA model. S3. Model Training and Output: Input the training dataset into the STRAW-MAMBA model for training, so that the model can output the detection bounding box of strawberry fruit and the coordinates of five picking key points; S4. Key point localization reasoning and information fusion: The real-time acquired strawberry RGB image is input into the trained STRAW-MAMBA model to obtain the strawberry detection result and the initial two-dimensional pixel coordinates of five picking key points; for occluded key points, based on the preset geometric constraints between the five picking key points, their two-dimensional pixel coordinates are recovered by calculation, and the depth values ​​corresponding to all key points are obtained from the synchronously acquired depth image. S5. Spatial coordinate calculation: Combining the two-dimensional pixel coordinates of the key points, the corresponding depth values, and the camera parameters, calculate the three-dimensional coordinates of the strawberry center and each picking key point in the camera coordinate system and the world coordinate system.

2. The method for strawberry fruit detection and key point localization based on deep learning according to claim 1, characterized in that, In step S1, the depth camera is a depth camera based on structured light or time-of-flight ranging.

3. The method for strawberry fruit detection and key point localization based on deep learning according to claim 1, characterized in that, In step S1, the annotation adopts the five-point annotation method, and the annotation content includes the strawberry bounding box, category, confidence level, and key points of picking; The categories include two types: those that can be picked and those that cannot be picked. The key points for picking are the cutting point, the upper vertex, the lower vertex, the left gripping point, and the right gripping point.

4. The method for strawberry fruit detection and key point localization based on deep learning according to claim 1, characterized in that, In step S2, constructing the STRAW-MAMBA model includes: S21. Construct and deploy the C2f-MAMBA module to enhance global and multi-scale feature extraction: Based on the EfficientVim model, a multi-scale convolutional attention module is introduced to achieve multi-scale feature extraction, and the LayerScale mechanism is used to dynamically weight the residual connections to construct the Straw-Vim module. The constructed Straw-Vim module is used to replace the Bottleneck component in the C2f module of the YOLOv8-Pose model, forming the C2f-MAMBA module. Replace the first two C2f modules in the backbone network and the last C2f module in the neck network of the YOLOv8-Pose model with the C2f-MAMBA module. S22. Introducing an HWD module to preserve high-frequency detail features: Replace the standard convolutions in the backbone and neck networks of the YOLOv8-Pose model, except for the first layer, with HWD modules; S23. Introduce FasterNet Block to achieve model lightweighting: Replace the last two C2f modules in the backbone network of the YOLOv8-Pose model with FasterNet Blocks to complete the model construction.

5. The method for strawberry fruit detection and key point localization based on deep learning according to claim 4, characterized in that, The algorithm flow of the C2f-MAMBA module includes the following steps: a) Sequence modeling of input features based on the State-Space Model (SSM) to capture long-range dependencies; b) Optimize the gating mechanism and output projection of features processed by SSM using the Hidden State Mixer (HSM); c) Introduce the Multi-Scale Convolutional Attention (MSCA) module to perform multi-scale convolutional attention calculations on the optimized features to improve the accuracy of feature fusion.

6. The method for strawberry fruit detection and key point localization based on deep learning according to claim 4, characterized in that, The HWD module decomposes and reconstructs the feature map through Haar wavelet downsampling, which preserves high-frequency detail features of the image while achieving feature downsampling and reducing the loss of detail information.

7. The method for strawberry fruit detection and key point localization based on deep learning according to claim 1, characterized in that, In step S4, the process of recovering the two-dimensional pixel coordinates based on the preset geometric constraints between the five key picking points specifically includes the following scenarios: When the shearing point is not visible, the coordinates of the shearing point are calculated based on the coordinates of the upper and lower vertices and the vector from the lower vertex to the upper vertex. When neither the cutting point nor the top vertex is visible, the coordinates of the cutting point and the top vertex are derived and calculated using the coordinates of the midpoint of the line segment connecting the left and right grab points: When the cut-off point, top vertex, and one grab point are all invisible, first calculate the coordinates of the missing grab point using the coordinates of the visible key points, and then deduce the coordinates of the remaining occluded points in the same way as in the case where the cut-off point and top vertex are both invisible.

8. The method for strawberry fruit detection and key point localization based on deep learning according to claim 1, characterized in that, In step S5, the spatial coordinate calculation specifically includes: S51. Based on the strawberry depth image and camera intrinsic parameters, calculate the three-dimensional coordinates of the pixels in the camera coordinate system: X camera =(u-cx)*depth(u,v) / fx; Y camera =(v-cy)*depth(u,v) / fy; Z camera =depth(u,v); Where (u,v) are pixel coordinates, depth(u,v) is the depth value at that pixel, (cx,cy) are the coordinates of the camera optical center, and (fx,fy) is the camera focal length. S52. Using the camera's extrinsic parameter matrix, the three-dimensional point P in the camera coordinate system... camera Transform to world coordinate system: P world =R -1 ×(P camera -t); Where R and t are the rotation matrix and translation vector from the world coordinate system to the camera coordinate system, respectively. -1 Let P be the inverse of the rotation matrix R. camera P is a 3D coordinate vector in the camera coordinate system. world This is the three-dimensional coordinate vector in the transformed world coordinate system.

9. The method for strawberry fruit detection and key point localization based on deep learning according to claim 1, characterized in that, It also includes S6, robot deployment and execution: deploying the STRAW-MAMBA model to the strawberry picking robot platform, and the robot completes the automated picking operation based on the key point positions in the world coordinate system.

10. A deep learning-based model for strawberry fruit detection and key point localization during picking, characterized in that, The model is the STRAW-MAMBA model constructed using any of the methods described in claims 1-9.

Images