Asparagus harvesting robot vision recognition closed loop system and deep learning detection method adaptive to cross-illumination environment

Abstract

The application provides a cross-illumination environment adaptive asparagus harvesting robot visual identification closed-loop system and a deep learning detection method. First, asparagus image acquisition, labeling and image enhancement processing are performed, then an improved deep learning network is used to identify asparagus and generate asparagus masks, the improved deep learning network includes an improved convolutional neural network architecture with added CBAM attention mechanism and spatial pyramid pooling structure, and a YOLACT++ instance segmentation network with improved anchor box; then candidate asparagus masks are extracted, combined with three-dimensional coordinate data of a depth camera to calculate asparagus stem length and base diameter, and then to identify asparagus maturity; finally, according to the spatial pose parameters of mature asparagus, the cutting surface coordinates are calculated, and the maturity identification result and cutting surface positioning data are transmitted to the asparagus harvesting robot. The application can ensure that the asparagus harvesting robot can efficiently and accurately complete the asparagus identification, segmentation, maturity identification and cutting surface positioning tasks during the working process.

Description

Technical Field

[0001] This invention belongs to the field of agricultural intelligent equipment and machine vision technology, specifically relating to a visual recognition closed-loop system and deep learning detection method for asparagus harvesting robots that are adaptable to different lighting environments. Background Technology

[0002] Asparagus, known as the "King of Vegetables," is rich in vitamins, trace elements, and amino acids, possessing extremely high nutritional and economic value. my country is the world's largest asparagus producer and exporter, accounting for over 90.6% of global asparagus cultivation. However, asparagus harvesting has long relied on manual labor, making it a typical labor-intensive activity. The asparagus harvesting season lasts up to five months, and the rapid growth rate, coupled with the inconsistent maturity times of different plants, necessitates daily assessment of maturity and selective harvesting. Furthermore, the tender asparagus stems are fragile and easily damaged, with a diameter of only 520mm, demanding extremely high precision and gentleness in harvesting.

[0003] Currently, the research and development of green asparagus harvesting robots has made some progress. Existing technology (CN202411488746.1) discloses an intelligent integrated harvesting and transportation device for green asparagus. This upgraded model has been demonstrated in the field at an organic asparagus planting base in Fuzhou, Jiangxi Province. It focuses on fully autonomous harvesting of green asparagus, taking into account both harvesting efficiency and the protection of tender stems. Except for the camera, the core components are all proprietary. Based on a tracked mobile chassis, it is equipped with a high-definition depth camera and intelligent recognition system, a lightweight robotic arm, an asparagus collection bin and a control cabinet. The visual recognition system avoids interference from weeds and leaves and identifies mature green asparagus. The control system enables fully autonomous walking, harvesting and collection. The harvesting time for a single asparagus is about 9.5 seconds. It can operate around the clock and is suitable for large-scale open-field planting scenarios. However, this model still suffers from several pain points: low harvesting efficiency and lack of multi-arm collaboration, making it difficult to meet the rapid harvesting needs during the harvest season; unstable visual recognition and underground cutting point positioning accuracy in complex field environments, easily leading to missed harvests, misharvesting, and improper cutting; insufficient precision in controlling the clamping force of the end effector, resulting in damage to tender stems due to compression; lack of optimization for facility cultivation scenarios, reliance on imported core sensors leading to high costs, and insufficient navigation stability of the tracked platform, making it prone to collisions with plants, thus hindering large-scale promotion. In addition, existing technologies generally suffer from problems such as a single maturity assessment standard, large cutting surface positioning errors, low model training efficiency, and weak anti-interference capabilities, resulting in low harvesting success rates and high damage rates, making it difficult to meet the actual needs of precise harvesting of asparagus in facilities.

[0004] In response to these core problems existing in the prior art, this invention conducts targeted research and development to break through technical bottlenecks and provide a green asparagus harvesting solution with higher harvesting efficiency, stronger environmental adaptability, more accurate positioning, and more controllable costs. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a visual recognition closed-loop system and deep learning detection method for asparagus harvesting robots that are adaptable to different lighting environments. This system ensures that the asparagus harvesting robot can efficiently and accurately complete tasks such as asparagus identification, segmentation, maturity determination, and cutting surface positioning during operation.

[0006] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0007] A deep learning detection method for asparagus harvesting robots that adapt to different lighting environments includes the following steps:

[0008] Step 1: Acquire asparagus images using a depth camera;

[0009] Step 2: Image annotation of asparagus;

[0010] Step 3: Asparagus image enhancement processing;

[0011] Step 4: Use an improved deep learning network to identify asparagus and generate an asparagus mask; the improved deep learning network includes an improved CNN architecture with added CBAM attention mechanism and SPP structure, and an improved anchor box YOLACT++ instance segmentation network; the improved CNN architecture is used to extract image features from the input image, and the improved anchor box YOLACT++ instance segmentation network is used to generate the asparagus mask;

[0012] Step 5: Extract candidate asparagus masks, and calculate the length of asparagus shoots and base diameter by combining the 3D coordinate data from the depth camera;

[0013] Step 6: Based on the pre-set maturity judgment criteria, and combined with the parameters of asparagus tender stem length and base diameter, determine the maturity of asparagus;

[0014] Step 7: Calculate the three-dimensional coordinates of the harvesting cutting surface based on the spatial pose parameters of mature asparagus, and transmit the maturity judgment results and cutting surface positioning data to the asparagus harvesting robot to provide a data basis for the harvesting action control of the asparagus harvesting robot.

[0015] Furthermore, the specific process of step 5 is as follows:

[0016] Step 5.1: Calibrate the depth camera to obtain the intrinsic parameter matrix and distortion matrix, and use the calibration parameters to convert the pixel coordinates of the asparagus mask into three-dimensional coordinates in the camera coordinate system;

[0017] Step 5.2: Pixel Coordinate System The origin is the top left corner of the image. shaft and The axes represent the two right-angled sides parallel to the image plane; first, the asparagus mask in the pixel coordinate system is obtained. The coordinates of the highest and lowest points of the axis projection are then used to determine the target asparagus. Divided equally along the axial projection direction Divide the data into equal parts and calculate the coordinates of the intersections of each division. Obtain the initial point coordinates ( , )and( , The initial point refers to the pixel point on the left and right edges of the asparagus cross-section after the asparagus is evenly divided into segments along its length on the pixel mask outline. This indicates the segment number of the asparagus. =[1, -1], This indicates the sequence number of asparagus spears in the image;

[0018] Step 5.3: Calculate the coordinates of each skeleton key point using the initial point coordinates. , )of The component, the skeleton key point refers to the feature point on the central skeleton line of the asparagus, which is obtained by averaging two initial points on the left and right edges of the same segment;

[0019] Step 5.4: Establish a fitted straight line using the key points of the skeleton in the first three segments;

[0020] Step 5.5: Determine the coordinates of the highest point of the asparagus. Simultaneously, the intersection point of the fitted straight line and the asparagus mask contour with the largest v component is taken as the coordinate of the lowest point of the asparagus. Then calculate the length of the tender asparagus stems;

[0021] Step 5.6: Calculate the roll angle and pitch angle of the asparagus, determine the posture of the asparagus on the soil ridge, and calculate the diameter of the base of the asparagus in combination with the roll angle.

[0022] Furthermore, in step 5.5, the length of the asparagus tender stems... for:

[0023]

[0024] In the formula, Indicates the first The length of the tender asparagus stems of root asparagus. Indicates the first The first asparagus The three-dimensional coordinates of each skeleton point in the camera coordinate system Indicates the first The first asparagus The three-dimensional coordinates of a skeleton point in the camera coordinate system, the skeleton point includes One key skeletal point, one highest point, and one lowest point; =[0, ], hour, Representing the The coordinates of the lowest point of the bottom of the asparagus stalk in the camera coordinate system. hour, Representing the The three-dimensional coordinates of the highest point of the asparagus root in the camera coordinate system.

[0025] Furthermore, in step 5.6, the lowest point of the asparagus bottom in the camera coordinate system is... Establish a spatial rectangular coordinate system OXYZ with the origin as the origin. The roll angle is the angle between the projection of the fitted line onto the OXY plane and the X-axis, and the pitch angle is the angle between the projection of the fitted line onto the OYZ plane and the Z-axis; as detailed below:

[0026]

[0027]

[0028] in, Indicates the first The corner of a stalk of asparagus, Indicates the first The pitch angle of a single asparagus shoot. Indicates the first The three-dimensional coordinates of the lowest point at the bottom of the asparagus stalk in the camera coordinate system. Indicates the first The three-dimensional coordinates of the second key skeletal point of the asparagus root in the camera coordinate system.

[0029] Furthermore, the diameter of the asparagus base is calculated as follows:

[0030]

[0031] in, Indicates the diameter at the base of the asparagus; The original value of the diameter parallel to the X-axis of the depth camera is calculated based on the 3D coordinates of the initial points of the first and second segments of the asparagus bottom contour, as follows:

[0032]

[0033] in, Indicates the first position in the camera coordinate system The three-dimensional coordinates of the initial left point of the outline at the junction of the first and second segments at the bottom of the asparagus root. Indicates the first position in the camera coordinate system The three-dimensional coordinates of the initial point on the right side of the outline at the junction of the first and second segments at the bottom of the asparagus root.

[0034] Furthermore, in step 6, the maturity determination criteria are as follows:

[0035] Young bamboo shoots (base diameter < 8mm): those with a tender stem length ≥ 20cm are considered mature, and those with a tender stem length < 20cm are considered immature;

[0036] Medium-sized bamboo shoots (8mm ≤ base diameter < 13mm): tender stems ≥ 24cm are considered mature, tender stems < 24cm are considered immature;

[0037] Large bamboo shoots (base diameter ≥13mm): those with a tender stem length ≥28cm are considered mature, and those with a tender stem length <28cm are considered immature;

[0038] For all sizes of asparagus: tender stems ≥40cm in length are considered overripe and are not included in the harvesting scope.

[0039] Furthermore, the specific process of step 7 is as follows:

[0040] Step 7.1: Based on the coordinates of the lowest point at the bottom of the asparagus and the key points of the 2nd to 3rd skeletal segments, calculate the 3D coordinates of the cutting point:

[0041] ( =3 / 20)

[0042] in,( () represents the coordinates of the cutting point in the camera coordinate system;

[0043] Step 7.2: The cutting surface parameters are expressed as follows:

[0044]

[0045] The cutting parameters are transmitted to the asparagus harvesting robot via RS485 or TCP / IP interface, which guides the six-degree-of-freedom robotic arm to adjust the attitude of the end effector so that the gripping fingers are parallel to the asparagus axis, and complete the flexible gripping and shearing harvesting at the cutting surface.

[0046] Furthermore, in step 4, the improved deep learning network also includes a branch for enhancing the texture features of asparagus tips, used to identify whether the asparagus tips are tight or loose. The identification results are transmitted to the asparagus harvesting robot to guide the robot's end effector to adaptively adjust the clamping force. The specific identification method is as follows:

[0047] Feature selection and enhancement: From the P3 and P4 feature maps output by the improved CNN architecture FPN feature pyramid network, features corresponding to the shoot tip region are selected. A self-designed texture feature enhancement operator is used to strengthen the texture details of the shoot tip region and suppress interfering features from the background and other areas of the asparagus stem.

[0048]

[0049] in, This indicates the enhanced texture image in pixels. The output value at that location, Represents the coordinates of each pixel in the image. Indicates the horizontal position of a pixel. Indicates the vertical position of a pixel; express Gaussian filtering; This represents the gradient operator, used to extract the texture of the bamboo shoot tip edge; Indicates a directional mask, used to enhance texture along the growth direction of the bamboo shoot tip;

[0050] Category recognition and output: The enhanced bamboo shoot tip features are input into the classification branch, and the class probability is output through the Softmax function. When the probability is greater than 95%, it is determined to be the corresponding class.

[0051]

[0052] when When the value is greater than 0.95, it is considered a tight end;

[0053] when A value greater than 0.95 is considered a divergent trend.

[0054] in, This indicates that the bamboo shoot tips belong to the first... The probability value of the class; Indicates the category number, 1 represents a tight head, and 2 represents a loose head; This indicates the probability that the bamboo shoot tip is the tightest point. This represents the probability that the bamboo shoot tip is a scattered head; This represents the original predicted value output by the classification branch; This represents the original prediction score of the improved deep learning network for the "tight head"; This represents the original prediction score of the improved deep learning network for "scattered heads"; This indicates the operation of the natural exponent.

[0055] A visual recognition closed-loop system for asparagus harvesting robots that adapts to different lighting environments, used to implement the aforementioned deep learning detection method for asparagus harvesting robots, includes:

[0056] The image acquisition module uses a depth camera, LED fill light, image stabilization module and data transmission unit to acquire asparagus image data in the facility asparagus cultivation environment, and uploads it to the computer for further processing by other modules;

[0057] The dataset construction module preprocesses the image data acquired by the image acquisition module to build a dataset that will later serve as input to the improved deep learning network. The preprocessing includes asparagus image enhancement and annotation file updates.

[0058] The network model building module builds an improved deep learning network to identify asparagus in images and generate asparagus masks, while also identifying whether the asparagus tips are tight or loose.

[0059] The asparagus segmentation and parameter calculation module extracts the asparagus mask generated by the network model building module, combines the camera calibration parameters to convert the pixel coordinates into three-dimensional coordinates, and calculates the asparagus tender stem length, base diameter, and spatial pose parameters.

[0060] The maturity determination and positioning module accurately determines the maturity based on the set maturity determination criteria and the parameters of asparagus length and base diameter. It calculates the three-dimensional coordinates of the harvesting cutting surface according to the spatial pose parameters and transmits the maturity determination results, the harvesting cutting surface positioning data, and the asparagus tight head and loose head category identification results to the asparagus harvesting robot.

[0061] The model optimization module employs transfer learning strategies and model lightweighting techniques to optimize the model, adapting it to the edge deployment requirements of the asparagus harvesting robot.

[0062] The system deploys an interactive module, provides standardized API interfaces, supports integration with asparagus harvesting robot control systems and planting management platforms, and is also equipped with a visual interactive unit and a distributed data storage unit.

[0063] The present invention has the following beneficial effects:

[0064] This invention combines machine vision, deep learning, and 3D localization technologies to achieve automatic identification, segmentation, maturity assessment, and cut surface localization of asparagus in facility environments. By designing an improved CNN architecture with added CBAM attention mechanism and SPP structure, this invention can extract rich edge and semantic features of asparagus, strengthen the expression of key region features, and improve pixel classification accuracy. Furthermore, by improving the anchor box design of the YOLACT++ instance segmentation network, this invention adapts to the slender shape of asparagus and effectively generates asparagus masks. Based on a two-parameter maturity assessment method using asparagus length and base diameter, combined with spatial pose estimation, this invention achieves precise cut surface localization with high accuracy and precision. This invention provides efficient and reliable visual support for asparagus harvesting robots, significantly improving harvesting efficiency and quality, reducing labor intensity, and possessing high application value. Attached Figure Description

[0065] Figure 1 This is a flowchart of the visual recognition and detection method for asparagus harvesting robots adapted to different lighting environments as described in this invention.

[0066] Figure 2 This is a schematic diagram of the improved deep learning network described in this invention;

[0067] Figure 3 A schematic diagram of the CBAM attention mechanism module;

[0068] Figure 4 This is a schematic diagram of the SPP structure;

[0069] Figure 5 This is a schematic diagram illustrating the initial point acquisition and skeleton fitting.

[0070] Figure 6 This is a schematic diagram of the pixel coordinate system;

[0071] Figure 2 In this diagram, Backbone represents the main network, Mask Prototypes Generate Network represents the mask prototype generation branch, Prediction Head represents the prediction branch, C1~C5 are feature maps at different stages of the Backbone, P3~P7 are multi-scale feature layers output by the Feature Pyramid Network (FPN), with P3 suitable for detecting small targets like asparagus tips, and P5~P7 suitable for detecting medium / large targets like asparagus; Conv 3×3 represents a 3×3 convolutional layer used for further feature extraction and channel adjustment of the feature maps; Classification represents the classification branch, outputting the class probability of the asparagus target; Box represents the bounding box branch, outputting the location and size of the detection box for the asparagus target; Mask coefficients represent the mask coefficient branch, outputting the mask weights corresponding to each detection box, used for fusion with the prototype mask; NMS (Non-Maximum Mask)... Suppression represents non-maximum suppression, removing duplicate detection boxes and retaining the optimal asparagus target detection result; Crop represents cropping operation, cropping the corresponding asparagus region from the feature map based on the detection box; Threshold represents thresholding operation, fusing the mask coefficients with the prototype mask and obtaining a binary asparagus mask through thresholding; Conv3×3×3 represents three consecutive 3×3 convolutional layers used to extract local features of asparagus; Upsample + Conv3×3 represents upsampling + 3×3 convolution, restoring the low-resolution feature map to high resolution and generating a more refined mask prototype; Conv1×1 represents a 1×1 convolutional layer, adjusting the number of channels and outputting the final mask prototype;

[0072] Figure 3 In this context, Max Pool represents the max pooling operation; Avg Pool represents the average pooling operation; Sigmoid represents the Sigmoid activation function; and Conv represents a convolutional layer.

[0073] Figure 4 In the table, W, H, and C represent width, height, and number of channels, respectively; MaxPool 5×5 represents max pooling with a kernel size of 5×5; MaxPool 13×13 represents max pooling with a kernel size of 13×13; and Concat represents feature concatenation operation. Detailed Implementation

[0074] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0075] Reference Figure 1 As shown, the visual recognition and detection method for asparagus harvesting robots adapted to different lighting environments according to the present invention includes the following process:

[0076] Step 1: Asparagus image acquisition;

[0077] Using an Intel RealSense D435 depth camera, LED fill light, image stabilization module, and data transmission unit, asparagus image data was collected in a facility-grown asparagus environment, including image data under different weather and lighting conditions. The collected image data was transmitted to a computer via the data transmission unit, and the normalized original image size was 640×480 pixels or 550×550 pixels. The various modules on the computer then performed subsequent analysis and processing to achieve asparagus identification, maturity determination, and cut surface positioning.

[0078] The depth camera has a resolution of ≥1080P, an imaging distance range of 0.6~1m, and an imaging height of 40cm. It supports simultaneous acquisition of static images and dynamic video streams with a frame rate of ≥25fps. The data transmission unit adopts a USB3.0 or Ethernet interface, with a transmission delay of ≤50ms and data encryption function.

[0079] Step 2: Image annotation of asparagus;

[0080] First, the asparagus in the image is manually judged as mature and ready for picking, immature and not ready for picking, or overripe. Then, its outline is annotated. LabelImg software is used to annotate the asparagus tip type (tight tip, loose tip), and the annotation file is saved in XML format. LabelMe software is used to annotate the asparagus outline, and the annotation file is saved in JSON format. This will be used to train an improved deep learning network.

[0081] Step 3: Asparagus Image Enhancement. Image enhancement includes performing image enhancement processing on the original image and updating and adjusting the generated file after annotation, as detailed below:

[0082] Step 3.1: In order to avoid overfitting during model training, improve the robustness and generalization of the model for asparagus recognition and segmentation, solve the problem of low training accuracy caused by image noise, image blur, and changes in lighting, and ensure that the model can adapt to the asparagus recognition needs in complex field environments, use geometric deformation methods such as rotation, cropping, horizontal flipping, translation, and motion blur to change the original image.

[0083] Step 3.2: Optimize image features using pixel-level enhancement methods such as adding Gaussian noise, salt-and-pepper noise, and randomly changing brightness and contrast;

[0084] Step 3.3: Synchronously update the coordinate data and mask information in the annotation file according to the transformation method of the image to ensure that the annotation file matches the image features;

[0085] Step 3.4: Divide the original dataset and the enhanced dataset into training set, validation set and test set in a 6:2:2 ratio, and use them as input for the subsequent improved deep learning network.

[0086] Step 4: Use as follows Figure 2 The improved deep learning network shown identifies asparagus and generates asparagus masks; the improved deep learning network includes an improved convolutional neural network (CNN) architecture with added CBAM (Convolutional Block Attention Module) attention mechanism and spatial pyramid pooling (SPP) structure, and an improved anchor box YOLACT++ instance segmentation network;

[0087] The improved deep learning network processes the input image in two parts: image feature extraction and asparagus mask generation, as detailed below:

[0088] Step 4.1: Image Feature Extraction; Features of the input image are extracted using an improved CNN architecture that incorporates the CBAM attention mechanism and SPP structure. The specific structural arrangement of the improved CNN architecture is as follows:

[0089] The backbone network adopts DCNv2-ResNet50, which outputs three levels of feature maps C3, C4, and C5 sequentially. DCNv2-ResNet50 is a deep learning model architecture that combines deformable convolutional network v2 (DCNv2) and ResNet-50 backbone network, and is used for the visual recognition task of green asparagus harvesting robot. CBAM attention mechanism module is set after feature maps C3 and C4, and SPP structure is set after feature map C5.

[0090] The implementation process of the CBAM attention mechanism module satisfies:

[0091] ,

[0092] The channel attention weights are:

[0093]

[0094] Spatial attention weights are:

[0095]

[0096] In the formula, Indicates the input feature layer. This indicates the channel attention weighting result. Indicates the output feature layer. This indicates the channel attention module. This represents convolution multiplication. This indicates a spatial attention module. Indicates a shared fully connected layer. Indicates average pooling. This indicates max pooling. This represents the Sigmoid activation function. This represents a 7×7 convolutional layer.

[0097] like Figure 3 As shown, the CBAM attention mechanism module strengthens the feature channel weights of asparagus through channel attention and focuses on the main areas of the shoot tip and tender stem through spatial attention, suppressing background interference such as weeds and soil ridges; Figure 4 As shown, the SPP structure performs max pooling on the C5 feature map using pooling kernels of sizes 5×5, 9×9, and 13×13. After fusing features from different receptive fields, it is concatenated with the original feature map. After adjusting the number of channels through a 3×3 convolutional layer, the fused feature map is output, enriching the local feature expression of asparagus. The key feature expression of asparagus is enhanced through a feature fusion strategy. The FPN (Feature Pyramid Network) is used to extract features from the fused feature map at multiple scales, outputting five-level feature maps of P3, P4, P5, P6, and P7, which are used to adapt to the detection and segmentation of asparagus targets of different sizes.

[0098] Step 4.2: Generate the asparagus mask: Generate the asparagus mask using the improved anchor frame YOLACT++ instance segmentation network. The specific process is as follows:

[0099] The aspect ratio of the original anchor boxes in the FPN feature map of the YOLACT++ algorithm is improved from {1:1, 1:2, 2:1} to {2:1, 4:1, 16:1} to adapt to the slender shape of asparagus and ensure coverage of asparagus targets of different sizes and postures. Using the P3 to P7 five-level feature maps output by the improved CNN architecture, 100 asparagus mask prototypes are generated through the mask prototype generation branch. The bounding box, category and mask coefficients corresponding to the mask prototypes are obtained through the prediction branch. The mask coefficients are multiplied by the mask prototypes to obtain the final asparagus mask. The non-maximum suppression (NMS) algorithm is used to remove redundant masks and output an accurate single asparagus mask.

[0100] Step 5: Extract candidate asparagus masks and, using the 3D coordinate data from the depth camera, calculate the length of the asparagus shoots and the diameter at the base; details are as follows:

[0101] Step 5.1: Calibrate the depth camera using Zhang Zhengyou's chessboard calibration method to obtain the intrinsic parameter matrix and distortion matrix:

[0102] Intrinsic parameter matrix for:

[0103]

[0104] Distortion matrix for:

[0105]

[0106] The pixel coordinates of the asparagus mask are converted into three-dimensional coordinates in the camera coordinate system using calibration parameters;

[0107] in, Represents the first-order radial distortion coefficient. Represents the second-order radial distortion coefficient. Represents the first-order tangential distortion coefficient. This represents the second-order tangential distortion coefficient. Represents the third-order radial distortion coefficient. Indicates the camera's intrinsic focal length. Indicates the camera's intrinsic focal length. This represents the x-coordinate value of the camera's intrinsic optical center. This represents the ordinate value of the optical center of the camera's intrinsic parameters.

[0108] Step 5.2: As Figure 6 As shown, pixel coordinate system It is a 2D Cartesian coordinate system, with its origin at the top left corner of the image. shaft and The axes represent the two right-angled sides that are parallel to the image plane. First, obtain the asparagus mask in the pixel coordinate system. The coordinates of the highest and lowest points of the axis projection:

[0109]

[0110]

[0111] In the formula, Represents the pixel coordinate system. Asparagus roots The coordinates of the lowest point of the axis projection. Represents the pixel coordinate system. Asparagus roots The coordinates of the highest point of the axis projection. Represents the pixel coordinate system. Asparagus roots Axis projection contour coordinates, =1, 2, 3…, This indicates the sequence number of asparagus roots in the image.

[0112] Target asparagus Divided equally along the axial projection direction The coordinates of the intersections of the equal segments (the first segment being the one closest to the soil surface) are calculated using the following formula:

[0113]

[0114] In the formula, Represents the pixel coordinate system. Asparagus roots In the axial projection Section and the The coordinates of the segment boundary, where The values ​​were set to 3, 6, and 9 in the preliminary experiment. The measurement error ratio of asparagus length when taking 6 and 9 is... =3 is small, but =9 The computational load is large when the value is 6, so considering all factors, we will... If the value is set to 6, then =[1, -1]. According to Iterate through the asparagus mask contour to obtain the initial point coordinates on the corresponding target contour. , )and( , ),like Figure 5 The (shown) , )and( , The initial point coordinates are the coordinates of one segment of the first asparagus. The initial point refers to the pixel point on the left and right edges of the cross-section of the asparagus after the asparagus is evenly divided along the length of the asparagus on the pixel mask contour. Location: Contour points on both sides of the tender asparagus stem, perpendicular to the asparagus growth direction. Source: Contour extreme points and segment intersections directly extracted from the segmentation mask. Function: Used to calculate the asparagus diameter, fit the center line, and solve the spatial pose.

[0115] Step 5.3: Calculate the coordinates of each skeleton key point using the initial point coordinates. of Quantity:

[0116]

[0117] Among them, the key points of the skeleton refer to the feature points on the central skeleton line of the asparagus, which are obtained by averaging the two initial points on the left and right at the same segment. In this embodiment, =6, then there are a total of 5 key points in the skeleton, such as Figure 5 The (shown) , ), ( , ), ( , ), ( , These represent the coordinates of the 2nd, 3rd, 4th, and 5th skeletal keypoints on the 1st asparagus spear in the pixel coordinate system. , This represents the coordinates of the lowest point of the bottom of the first asparagus spear in pixel coordinates. , () represents the coordinates of the highest point of the top of the first asparagus in the pixel coordinate system; In pixel coordinates, the asparagus is at the th Section and the Coordinates of key points of the skeleton at the segment boundary Quantity.

[0118] Step 5.4: Through the first 3 key points of the skeleton ( , ), ( , Establish the fitted line:

[0119]

[0120] in:

[0121]

[0122]

[0123] In the formula, This represents the slope of the fitted line. This represents the intercept of the fitted line.

[0124] Step 5.5: The highest point of the asparagus remains unchanged in this embodiment. The value is set to 6, therefore the coordinates of the highest point are represented as follows: However, since asparagus grows from the soil surface and its bottom is often obscured by soil, resulting in an irregular shape, using the maximum value of the v-axis projection as the lowest point would lead to significant deviations in length measurement and pose estimation of the asparagus outline, affecting the judgment of mature asparagus and the positioning of the cutting surface. Therefore, it is necessary to reselect the lowest point; the intersection of the fitted line in step 5.4 and the point with the largest v-component of the asparagus mask outline is taken as the new lowest point. Re-determine the coordinates of the lowest point at the bottom of the asparagus. Then combine the coordinates of the highest point Calculate the length of asparagus tender stems:

[0125]

[0126] In the formula, Indicates the first The length of the tender asparagus stems of root asparagus. Indicates the first The first asparagus One skeleton point (skeleton points include) The 3D coordinates of the skeleton key points, the highest point, and the lowest point in the camera coordinate system. Indicates the first The first asparagus The three-dimensional coordinates of each skeleton point in the camera coordinate system =[0, ], hour, Representing the The coordinates of the lowest point of the bottom of the asparagus stalk in the camera coordinate system. hour, Representing the The three-dimensional coordinates of the highest point of the asparagus root in the camera coordinate system.

[0127] Step 5.6: Based on the 3D coordinates of the initial points of the first and second segments of the asparagus bottom contour, calculate the original diameter value parallel to the X-axis of the depth camera. :

[0128]

[0129] In this embodiment, the roll angle and pitch angle are used to determine the posture of the asparagus on the soil ridge, with the lowest point of the asparagus bottom in the camera coordinate system as the reference. Establish a spatial rectangular coordinate system OXYZ with the origin, where the roll angle is the angle between the projection of the fitted line established in step 5.4 onto the OXY plane and the X-axis, and the pitch angle is the angle between the projection of the fitted line established in step 5.4 onto the OYZ plane and the Z-axis; the formulas for calculating the roll angle and pitch angle are as follows:

[0130]

[0131]

[0132] In the formula, Indicates the first The corner of a stalk of asparagus, Indicates the first The pitch angle of a single asparagus shoot. Indicates the first The three-dimensional coordinates of the lowest point at the bottom of the asparagus stalk in the camera coordinate system. Indicates the first The three-dimensional coordinates of the second key skeletal point of the asparagus root in the camera coordinate system.

[0133] Combined with roll angle Calculate the diameter of the asparagus base:

[0134]

[0135] In the formula: This indicates the diameter at the base of the asparagus. This represents the three-dimensional coordinates of the initial left point of the outline at the junction of the first and second segments at the bottom of the asparagus, in the camera coordinate system. This represents the three-dimensional coordinates of the initial right point of the outline at the junction of the first and second segments at the bottom of the asparagus in the camera coordinate system.

[0136] Step 6: Based on the established maturity criteria, and combined with the asparagus stem length and base diameter parameters calculated in Step 5, accurately determine the maturity of the asparagus;

[0137] The criteria for determining maturity are as follows:

[0138] Young bamboo shoots (base diameter < 8mm): those with a tender stem length ≥ 20cm are considered mature, and those with a tender stem length < 20cm are considered immature;

[0139] Medium-sized bamboo shoots (8mm ≤ base diameter < 13mm): tender stems ≥ 24cm are considered mature, tender stems < 24cm are considered immature;

[0140] Large bamboo shoots (base diameter ≥13mm): those with a tender stem length ≥28cm are considered mature, and those with a tender stem length <28cm are considered immature;

[0141] For all sizes of asparagus: tender stems ≥40cm in length are considered overripe and are not included in the harvesting scope.

[0142] Step 7: Based on the spatial pose parameters of mature asparagus in Step 5.6, calculate the three-dimensional coordinates of the harvesting cutting surface, and transmit the maturity determination results and the cutting surface positioning data to the asparagus harvesting robot. This provides a data foundation for the harvesting action control of the asparagus harvesting robot, as detailed below:

[0143] Step 7.1: Based on the coordinates of the lowest point at the bottom of the asparagus and the key points of the 2nd to 3rd skeletal segments, calculate the 3D coordinates of the cutting point:

[0144] ( =3 / 20)

[0145] in,( () represents the coordinates of the cutting point in the camera coordinate system.

[0146] Step 7.2: The cutting surface parameters are expressed as follows:

[0147]

[0148] The data is transmitted to the asparagus harvesting robot via RS485 or TCP / IP interface, guiding the six-degree-of-freedom robotic arm to adjust the attitude of the end effector so that the gripping fingers are parallel to the asparagus axis, and to complete flexible gripping and shearing harvesting at the cutting surface.

[0149] In this embodiment, preferably, the present invention also adds a branch for enhancing the texture features of asparagus tips to the improved deep learning network, thereby achieving accurate identification of tight and loose asparagus tips. This solves the problem that existing technologies cannot distinguish between different types of asparagus tips. The identification results can be directly used to guide the end effector of the asparagus harvesting robot to adaptively adjust the clamping force (the clamping force is slightly larger for tight-tipped asparagus to prevent slippage during harvesting; the clamping force is slightly smaller for loose-tipped asparagus to prevent crushing damage), further reducing the harvesting damage rate; specifically as follows:

[0150] Feature Filtering and Enhancement: Features corresponding to the shoot tip region are filtered from the P3 and P4 feature maps of the FPN feature pyramid. A self-designed texture feature enhancement operator is used to strengthen the texture details of the shoot tip region (tight-tipped shoot tips have dense textures and regular edges, while loose-tipped shoot tips have loose textures and irregular edges), and to suppress interfering features from the background and other areas of the asparagus stem.

[0151]

[0152] in, This represents the output value of the enhanced texture image at pixel (u,v). Represents the coordinates of each pixel in the image. Indicates the horizontal position (column) of a pixel. Indicates the vertical position (row) of a pixel; express Gaussian filtering; This represents the gradient operator, used to extract the texture of the bamboo shoot tip edge; Indicates a directional mask, used to enhance texture along the growth direction of the bamboo shoot tip;

[0153] Category Recognition and Output: The enhanced bamboo shoot tip features are input into the classification branch. The class probability is output through the Softmax function. When the probability is greater than 95%, it is determined to be the corresponding class (tight-head / spread-head). The recognition accuracy is over 97%.

[0154]

[0155] when When the value is greater than 0.95, it is considered a tight end;

[0156] when A value greater than 0.95 is considered a divergent trend.

[0157] in, This indicates that the bamboo shoot tips belong to the first... The probability value of the class; Indicates the category number, 1 represents a tight head, and 2 represents a loose head; This indicates the probability that the bamboo shoot tip is the tightest point. This represents the probability that the bamboo shoot tip is a scattered head; This represents the original predicted value output by the classification branch (obtained by passing the enhanced bamboo shoot tip features through a fully connected layer). This represents the original prediction score of the improved deep learning network for the "tight head"; This represents the original prediction score of the improved deep learning network for "scattered heads"; This indicates the operation of the natural exponent.

[0158] The visual recognition and detection system for asparagus harvesting robots adapted to different lighting environments as described in this invention includes:

[0159] The image acquisition module uses a depth camera, LED fill light, image stabilization module and data transmission unit to acquire asparagus image data in the facility asparagus cultivation environment, and uploads it to the computer for further processing by other modules.

[0160] The dataset construction module preprocesses the image data acquired by the image acquisition module to build a dataset that will later serve as input to the improved deep learning network. The preprocessing includes asparagus image enhancement and annotation file updates.

[0161] The network model building module constructs an improved deep learning network to identify asparagus in images and generate asparagus masks, while also recognizing asparagus tips that are either tight or loose. The improved deep learning network includes an improved CNN architecture with added CBAM attention mechanism and SPP structure, and a YOLACT++ instance segmentation network with improved anchor boxes.

[0162] The asparagus segmentation and parameter calculation module extracts the asparagus mask generated by the network model building module, and converts the pixel coordinates into three-dimensional coordinates by combining the camera calibration parameters, and calculates the length of the asparagus tender stem, the base diameter, and the spatial pose parameters.

[0163] The maturity assessment and positioning module accurately determines the maturity level based on the set maturity assessment criteria and the parameters of asparagus length and base diameter. It calculates the three-dimensional coordinates of the harvesting cutting surface based on the spatial pose parameters and transmits the maturity assessment results, harvesting cutting surface positioning data, and asparagus tight head and loose head classification results to the asparagus harvesting robot, providing a data foundation for the harvesting action control of the asparagus harvesting robot.

[0164] The model optimization module employs transfer learning strategies and model lightweighting techniques to optimize the model, adapting it to the edge deployment requirements of asparagus harvesting robots.

[0165] The deployment interaction module supports three deployment methods: cloud, edge, and embedded. The memory usage for edge and embedded deployments is ≤512MB, while cloud deployment supports concurrent recognition of ≥100 video streams. The deployment interaction module provides a standardized API interface to support integration with asparagus harvesting robot control systems and planting management platforms. It is also equipped with a visualization interaction unit and a distributed data storage unit, which has data encryption and regular backup functions.

[0166] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A deep learning detection method for asparagus harvesting robots adapted to different lighting environments, characterized in that, The process includes the following: Step 1: Acquire asparagus images using a depth camera; Step 2: Image annotation of asparagus; Step 3: Asparagus image enhancement processing; Step 4: Use an improved deep learning network to identify asparagus and generate an asparagus mask; the improved deep learning network includes an improved CNN architecture with added CBAM attention mechanism and SPP structure, and an improved anchor box YOLACT++ instance segmentation network; the improved CNN architecture is used to extract image features from the input image, and the improved anchor box YOLACT++ instance segmentation network is used to generate the asparagus mask; Step 5: Extract candidate asparagus masks, and calculate the length of asparagus shoots and base diameter by combining the 3D coordinate data from the depth camera; Step 6: Based on the pre-set maturity judgment criteria, and combined with the parameters of asparagus tender stem length and base diameter, determine the maturity of asparagus; Step 7: Calculate the three-dimensional coordinates of the harvesting cutting surface based on the spatial pose parameters of mature asparagus, and transmit the maturity judgment results and cutting surface positioning data to the asparagus harvesting robot to provide a data basis for the harvesting action control of the asparagus harvesting robot.

2. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments as described in claim 1, characterized in that, The specific process of step 5 is as follows: Step 5.1: Calibrate the depth camera to obtain the intrinsic parameter matrix and distortion matrix, and use the calibration parameters to convert the pixel coordinates of the asparagus mask into three-dimensional coordinates in the camera coordinate system; Step 5.2: Pixel Coordinate System The origin is the top left corner of the image. shaft and The axes represent the two right-angled sides parallel to the image plane; first, the asparagus mask in the pixel coordinate system is obtained. The coordinates of the highest and lowest points of the axis projection are then used to determine the target asparagus. Divided equally along the axial projection direction Divide the data into equal parts and calculate the coordinates of the intersections of each division. Obtain the initial point coordinates ( , )and( , The initial point refers to the pixel point on the left and right edges of the asparagus cross-section after the asparagus is evenly divided into segments along its length on the pixel mask outline. This indicates the segment number of the asparagus. =[1, -1], This indicates the sequence number of asparagus spears in the image; Step 5.3: Calculate the coordinates of each skeleton key point using the initial point coordinates. , )of The component, the skeleton key point refers to the feature point on the central skeleton line of the asparagus, which is obtained by averaging two initial points on the left and right edges of the same segment; Step 5.4: Establish a fitted straight line using the key points of the skeleton in the first three segments; Step 5.5: Determine the coordinates of the highest point of the asparagus. Simultaneously, the intersection point of the fitted straight line and the asparagus mask contour with the largest v component is taken as the coordinate of the lowest point of the asparagus. Then calculate the length of the tender asparagus stems; Step 5.6: Calculate the roll angle and pitch angle of the asparagus, determine the posture of the asparagus on the soil ridge, and calculate the diameter of the base of the asparagus in combination with the roll angle.

3. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments according to claim 2, characterized in that, In step 5.5, the length of the asparagus tender stems for: In the formula, Indicates the first The length of the tender asparagus stems of root asparagus. Indicates the first The first asparagus The three-dimensional coordinates of each skeleton point in the camera coordinate system Indicates the first The first asparagus The three-dimensional coordinates of a skeleton point in the camera coordinate system, the skeleton point includes One key skeletal point, one highest point, and one lowest point; =[0, ], hour, Representing the The coordinates of the lowest point of the bottom of the asparagus stalk in the camera coordinate system. hour, Representing the The three-dimensional coordinates of the highest point of the asparagus root in the camera coordinate system.

4. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments according to claim 2, characterized in that, In step 5.6, the lowest point of the asparagus bottom in the camera coordinate system is used. Establish a spatial rectangular coordinate system OXYZ with the origin as the origin. The roll angle is the angle between the projection of the fitted line onto the OXY plane and the X-axis, and the pitch angle is the angle between the projection of the fitted line onto the OYZ plane and the Z-axis; as detailed below: in, Indicates the first The corner of a stalk of asparagus, Indicates the first The pitch angle of a single asparagus shoot. Indicates the first The three-dimensional coordinates of the lowest point at the bottom of the asparagus stalk in the camera coordinate system. Indicates the first The three-dimensional coordinates of the second key skeletal point of the asparagus root in the camera coordinate system.

5. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments according to claim 4, characterized in that, The diameter of the asparagus base is calculated as follows: in, Indicates the diameter at the base of the asparagus; The original value of the diameter parallel to the X-axis of the depth camera is calculated based on the 3D coordinates of the initial points of the first and second segments of the asparagus bottom contour, as follows: in, Indicates the first position in the camera coordinate system The three-dimensional coordinates of the initial left point of the outline at the junction of the first and second segments at the bottom of the asparagus root. Indicates the first position in the camera coordinate system The three-dimensional coordinates of the initial point on the right side of the outline at the junction of the first and second segments at the bottom of the asparagus root.

6. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments according to claim 1, characterized in that, In step 6, the maturity determination criteria are as follows: Young bamboo shoots (base diameter < 8mm): those with a tender stem length ≥ 20cm are considered mature, and those with a tender stem length < 20cm are considered immature; Medium-sized bamboo shoots (8mm ≤ base diameter < 13mm): tender stems ≥ 24cm are considered mature, tender stems < 24cm are considered immature; Large bamboo shoots (base diameter ≥13mm): those with a tender stem length ≥28cm are considered mature, and those with a tender stem length <28cm are considered immature; For all sizes of asparagus: tender stems ≥40cm in length are considered overripe and are not included in the harvesting scope.

7. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments according to claim 5, characterized in that, The specific process of step 7 is as follows: Step 7.1: Based on the coordinates of the lowest point at the bottom of the asparagus and the key points of the 2nd to 3rd skeletal segments, calculate the 3D coordinates of the cutting point: （ =3 / 20） in,( () represents the coordinates of the cutting point in the camera coordinate system; Step 7.2: The cutting surface parameters are expressed as follows: The cutting parameters are transmitted to the asparagus harvesting robot via RS485 or TCP / IP interface, which guides the six-degree-of-freedom robotic arm to adjust the attitude of the end effector so that the gripping fingers are parallel to the asparagus axis, and complete the flexible gripping and shearing harvesting at the cutting surface.

8. The deep learning detection method for asparagus harvesting robots adapted to different lighting environments according to claim 1, characterized in that, In step 4, the improved deep learning network also includes a new branch for enhancing the texture features of asparagus tips, used to identify tight and loose asparagus tips. The identification results are transmitted to the asparagus harvesting robot to guide the robot's end effector to adaptively adjust the clamping force. The specific identification method is as follows: Feature selection and enhancement: From the P3 and P4 feature maps output by the improved CNN architecture FPN feature pyramid network, features corresponding to the shoot tip region are selected. A self-designed texture feature enhancement operator is used to strengthen the texture details of the shoot tip region and suppress interfering features from the background and other areas of the asparagus stem. in, This indicates the enhanced texture image in pixels. The output value at that location, Represents the coordinates of each pixel in the image. Indicates the horizontal position of a pixel. Indicates the vertical position of a pixel; express Gaussian filtering; This represents the gradient operator, used to extract the texture of the bamboo shoot tip edge; Indicates a directional mask, used to enhance texture along the growth direction of the bamboo shoot tip; Category recognition and output: The enhanced bamboo shoot tip features are input into the classification branch, and the class probability is output through the Softmax function. When the probability is greater than 95%, it is determined to be the corresponding class. when When the value is greater than 0.95, it is considered a tight end; when A value greater than 0.95 is considered a divergent trend. in, This indicates that the bamboo shoot tips belong to the first... The probability value of the class; Indicates the category number, 1 represents a tight head, and 2 represents a loose head; This indicates the probability that the bamboo shoot tip is the tightest point. This represents the probability that the bamboo shoot tip is a scattered head; This represents the original predicted value output by the classification branch; This represents the original prediction score of the improved deep learning network for the "tight head"; This represents the original prediction score of the improved deep learning network for "scattered heads"; This indicates the operation of the natural exponent.

9. A visual recognition closed-loop system for an asparagus harvesting robot adapted to different lighting environments, used to implement the deep learning detection method for asparagus harvesting robots adapted to different lighting environments as described in claim 1, characterized in that, include: The image acquisition module uses a depth camera, LED fill light, image stabilization module and data transmission unit to acquire asparagus image data in the facility asparagus cultivation environment, and uploads it to the computer for further processing by other modules; The dataset construction module preprocesses the image data acquired by the image acquisition module to build a dataset that will later serve as input to the improved deep learning network. The preprocessing includes asparagus image enhancement and annotation file updates. The network model building module builds an improved deep learning network to identify asparagus in images and generate asparagus masks, while also identifying whether the asparagus tips are tight or loose. The asparagus segmentation and parameter calculation module extracts the asparagus mask generated by the network model building module, combines the camera calibration parameters to convert the pixel coordinates into three-dimensional coordinates, and calculates the asparagus tender stem length, base diameter, and spatial pose parameters. The maturity determination and positioning module accurately determines the maturity based on the set maturity determination criteria and the parameters of asparagus length and base diameter. It calculates the three-dimensional coordinates of the harvesting cutting surface according to the spatial pose parameters and transmits the maturity determination results, the harvesting cutting surface positioning data, and the asparagus tight head and loose head category identification results to the asparagus harvesting robot. The model optimization module employs transfer learning strategies and model lightweighting techniques to optimize the model, adapting it to the edge deployment requirements of the asparagus harvesting robot. The system deploys an interactive module, provides standardized API interfaces, supports integration with asparagus harvesting robot control systems and planting management platforms, and is also equipped with a visual interactive unit and a distributed data storage unit.

Images