Tea leaf contour extraction and picking point intelligent detection method and equipment

Abstract

The invention relates to the field of agricultural automation, and discloses a tea contour extraction and picking point intelligent detection method and equipment, and the method comprises the steps: collecting an RGB original image and a depth image of a tea tree tender shoot, and an RGBD image synthesized by the depth image, and constructing a training set; based on the instance segmentation branches and the tea tender shoot picking point detection branches, constructing a lightweight tea tender shoot contour extraction and picking point detection model; training the lightweight tea tender shoot contour extraction and picking point detection model by using a training set and a back propagation algorithm to obtain a trained model; obtaining an image of a to-be-picked area, obtaining a 2D picking frame and a tea tender shoot contour extraction result by using the trained model, and completing tea tender shoot contour extraction and picking point detection; according to the invention, the problem that the outline and the picking point of the tender shoots of the tea leaves cannot be rapidly and accurately extracted when the environment of the tea garden changes is solved, the picking efficiency and the automation level are obviously improved, and the economic benefit is further improved.

Description

Technical Field

[0001] The present invention relates to the field of agricultural automation, and in particular to a method and equipment for extracting tea contours and intelligently detecting picking points. Background Art

[0002] As people's demand for high-quality tea continues to increase, the market share of famous and high-quality tea is also gradually expanding. However, the picking of famous and high-quality tea usually needs to rely on manual labor, which is not only inefficient but also costly. In order to improve the picking efficiency and quality of famous and high-quality tea and realize intelligent picking, it is of great practical significance to study a tea bud contour extraction and picking point detection method for intelligent picking of famous and high-quality tea.

[0003] In the past, there have been some methods for identifying and picking tea buds. Among them, the technology based on machine vision is a common solution. This technology uses a camera to obtain tea images, and then uses image processing algorithms to analyze the images to extract the outline of tea buds and detect picking points. Specifically, this method usually includes the following steps:

[0004] 1. Image acquisition: Use a high-definition camera to shoot the tea leaves in the tea garden to obtain clear images.

[0005] 2. Image preprocessing: Perform denoising, enhancement, grayscale and other preprocessing operations on the collected images to improve the image quality and the accuracy of subsequent processing.

[0006] 3. Extraction of young bud contours: Edge detection, morphological processing and other algorithms are used to extract the contours of young tea buds from the preprocessed images.

[0007] 4. Picking point detection: Determine the appropriate picking point based on the outline and growth characteristics of the buds. This usually requires considering factors such as the length, thickness, and curvature of the buds.

[0008] 5. Control execution: The detected picking point information is transmitted to the control system of the picking robot to control the robot to complete the picking action.

[0009] Although the tea bud contour extraction and picking point detection method based on machine vision has improved the picking efficiency of famous and high-quality tea to a certain extent, there are still some shortcomings:

[0010] 1. Environmental adaptability: The environment of the tea garden is complex and changeable. Factors such as lighting conditions and background color will affect image acquisition and processing. For example, under strong light, the shadow of the tea leaves may lead to misjudgment of the outline of the buds; under complex backgrounds, it is difficult to distinguish between the buds and the background, which is prone to missed detection or false detection.

[0011] 2. Complexity of bud features: The morphology and features of tea buds are diverse, and buds of different varieties and different growth stages differ in shape, color, texture, etc. This makes it difficult for existing traditional image processing algorithms to accurately identify and extract all types of bud contours, thus affecting the detection accuracy of picking points.

[0012] 3. Real-time requirements are difficult to meet: The picking of high-quality tea needs to be completed in a short time to ensure the quality of the tea. However, existing image processing algorithms usually have large computational complexity and slow processing speed, making it difficult to meet real-time requirements. This may lead to low efficiency of the picking robot and failure to complete the picking task in time.

[0013] 4. High equipment cost: In order to realize the intelligent picking of tea buds, high-performance cameras, image processing equipment and control systems are required, which makes the cost of the entire picking system high and limits its wide application in actual production.

[0014] In summary, although the existing tea bud identification and picking point positioning methods for intelligent picking of high-quality tea have achieved certain results, they still need to be further improved and perfected to improve their adaptability, accuracy and real-time performance in complex environments, reduce equipment costs, and promote the development and application of intelligent picking technology for high-quality tea. Summary of the invention

[0015] The purpose of the present invention is to propose a method and device for extracting tea contours and intelligently detecting picking points, so as to solve the technical problems of high cost, poor real-time performance and inaccurate positioning in the prior art for identifying tea buds and locating picking points.

[0016] The present invention provides a tea contour extraction and picking point intelligent detection method, comprising the following steps:

[0017] S1: Collect the original RGB images, depth images, and RGBD images synthesized from the depth images of tea tree buds, and build a training set;

[0018] S2: Based on the instance segmentation branch and the tea bud picking point detection branch, a lightweight tea bud contour extraction and picking point detection model is constructed;

[0019] S3: using the training set and the back propagation algorithm to train the lightweight tea bud contour extraction and picking point detection model to obtain a trained model;

[0020] S4: Obtain an image of the area to be picked, use the trained model to obtain the 2D picking frame and tea bud contour extraction results, and complete the tea bud contour extraction and picking point detection.

[0021] A storage medium stores instructions and data for implementing a tea contour extraction and picking point intelligent detection method.

[0022] A tea contour extraction and picking point intelligent detection device comprises: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a tea contour extraction and picking point intelligent detection method.

[0023] The beneficial effects provided by the present invention are: a lightweight network is constructed to extract the contours of tea buds and detect their picking points. The network solves the problem that the contours of tea buds and picking points cannot be extracted quickly and accurately when the tea garden environment changes, significantly improves the picking efficiency and automation level, helps to reduce mispicking and missed picking, and thus improves economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic flow chart of the method of the present invention;

[0025] Figure 2 The location of the tea bud picking point is shown;

[0026] Figure 3 An example of annotation of tea bud images is shown;

[0027] Figure 4 This is a schematic diagram of the effect of collecting young tea shoots;

[0028] Figure 5 It is a schematic diagram of RGBD image;

[0029] Figure 6 It is a schematic diagram of the specific structure of the lightweight tea bud contour extraction and picking point detection model;

[0030] Figure 7 It is a schematic diagram of the structure of the BiP-FPN module;

[0031] Figure 8 It is a schematic diagram of the FASP module structure;

[0032] Fig. 9 This is a schematic diagram of the RGBDF module structure;

[0033] Fig.10 It is a schematic diagram of the GFF module structure;

[0034] Fig.11 It is the specific structure of HiLo attention mechanism;

[0035] Fig.12 It is the overall structure of the famous tea picking robot;

[0036] Fig.13It is a schematic diagram of the overall structure of the control system of the intelligent tea picking robot and the contents of each part;

[0037] Fig.14 It is a control system connection diagram;

[0038] Fig.15 The specific results of tea bud contour extraction and picking point detection and positioning;

[0039] Fig.16 It is a working schematic diagram of the hardware device of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0041] Before formally describing the present invention, the scheme of the present invention is first generally described for easy understanding.

[0042] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the process of the present invention.

[0043] The present invention provides a tea contour extraction and picking point intelligent detection method, comprising the following steps:

[0044] S1: Collect the original RGB images, depth images, and RGBD images synthesized from the depth images of tea tree buds, and build a training set;

[0045] It should be noted that, in the present invention, the training data set includes RGB original images, depth images, and RGBD images synthesized from RGB and depth images of tea tree buds under different lighting conditions and at different distances in a large outdoor scene.

[0046] Observing the RGB original image data of tea buds at the canopy scale, given the wide field of view of a single image, the number of buds covered is large and scattered, which has caused certain obstacles to the subsequent labeling work. Therefore, a 2×4 ratio segmentation operation was performed on the original image, and images with clear features of buds and tea stems and clear picking points were selected, totaling 491 images.

[0047] After the above treatment, the tea buds in the tea buds image used in the present invention are reduced to single or multiple buds and stored in JPG format with a resolution of 768×768px. Subsequently, the data set is divided into a training set, a validation set, and a test set. Among them, the training set accounts for 70% of all data sets and is used for learning weight parameters in the model training process; the validation set accounts for 20% and is intended to optimize the network model structure; the test set accounts for 10% and is used to verify the accuracy of the proposed method.

[0048] In the present invention, by consulting with a number of tea experts and going to the field during the tea bud picking period to exchange picking experience with tea farmers, the location of the picking points when picking high-quality tea is finally clarified, and then the labeling scheme of the tea bud images is determined.

[0049] Please refer to Figure 2 The location of the tea bud picking point is shown. Figure 3 The example of annotating tea bud images is shown. The depth image is collected by an RGB-D camera based on the binocular principle. This series of cameras obtains the depth value of the target point based on the binocular stereo vision principle. Figure 4 shown.

[0050] In common object detection tasks, there are multiple types of objects in one image, which means there are great differences between the objects. However, in the task of this study, there are slight differences in the internodes of tea buds, and the color and texture features between tea buds are similar. But their lighting, deformation and posture are different. The most important thing is to distinguish between the upper and lower buds of the tea tree canopy, which is conducive to lossless picking. Four-channel RGBD images are a good choice.

[0051] Therefore, as an embodiment, the present invention proposes a new method for fusing RGB images and depth images to generate new images. The proposed fusion method aims to expand the feature differences between tea buds and maintain the good characteristics of fine-tuning training. The new three-channel image is calculated by formula (1).

[0052]

[0053] Where R, G, B, D refer to the red, green, blue and depth values ​​in each pixel respectively; α is a parameter. C1, C2, C3 are the new values ​​of each pixel in the three channels of the image. Then, each pixel value in the new image is normalized from 0 to 255 to obtain an RGBD image. By visual inspection, the parameter α is 0.70 by analyzing the pixels of the background and foreground to maximize the g value in equation (2) with the constraint of avoiding excessive color distortion.

[0054] g=w0×(μ0-μ) 2 +w1×(μ1-μ) 2 (2)

[0055] μ=w0×μ0+w1×μ1 (3)

[0056] Among them, g represents the difference between background and foreground. w0 represents the ratio of the number of background pixels to the number of pixels in the whole image. w1 represents the ratio of the number of foreground pixels to the number of pixels in the whole image. μ, μ0, and μ1 represent the average pixel values ​​of the whole image, background, and foreground pixels, respectively. Finally, these new three-channel images are named RGBD images, such as Figure 5 shown.

[0057] S2: Based on the instance segmentation branch and the tea bud picking point detection branch, a lightweight tea bud contour extraction and picking point detection model is constructed;

[0058] It should be noted that accurate identification of tea buds and precise positioning of picking points are important foundations for intelligent picking of premium tea. Deep learning-based identification and detection of tea buds can better adapt to the changes in the complex environment of the real field, but most current research is limited to the detection stage of tea buds and cannot directly achieve end-to-end segmentation of tea buds and positioning of picking points.

[0059] To meet this demand, the present invention innovatively integrates segmentation and picking detection into the same framework, and constructs a lightweight tea bud contour extraction and picking point detection model based on the instance segmentation branch and the tea bud picking point detection branch, in order to provide theoretical support for the intelligent picking of famous and high-quality tea in terms of algorithm.

[0060] Segmentation can remove noise information in the background, allowing the high-quality tea picking robot to robustly pick tea in different backgrounds. Using the segmentation results, we can refine the picking detection and find the best picking posture for the robot in complex scenes.

[0061] The existing general grasping framework uses a five-dimensional rectangle to represent the grasping posture. The present invention proposes a four-dimensional grasping representation by ignoring the width of the robot's grasping fingers, which simplifies the grasping representation. The grasping candidate G is defined as: G = (P, Q, A, W), where P is the center point of the grasping candidate. Q represents the probability of successfully grasping each grasping point, expressed as a score between 0 and 1. A and W are the grasping angle and width, respectively.

[0062] The present invention adopts a simplified four-dimensional grasping representation. At present, most of the picking detection methods are oriented to the entire image. These methods are easily disturbed by background information in unstructured environments, resulting in a low grasping success rate.

[0063] Specifically, an improved Mask-RCNN model is adopted based on the instance segmentation branch; ResNet and BiP-FPN are used as the backbone network to extract feature maps based on the improved Mask-RCNN model; for the extracted feature maps, the region proposal network RPN is used to predict the region of interest ROI; the feature map is cropped using the region of interest ROI, and finally the tea bud picking point detection branch is used to predict the object category.

[0064] It should be noted that the tea bud picking point detection branch obtains a 7×7×N feature map for each ROI predicted in the first step. In the second stage, the feature map is cropped and distorted into a uniform shape at the ROI location. After that, the leaf bud picking point detection branch predicts the object category.

[0065] For the instance segmentation branch, masks are generated using the k×28×28 output from RoIAlign.

[0066] The tea bud picking point detection branch projects the k×28×28 output onto a feature map of the same size as the captured feature according to the position of the captured feature on the original feature map.

[0067] The feature map from the instance segmentation branch distinguishes the background from the object and reduces the background noise. Therefore, it sets the picking position and obtains the category and shape of the object.

[0068] Please refer to Figure 6 , Figure 6 It is a schematic diagram of the specific structure of the lightweight tea bud contour extraction and picking point detection model;

[0069] The model of the present invention is mainly used for end-to-end tea bud contour extraction and picking point positioning, further improving the accuracy of tea bud positioning and the efficiency of tea bud picking. The main idea behind it is:

[0070] Use ROI to crop the feature map, remove interference information in the background, and optimize the picking point detection to reduce the impact of complex background.

[0071] Through the proposed FASP module, picking candidates can be generated for tea buds of different sizes.

[0072] The depth map is assigned a reliability score using the RGBDF module. The higher the score, the more important the feature at that location.

[0073] The output features of the FASP module and the RGBDF module are connected through the GFF module, and attention weights are assigned to the two different features, effectively fusing features of different scales. These features are then used for decoding for picking point positioning.

[0074] The overall processing process of the model is shown in the following formula.

[0075]

[0076] Through the backbone feature extraction network Then extract the RGB image features. The multi-scale fusion feature F is obtained through the FASP module FA , after instance segmentation branch Then extract the instance segmentation branch feature F ROI By connecting two features, we can obtain high-level semantic features

[0077] The processing of RGBDF is then as follows.

[0078] The RGBDF module has three branches. The first branch generates a confidence map from the depth image. It calculates the element-wise addition between the RGBD image and the depth image features, and then multiplies the output with the depth map to enhance the representation of the contour information.

[0079] In the second branch, the RGBDF module uses multiple convolutions to obtain features at different scales.

[0080] In the third branch, the RGBDF module calculates the absolute value of element-wise subtraction between the RGBD image and the depth image features. The output is then multiplied with the depth map to further enhance the representation of edge semantic information.

[0081] Finally, RGBDF connects multiple feature maps obtained from the three branches. Then, these features are fused through 3×3 convolution operations to obtain low-level RGB-D fusion features. Finally, the final picking representation can be obtained by processing the following equations.

[0082]

[0083] Advanced Features Upsampling to low-level features The size of F is then fed into the GFF module. In order to improve the quality of picking point prediction, GFF Decode and then enter the Sigmoid activation function to obtain quality, angle, and width.

[0084] As an example, please refer to Figure 7 , Figure 7 It is a structural diagram of the BiP-FPN module.

[0085] The multi-scale feature fusion module in the target detector is based on P out =(P in1 +P in2...) or P out =(P in1e P in2 ...) to aggregate and fuse multi-level feature information of different resolutions.

[0086] In order to achieve more efficient feature fusion, the present invention proposes a learnable weighted complex BiP-FPN network. For a node with only one input, if there is no feature fusion with other scale features, the node will contain very little multi-scale feature information. This will significantly weaken the multi-level feature fusion of the bidirectional network. Therefore, the present invention strengthens the bidirectional fusion mechanism in a more complex weighted manner based on BiFPN.

[0087] Specifically, maximum pooling and linear interpolation operations are added to one or more additional information streams at each node to fully aggregate hierarchical features from multiple scales. This full-scale cascade fusion strategy is conducive to suppressing the loss and deviation of defect information during multi-scale feature transmission, and is conducive to the fusion of fusion features. By assigning learnable weights to different inputs of each node, the BiP-FPN network can automatically optimize and determine the importance of each input. Therefore, the information redundancy and computational complexity of the model participating in the operation can be effectively reduced. The BiP-FPN network also helps to avoid information dislocation caused by a few useless feature streams dominating the nodes in the traditional FPN structure.

[0088] The detailed operations in BiP-FPN can be expressed as follows.

[0089]

[0090] Where P i represents the i-th node, which is formed by connecting P nodes adjacent to or of the same size as the node k Calculated by each node. Indicates P i Different input nodes P k The learnability coefficient of a node P i Consists of 3-5 nodes.

[0091] The BiP-FPN network is more like a full-size cross-fusion module, which can more effectively utilize features at different levels and significantly increase the available features of each node. In addition, BiP-FPN can alleviate the dominant influence of previous feature flows on nodes by using a learnable weight strategy. The weights of multiple information flows in a node can also be dynamically adjusted according to their contribution to the result, which is beneficial to improving the multi-scale feature extraction capability of the model.

[0092] As an example, please refer to Figure 8 , Figure 8 This is a schematic diagram of the FASP module structure.

[0093] It has several advantages as described below. Due to the lack of geometric information, high-level networks cannot detect small-scale objects. However, picking point detection requires the detection of small-scale objects, so the feature output of the third layer of BiP-FPN is used. The feature map has only 256 layers, which contains both high-level semantic information and high-resolution low-level features. This module generates multi-scale information feature maps, thereby improving the utilization of information at different levels.

[0094] The specific implementation of the FASP module is as follows.

[0095]

[0096] For a given input F FP3 , dilated convolutions of different sampling rates are sampled in parallel, and the results are concatenated with the BiP-FPN output at the channel level.

[0097] As an example, please refer to Fig. 9 , Fig. 9 This is a schematic diagram of the RGBDF module structure.

[0098] In the field of object detection and semantic segmentation, more and more RGB-D fusion methods have been proposed. However, RGB-D fusion methods have not been well applied in robotic tea picking. Fusion should be performed in a way that does not significantly increase the computational cost. RGB-D fusion picking detection can be divided into three categories according to the fusion strategy: early fusion, late fusion, and dense fusion. In early fusion, RGB and depth data are connected as input, but the two types of information are not distinguished, and the two types of information are not fully combined. In late fusion, RGB and depth are encoded separately. They are merged at the last feature level, so it is difficult to fully utilize the intrinsic relationship between the two data types. In addition, dense fusion extracts RGB and depth features and fuses them at multiple levels, but this will generate redundant information. Therefore, the RGBDF module is proposed. Its structure is as follows Figure 8 shown.

[0099] The three branches obtain confident fusion features of different scales. Finally, multi-scale RGB-D feature aggregation is achieved by connecting the features extracted from the three branches and adding very few parameters.

[0100] The first branch generates a confidence map from the depth image, calculates the element-wise addition between the RGBD image and the depth image features, and then multiplies the output with the depth map to enhance the representation of contour information.

[0101] In the second branch, the RGBDF module uses multiple convolutions to obtain features at different scales.

[0102] In the third branch, the RGBDF module calculates the absolute value of element-wise subtraction between the RGBD image and the depth image features. The output is then multiplied with the depth map to further enhance the representation of edge semantic information.

[0103] Finally, RGBDF connects multiple feature maps obtained from the three branches. Then, these features are fused through 3×3 convolution operations to obtain low-level RGB-D fusion features.

[0104] As an example, please refer to Fig.10 , Fig.10 This is a schematic diagram of the GFF module structure.

[0105] The FASP module outputs high-level semantic features with less geometric information. The RGBDF module outputs more geometric information but few semantic features. Using either information alone is ineffective for picking detection. Existing robotic picking point detection methods combine the two features through simple operations such as addition or concatenation, but fail to fully utilize the properties of the two features. Based on the above problems, the GFF module is proposed. Fig.10 Represents its structure. First, the two input features X and Y are fused, and then the Sigmoid activation function is applied, and the output value is between 0 and 1. Then the weighted average of X and Y is calculated. By assigning attention weights to two different features, semantically and scale-inconsistent features can be better fused, thereby improving the accuracy of picking point prediction. The specific implementation of the GFF module is as follows.

[0106]

[0107] After HiLo(f HiLo ) is labeled as H. and The features to be weighted are labeled F GFF .

[0108] It should be noted that the specific structure of the HiLo Attention mechanism is as follows Fig.11 shown.

[0109] HiLo adopts two effective attentions to decouple high / low frequencies in feature maps. Fig.11As shown, in the upper path, several Heads are assigned to high-frequency attention (Hi-Fi) to capture fine-grained high frequencies through Local Window Self-Attention (e.g., 2×2 windows), which is more effective than standard multi-head self-attention. In the lower path, low-frequency attention (Lo-Fi) is implemented, and average pooling is first applied to each window to obtain low-frequency signals. Then, the remaining Heads are assigned to Lo-Fi to model the relationship between each Query position in the input feature map and the average pooled low-frequency Key of each window. Benefiting from the reduction in Key and Value length, the Lo-Fi complexity is significantly reduced. Finally, the refined high / low-frequency features are concatenated and the resulting output is forwarded to subsequent layers.

[0110] SimAM Attention is a lightweight, parameter-free convolutional neural network attention mechanism that generates attention weights by calculating the local self-similarity of feature maps. SimAM does not require any additional parameters and can effectively enhance features.

[0111] S3: using the training set and the back propagation algorithm to train the lightweight tea bud contour extraction and picking point detection model to obtain a trained model;

[0112] As an embodiment, the model proposed in the present invention uses multi-task loss for end-to-end training.

[0113] The multi-task loss function contains different subtasks, and minimizing the value of the loss function is the goal of model learning. But different subtasks correspond to different scales. One way to solve this problem is to replace the direct addition of the loss functions of the subtasks with the weighted addition of the loss functions of the subtasks. Through weighting, each loss function can be scaled consistently. For example, for tasks where the loss decreases rapidly, a relatively small learning rate should be used, but for tasks where the loss decreases slowly, a relatively large learning rate should be used. The loss function L for picking point detection and instance segmentation is as follows:

[0114]

[0115] L grasp represents the picking point detection loss; L seg represents the instance segmentation loss. L grasp It is a combination of regression loss for predicting picking quality and width and classification loss for predicting direction. seg It consists of four parts: RPN , L cls , L box is generated by RPN for object detection, L maskis the mask loss. In order to balance the loss, two specific hyperparameters (λ grasp ,λ seg ). Since the loss of the picking point detection loss decreases faster than the segmentation loss, we set λ grasp Set to 1.5, and seg Set to 1.

[0116] The method is developed based on PyTorch, and all training and testing are performed using a single Nvidia RTX 3060 GPU. The batch size is 2, the learning rate is 0.001, and the weight decay is 0.001. 300 epochs are trained. The Adam optimizer is used to optimize the backpropagation network during training. The present invention uses the pre-trained ImageNet weights to initialize the backbone network and freeze the parameters of the first two network modules (Conv 1, Layer 1).

[0117] S4: Obtain an image of the area to be picked, use the trained model to obtain the 2D picking frame and tea bud contour extraction results, and complete the tea bud contour extraction and picking point detection.

[0118] As an embodiment, the present invention uses a trained lightweight tea bud contour extraction and picking point detection model to perform partition shooting and detection on the picking area, obtains a 2D picking frame and tea bud contour extraction results, and completes the tea bud contour extraction and picking point detection.

[0119] Based on the growth characteristics of field tea and picking needs, a field high-quality tea picking robot was built for tea bud contour extraction and picking point detection. Calibration experiments and picking tests were carried out, and the picking performance of each link and the overall picking performance were evaluated.

[0120] The overall structure of the famous tea picking robot is as follows Fig.12 As shown. Based on full consideration of the tea garden terrain and tea picking technical requirements, the present invention proposes an overall design scheme for an intelligent tea picking robot with a crawler self-propelled mechanism and a six-axis robotic arm. The tea picking robot includes a crawler self-propelled mechanism, a picking robotic arm, a tea picking end effector, a negative pressure collection device and a visual system; wherein the crawler self-propelled mechanism includes a driving wheel, a DC brushless driving motor, a bearing chamber, a frame, a supporting wheel, a guide wheel, a tensioning device, etc.; the tea picking end effector includes a stepping motor, a four-bar linkage, a flexible fingertip, a negative pressure hose connector, etc.; the negative pressure collection device includes a detachable negative pressure collection cover and a pressure-covered plastic packaging barrel; the visual system includes a depth camera and an angle-adjustable depth camera adapter plate.

[0121] By studying the control system of the tea picking robot, a reference can be provided for the overall machine structure design and hardware selection. In order to meet the actual application needs of the robot for precise picking of single buds of famous and high-quality tea, the present invention uses the FPGA+ARM solution to control the intelligent tea picking robot, so that it has the functions of movement, bud picking, perception interaction, etc.

[0122] After disassembling the functions of the entire system, the intelligent tea picking robot system is divided into two parts: tea picking control and chassis motion control. The overall structure of the control system of the intelligent tea picking robot and the contents of each part are as follows: Fig.13 shown.

[0123] The camera and the fixture are fixed to the end of the robot arm using a custom connector. The flexible fixture is driven by a motor. The I / O control signal is connected to the output port of the robot arm controller. The controller is connected to the host computer via a network cable. The host computer sends instructions to control the camera to capture images and control the movements of the robot and the flexible fixture. For the eye-hand system, the translation matrix T and rotation matrix R between the camera coordinate system and the robot arm coordinate system are obtained through hand-eye calibration. Through the hand-eye system, the picking point position and grasping line obtained by the recognition algorithm can be converted to the robot base coordinate system, thereby controlling the robot arm to complete the picking task. The control system is connected as follows: Fig.14 shown.

[0124] Finally, please refer to Fig.15 , Fig.15 These are the specific results of tea bud contour extraction and picking point detection and positioning.

[0125] See also Fig.16 , Fig.16 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a tea contour extraction and picking point intelligent detection device 401, a processor 402 and a storage medium 403.

[0126] A tea contour extraction and picking point intelligent detection device 401: The tea contour extraction and picking point intelligent detection device 401 implements the tea contour extraction and picking point intelligent detection method.

[0127] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the tea contour extraction and picking point intelligent detection method.

[0128] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the tea contour extraction and picking point intelligent detection method.

[0129] The beneficial effects of the present invention are: a lightweight network is constructed to extract the contours of tea buds and detect their picking points. The network solves the problem of being unable to quickly and accurately extract the contours and picking points of tea buds when the tea garden environment changes, significantly improves the picking efficiency and automation level, helps to reduce mispicking and missed picking, and thus improves economic benefits.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting tea contours and intelligently detecting picking points, characterized in that: The method comprises the following steps: S1: Collect the original RGB images, depth images, and RGBD images synthesized from the depth images of tea tree buds, and build a training set; S2: Based on the instance segmentation branch and the tea bud picking point detection branch, a lightweight tea bud contour extraction and picking point detection model is constructed; S3: using the training set and the back propagation algorithm to train the lightweight tea bud contour extraction and picking point detection model to obtain a trained model; S4: Obtain an image of the area to be picked, use the trained model to obtain the 2D picking frame and tea bud contour extraction results, and complete the tea bud contour extraction and picking point detection.

2. The method for extracting tea contours and intelligently detecting picking points according to claim 1, characterized in that: In step S1, the calculation formula of the RGBD image synthesized by the depth image is as follows: Among them, R, G, B, D refer to the red, green, blue and depth values ​​of each pixel respectively; α is a preset parameter.

3. The method for extracting tea contours and intelligently detecting picking points according to claim 1, characterized in that: The lightweight tea bud contour extraction and picking point detection model in step S2 includes: an instance-based segmentation branch and a tea bud picking point detection branch; Among them, the instance segmentation branch adopts the improved Mask-RCNN model; the tea bud picking point detection branch includes RGBDF module, FASP module and GFF module.

4. A tea contour extraction and picking point intelligent detection method as claimed in claim 3, characterized in that: Based on the improved Mask-RCNN model, ResNet and BiP-FPN are used as the backbone network to extract feature maps; for the extracted feature maps, the region proposal network RPN is used to predict the region of interest ROI; the feature map is cropped using the region of interest ROI, and finally the tea bud picking point detection branch is used to predict the object category.

5. A tea contour extraction and picking point intelligent detection method as claimed in claim 4, characterized in that: The input of the FASP module is the feature map output by BiP-FPN, and the output is the grasping candidates of objects of different sizes; The input of the RGBDF module is the depth image and the RGBD image synthesized from the depth image, and the output is the reliability score assigned to the depth map; The input of the GFF module is the reliability scores assigned to the grasping candidates and depth maps of objects of different sizes, and the output is the fused high-level semantic features.

6. A tea contour extraction and picking point intelligent detection method as claimed in claim 3, characterized in that: The RGBDF module consists of three branches: The first branch generates a confidence map from the depth image. It calculates the element-wise addition between the RGBD image and the depth image features, and then multiplies the output with the depth image to enhance the representation of the contour information. In the second branch, the RGBDF module uses multiple convolutions to obtain features of different scales; In the third branch, the RGBDF module calculates the absolute value of the element-wise subtraction between the RGBD image and the depth image features. The output is then multiplied with the depth map; Finally, the RGBDF module connects multiple feature maps obtained from the three branches and fuses these features through 3×3 convolution operations to obtain low-level RGB-D fusion features.

7. A tea contour extraction and picking point intelligent detection method as claimed in claim 4, characterized in that: In the BiP-FPN module, the multi-scale feature fusion module in the target detector is based on P out =(P in1 +P in2 ...) or P out =(P in1 ⊙P in2 ...) to aggregate and fuse multi-level feature information of different resolutions.

8. The method for extracting tea contours and intelligently detecting picking points according to claim 3, characterized in that: In the GFF module, the two input features X and Y are first fused, and then the Sigmoid activation function is applied, and the output value is between 0 and 1; then the weighted average of X and Y is calculated; by assigning attention weights to two different features.

9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a tea contour extraction and picking point intelligent detection method as described in any one of claims 1 to 8.

10. A tea contour extraction and picking point intelligent detection device, characterized in that: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a tea contour extraction and picking point intelligent detection method as described in any one of claims 1 to 8.

Images