A tea leaf profile extraction and picking point intelligent detection method and device

Abstract

The present application relates to the field of agricultural automation, and discloses a tea leaf contour extraction and picking point intelligent detection method and device, the method comprising: collecting tea tree bud RGB original image, depth image and depth image synthesized RGBD image, and constructing a training set; based on instance segmentation branch and tea leaf bud picking point detection branch, constructing a lightweight tea leaf bud contour extraction and picking point detection model; training the lightweight tea leaf bud contour extraction and picking point detection model using the training set and back propagation algorithm, obtaining the trained model; obtaining the image of the region to be picked, obtaining the 2D picking frame and tea leaf bud contour extraction result using the trained model, and completing the detection of tea leaf bud contour extraction and picking point; the present application solves the problem that the tea leaf bud contour and picking point cannot be quickly and accurately extracted when the tea garden environment changes, significantly improves the picking efficiency and automation level, and further improves the economic benefit.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural automation, and particularly relates to a tea leaf contour extraction and picking point intelligent detection method and device. BACKGROUND

[0002] With the increasing demand for high-quality tea, the market share of famous and high-quality tea is gradually expanding. However, the picking of famous and high-quality tea usually relies on manual labor, which is not only inefficient but also costly. In order to improve the picking efficiency and quality of famous and high-quality tea and realize intelligent picking, it is of great practical significance to study a tea bud contour extraction and picking point detection method for intelligent picking of famous and high-quality tea.

[0003] In the past research, there have been some methods about tea bud recognition and picking. Among them, the technology based on machine vision is a common solution. This technology acquires tea leaf images through a camera, and then uses image processing algorithms to analyze the images to extract the contour of tea buds and detect the picking points. Specifically, this method usually includes the following steps:

[0004] 1. Image acquisition: use a high-definition camera to shoot the tea leaves in the tea garden to obtain clear images.

[0005] 2. Image preprocessing: perform denoising, enhancement, grayscale, and other preprocessing operations on the collected images to improve the quality of the images and the accuracy of subsequent processing.

[0006] 3. Bud contour extraction: use edge detection, morphological processing, and other algorithms to extract the contour of tea buds from the preprocessed images.

[0007] 4. Picking point detection: determine the appropriate picking point according to the contour and growth characteristics of the buds. This usually needs to consider the length, thickness, and bending degree of the buds.

[0008] 5. Control execution: pass the detected picking point information to the control system of the picking robot to control the robot to complete the picking action.

[0009] Although the tea bud contour extraction and picking point detection method based on machine vision has improved the picking efficiency of famous and high-quality tea to some extent, there are still some shortcomings:

[0010] 1. Environmental adaptability problem: the environment of the tea garden is complex and changeable, and factors such as lighting conditions and background color will affect image acquisition and processing. For example, in strong light, the shadow of tea leaves may cause misjudgment of the bud contour; in a complex background, it is difficult to distinguish the bud from the background, and it is easy to miss or misjudge.

[0011] 2. The complexity of the bud features: The shape and features of tea buds have certain diversity. The buds of different varieties and different growth stages differ in shape, color, texture, etc. This makes it difficult for existing traditional image processing algorithms to accurately identify and extract the contours of all types of buds, affecting the detection accuracy of the picking points.

[0012] 3. Real-time requirements are difficult to meet: The picking of famous tea needs to be completed in a short time to ensure the quality of tea. However, existing image processing algorithms usually have large computational load and slow processing speed, making it difficult to meet the real-time requirements. This may result in low efficiency of the picking robot and failure to complete the picking task in time.

[0013] 4. High equipment cost: In order to realize intelligent picking of tea buds, high-performance cameras, image processing equipment and control systems are required, which makes the cost of the entire picking system high, limiting its widespread application in actual production.

[0014] In summary, although the existing tea bud recognition and picking point positioning method for famous tea intelligent picking has achieved certain results, further improvement and perfection are still needed to improve its adaptability, accuracy and real-time performance in complex environments, reduce equipment cost and promote the development and application of famous tea intelligent picking technology. SUMMARY

[0015] The purpose of the present application is to provide a tea contour extraction and intelligent picking point detection method and device, which solves the technical problems of high cost, poor real-time performance and inaccurate positioning in the existing technology of tea bud recognition and picking point positioning.

[0016] The tea contour extraction and intelligent picking point detection method provided by the present application comprises the following steps:

[0017] S1: Collecting tea bud RGB original image, depth image and RGBD image synthesized by depth image, and constructing training set;

[0018] S2: Based on the instance segmentation branch and the tea bud picking point detection branch, a lightweight tea bud contour extraction and picking point detection model is constructed;

[0019] S3: Training the lightweight tea bud contour extraction and picking point detection model using the training set and back propagation algorithm to obtain the trained model;

[0020] S4: Obtaining the image of the area to be picked, using the trained model to obtain the 2D picking frame and the tea bud contour extraction result, and completing the detection of the tea bud contour extraction and picking point.

[0021] A storage medium stores instructions and data for implementing a tea leaf contour extraction and picking point intelligent detection method.

[0022] A tea leaf contour extraction and picking point intelligent detection device comprises a processor and the storage medium; the processor loads and executes the instructions and data in the storage medium to implement a tea leaf contour extraction and picking point intelligent detection method.

[0023] The present application provides the beneficial effect: a lightweight network is constructed, aiming at extracting the contour of tea sprouts and detecting the picking point, which solves the problem that the contour of tea sprouts and the picking point cannot be quickly and accurately extracted when the environment of the tea garden changes, significantly improves the picking efficiency and the automation level, helps to reduce the mispicking and missed picking phenomenon, and further improves the economic benefit. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a method flowchart of the present application;

[0025] Figure 2 shows the picking point position of tea sprouts;

[0026] Figure 3 shows an annotation example of a tea sprout image;

[0027] Figure 4 is a tea sprout picking effect schematic diagram;

[0028] Figure 5 is an RGBD image schematic diagram;

[0029] Figure 6 is a lightweight tea sprout contour extraction and picking point detection model specific structure schematic diagram;

[0030] Figure 7 is a structure schematic diagram of the BiP-FPN module;

[0031] Figure 8 is a FASP module structure schematic diagram;

[0032] Figure 9 is an RGBDF module structure schematic diagram;

[0033] Figure 10 is a GFF module structure schematic diagram;

[0034] Figure 11 is the specific structure of the HiLo attention mechanism;

[0035] Figure 12 is the overall structure of the high-quality tea picking robot;

[0036] Figure 13It is a control system overall structure of intelligent tea picking robot and a content schematic view of each part;

[0037] Figure 14 It is a control system connection schematic view;

[0038] Figure 15 It is a specific result of tea bud contour extraction and picking point detection and positioning;

[0039] Figure 16 It is a hardware device working schematic view of the application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described below with reference to the drawings.

[0041] Before formally describing the present application, the scheme of the present application is first described generally for easy understanding.

[0042] Please refer to Figure 1 , Figure 1 It is a flowchart of the method of the present application.

[0043] The present application provides a tea contour extraction and picking point intelligent detection method, comprising the following steps:

[0044] S1: Collecting tea bud RGB original image, depth image and RGBD image synthesized by depth image, and constructing a training set;

[0045] It should be noted that in the present application, the training data set includes tea bud RGB original image, depth image and RGBD image synthesized by RGB and depth image under different illumination conditions and different distances in a large outdoor scene.

[0046] Observing the RGB original image data of tea buds at the crown layer scale, in view of the wide field of view of a single image, the number of buds covered is large and distributed scattered, which causes certain obstacles to subsequent labeling work. Therefore, the original image is subjected to a 2x4 ratio segmentation operation, and images with clear bud and tea stem features and clear picking points are selected from the images, totaling 491 images.

[0047] After the above treatment, the bud targets in the tea bud images adopted by the present application are reduced to single or multiple, and are stored in JPG format with a resolution of 768x768px. Subsequently, the data set is divided into a training set, a validation set and a test set. The training set accounts for 70% of all data sets and is used for learning of weight parameters in the model training process; the validation set accounts for 20% and is used for optimization of the network model structure; the test set accounts for 10% and is used for precision verification of the proposed method.

[0048] In the present application, through consulting to many tea experts and communicating with tea farmers in the field during the tea bud picking period, the picking point position of famous and excellent tea is finally determined, and the labeling scheme of tea bud image is determined.

[0049] Please refer to Figure 2 The tea bud picking point position is shown. Figure 3 The labeling example of tea bud image is shown. The collection of depth image is based on the RGB-D camera based on binocular principle. The series of cameras are based on the binocular stereo vision principle to obtain the depth value of target point. The collection effect of tea bud is shown in Figure 4 .

[0050] In the common target detection task, there are many types of targets in an image, which means that there is a big difference between targets. However, in the task of this research, there is a small difference between the internodes of tea buds, and the color and texture features of tea buds are similar. However, their illumination, deformation and posture are different. The most important thing is to distinguish the upper and lower buds on the tea tree canopy, which is beneficial to non-destructive picking. Four-channel RGBD image is a good choice.

[0051] Therefore, as an embodiment, the present application proposes a new method of fusing RGB image and depth image to generate a new image. The proposed fusion method aims to expand the feature difference between tea buds and maintain the good characteristics of fine-tuning training. The new three-channel image is calculated by formula (1).

[0052]

[0053] Wherein, R, G, B, D respectively indicate the red, green, blue and depth value in each pixel; alpha is a parameter. C1, C2, C3 are the new values of each pixel of the three channels of the image. Then, each pixel value in the new image is normalized from 0 to 255 to obtain the RGBD image. Through visual inspection, by analyzing the pixels of the background and foreground to maximize the g value in formula (2), and with the constraint of avoiding excessive color distortion, the parameter alpha is 0.70.

[0054] g=w0×(μ0-μ) 2 +w1×(μ1-μ) 2 (2)

[0055] μ=w0×μ0+w1×μ1 (3)

[0056] where g denotes the difference between background and foreground, w0 denotes the ratio of the number of background pixels to the number of pixels in the whole image, w1 denotes the ratio of the number of foreground pixels to the number of pixels in the whole image, and mu, mu0, mu1 denote the average pixel value of the whole image, background and foreground respectively. Finally, these new three-channel images are named as RGBD images, as shown in Figure 5 .

[0057] S2: based on the instance segmentation branch and the tea bud picking point detection branch, a lightweight tea bud contour extraction and picking point detection model is constructed;

[0058] It should be noted that accurate recognition of tea buds and accurate positioning of picking points are important foundations for realizing intelligent picking of famous and high-quality tea. The recognition and detection of tea buds based on deep learning can better adapt to changes in real field complex environments, but current researches are mostly limited to the detection stage of tea buds, and cannot directly realize end-to-end tea bud segmentation and picking point positioning.

[0059] In order to realize this demand, the present application innovatively integrates segmentation and picking detection into the same framework, based on the instance segmentation branch and the tea bud picking point detection branch, a lightweight tea bud contour extraction and picking point detection model is constructed, in order to provide theoretical support for intelligent picking of famous and high-quality tea in algorithm.

[0060] Segmentation can remove noise information in the background, so that the famous and high-quality tea picking robot can robustly pick in different backgrounds. Using the segmentation result, the picking detection can be refined and the best picking pose of the robot picking in a complex scene can be found.

[0061] The existing general grasping framework uses a five-dimensional rectangle to represent the grasping pose. The present application proposes a four-dimensional grasping representation by ignoring the width of the robot's grasping fingers, which simplifies the grasping representation. The grasping candidate G is defined as: G=(P,Q,A,W), where P is the center point of the grasping candidate. Q represents the probability of successfully grasping each grasping point, represented as a score between 0 and 1. A and W are the grasping angle and width respectively.

[0062] The present application adopts a simplified four-dimensional grasping representation. Most of the current picking detection methods are image-oriented. These methods are easily disturbed by background information in unstructured environments, resulting in low success rate of grasping.

[0063] Specifically, the improved Mask-RCNN model is adopted based on the instance segmentation branch; the ResNet and BiP-FPN are adopted as backbone network to extract feature maps based on the improved Mask-RCNN model; the region proposal network RPN is adopted to predict the region of interest ROI for the extracted feature maps; the feature maps are cropped by the region of interest ROI, and finally the tea bud picking point detection branch is used to predict the object category.

[0064] It should be noted that the tea bud picking point detection branch obtains a 7x7xN feature map for each ROI predicted in the first step. In the second stage, the feature map is cropped and distorted into a uniform shape at the ROI position. Then, the tea bud picking point detection branch predicts the object category.

[0065] For the instance segmentation branch, the mask is generated using the kx28x28 output generated by RoIAlign.

[0066] The tea bud picking point detection branch projects the kx28x28 output onto a feature map of the same size as the grasp feature according to the position of the grasp feature on the original feature map.

[0067] The feature map from the instance segmentation branch distinguishes between background and target, reducing background interference. Therefore, it sets the picking position and obtains the category and shape of the target.

[0068] Please refer to Figure 6 , Figure 6 is a specific structure diagram of a lightweight tea bud outline extraction and picking point detection model;

[0069] The model of the application is mainly used for end-to-end tea bud outline extraction and picking point positioning, further improving the accuracy of tea bud positioning and the work efficiency of tea bud picking. The main idea behind it is:

[0070] The ROI crops the feature map to remove interference information in the background and optimizes the picking point detection to reduce the influence of complex background.

[0071] Through the proposed FASP module, picking candidate objects can be generated for tea buds of different sizes.

[0072] The RGBDF module is used to assign a reliability score to the depth map. The higher the score, the more important the function of this position.

[0073] The output features of the FASP module and the RGBDF module are connected through the GFF module, and attention weights are assigned to the two different features, effectively fusing features of different scales. Then these features are decoded for picking point positioning.

[0074] The overall processing process of the model is as follows.

[0075]

[0076] Through the backbone feature extraction network RGB image features are extracted. Multi-scale fusion features F FA are obtained through the FASP module Instance segmentation branch features F ROI are extracted after instance segmentation branch

[0077] After that, the RGBDF processing is as follows.

[0078] The RGBDF module has three branches. The first branch generates a confidence map from the depth image. The element-wise addition value between the RGBD image and the depth image features is calculated, and the output result is multiplied by the depth map to enhance the representation of contour information.

[0079] In the second branch, the RGBDF module uses multiple convolutions to obtain features of different scales.

[0080] In the third branch, the RGBDF module calculates the absolute value of the element-wise subtraction between the RGBD image and the depth image features. Then, the output result is multiplied by the depth map to further enhance the representation of edge semantic information.

[0081] Finally, the RGBDF connects multiple feature maps obtained from the three branches. Then, through 3x3 convolution operation, these features are fused to obtain low-level RGB-D fusion features F Finally, through the following equation processing, the final picking representation can be obtained.

[0082]

[0083] High-level features F are upsampled to the size of low-level features F , and then they are fed into the GFF module. In order to improve the quality of picking point prediction, F GFF is decoded, and then enters the Sigmoid activation function to obtain quality (Quality), angle (Angle) and width (Width).

[0084] As an embodiment, please refer to Figure 7 , Figure 7 is a structural diagram of the BiP-FPN module.

[0085] The multi-scale feature fusion module in the target detector is P out = (P in1 + P in2...) or P out = (P in1e P in2 ...) are aggregated and fused in the form of multi-level feature information of different resolutions.

[0086] In order to realize more efficient feature fusion, the present application proposes a learnable weighted complex BiP-FPN network. For a node with only one input, if there is no feature fusion with other scale features, the node will contain little multi-scale feature information. This will significantly weaken the multi-level feature fusion of the bidirectional network. Therefore, the present application strengthens the bidirectional fusion mechanism in a more complex weighted manner on the basis of BiFPN.

[0087] Specifically, maximum pooling and linear interpolation operations are added on one or more additional information streams at each node to fully aggregate hierarchical features from multiple scales. This full-scale cascading fusion strategy is conducive to suppressing the loss and deviation of defective information in the multi-scale feature transmission process and conducive to the fusion of fused features. By assigning learnable weights to different inputs of each node, the BiP-FPN network can automatically optimize and determine the importance of each input. Therefore, the information redundancy and computational amount involved in the operation of the model can be effectively reduced. BiP-FPN network also helps to avoid the misplacement of information caused by the dominance of a few useless feature streams in the node in the traditional FPN structure.

[0088] The detailed operation in BiP-FPN can be represented as follows.

[0089]

[0090] where P i represents the i-th node, which is calculated by connecting P k adjacent or the same size nodes of the node. represents the learnable coefficient of the different input nodes P k of P i . Usually, a node P i is composed of 3-5 nodes.

[0091] BiP-FPN network is more like a full-size cross-fusion module, which can more effectively utilize features of different levels and significantly increase the available features of each node. In addition, BiP-FPN can reduce the dominant effect of previous feature streams on the node by using a learnable weight strategy. The weights of multiple information streams in the node can also be dynamically adjusted according to their contribution to the result, which is conducive to improving the multi-scale feature extraction capability of the model.

[0092] As an embodiment, please refer to Figure 8 , Figure 8 is a schematic diagram of the FASP module structure.

[0093] It has several advantages as described below. Due to the lack of geometric information, high-level networks cannot detect small-scale targets. However, picking point detection needs to detect small-scale targets, so the feature output of the third layer of BiP-FPN is used. The feature map has only 256 layers, which contains both high-level semantic information and high-resolution low-level features. This module generates multi-scale information feature maps, thereby improving the utilization of information at different levels.

[0094] The specific implementation of the FASP module is as follows.

[0095]

[0096] For a given input F FP3 Dilated convolutions with different sampling rates are sampled in parallel, and the results are connected with the BiP-FPN output at the channel level.

[0097] As an embodiment, please refer to Figure 9 , Figure 9 is a schematic diagram of the structure of the RGBDF module.

[0098] In the field of target detection and semantic segmentation, more and more RGB-D fusion methods have been proposed. However, RGB-D fusion methods have not been well applied in robot tea picking. Fusion should be done in a way that does not significantly increase the computational cost. RGB-D fusion picking detection can be divided into three categories according to the fusion strategy: early fusion, late fusion and dense fusion. In early fusion, RGB and depth data are connected as input, but the two types of information are not distinguished, and the two types of information are not fully combined. In late fusion, RGB and depth are encoded separately. They are merged at the last feature level, so it is difficult to fully utilize the intrinsic relationship between the two data types. In addition, dense fusion extracts RGB and depth features and fuses them at multiple levels, but this produces redundant information. Therefore, the RGBDF module is proposed. Its structure is shown in Figure 8 .

[0099] The three branches obtain confidence fusion features of different scales. Finally, multi-scale RGB-D feature aggregation is achieved by connecting the features extracted from the three branches and adding a few parameters.

[0100] The first branch generates a confidence map from the depth image. The element-wise addition value between the RGBD image and the depth image feature is calculated, and the output result is multiplied by the depth map to enhance the representation of the contour information.

[0101] In the second branch, the RGBDF module uses multiple convolutions to obtain features of different scales.

[0102] In the third branch, the RGBDF module calculates the absolute value of the element-wise subtraction between the RGBD image and the depth image features. The output is then multiplied by the depth image to further enhance the representation of edge semantic information.

[0103] Finally, the RGBDF concatenates the multiple feature maps obtained from the three branches. Then, these features are fused by a 3x3 convolution operation to obtain low-level RGB-D fusion features

[0104] As an embodiment, please refer to Figure 10 , Figure 10 is the structure diagram of the GFF module.

[0105] The FASP module outputs high-level semantic features with less geometric information. The RGBDF module outputs more geometric information with less semantic features. Using either kind of information alone is ineffective for picking detection. Existing robot picking point detection methods combine the two features through simple operations (e.g., addition or concatenation), but fail to fully utilize the characteristics of the two features. Based on the above problems, the GFF module is proposed. Figure 10 The structure is shown. First, the two input features X, Y are fused, and then a Sigmoid activation function is applied, with the output value between 0 and 1. Then the weighted average of X and Y is calculated. By assigning attention weights to the two different features, the features with inconsistent semantics and scales can be better fused, thereby improving the picking point prediction accuracy. The specific implementation of the GFF module is as follows.

[0106]

[0107] After HiLo(f HiLo ), the adaptive weight obtained is marked as H. The features created by weighting the input and are marked as F GFF .

[0108] It should be noted that the specific structure of the HiLo attention mechanism is shown in Figure 11 .

[0109] HiLo uses two effective attentions to decouple high / low frequencies in the feature map. As Figure 11As shown, in the upper path, several Heads are assigned to high-frequency attention (Hi-Fi) to capture fine-grained high frequencies through Local Window Self-Attention (e.g., 2x2 window), which is more efficient than standard multi-head self-attention. In the lower path, low-frequency attention (Lo-Fi) is implemented by first applying average pooling to each window to obtain low-frequency signals. Then, the remaining Heads are assigned to Lo-Fi to model the relationship between each Query position in the input feature map and the average-pooled low-frequency Key of each window. Benefiting from the reduction in Key and Value length, Lo-Fi complexity is significantly reduced. Finally, the refined high / low-frequency features are concatenated, and the result is forwarded to the subsequent layer.

[0110] The SimAM attention mechanism (SimAM Attention) is a lightweight, parameter-free convolutional neural network attention mechanism that generates attention weights by calculating the local self-similarity of feature maps. SimAM does not require the introduction of any additional parameters and can effectively enhance the processing of features.

[0111] S3: training the lightweight tea bud outline extraction and picking point detection model using the training set and a back propagation algorithm to obtain a trained model;

[0112] As an embodiment, the model uses multi-task loss for end-to-end training.

[0113] The multi-task loss function includes different sub-tasks, and the value of the loss function is the goal of model learning. However, different sub-tasks correspond to different scales. One way to solve this problem is to use weighted addition of sub-task loss functions instead of direct addition of sub-task loss functions. Through weighting, each loss function can be uniformly scaled. For example, for tasks with rapidly decreasing loss, a relatively small learning rate should be used, but for tasks with slowly decreasing loss, a relatively large learning rate should be used. The loss function L of picking point detection and instance segmentation is as follows:

[0114]

[0115] L grasp represents the picking point detection loss; L seg represents the instance segmentation loss. L grasp is the combination of regression loss for predicting picking quality and width and classification loss for predicting direction. L seg consists of four parts: L RPN , L cls , L box is generated by RPN for object detection, L maskis the mask loss. To balance the loss, two specific hyperparameters (l grasp , l seg ) are set. Since the picking point detection loss decreases faster than the segmentation loss, l grasp is set to 1.5 and l seg is set to 1.

[0116] The method is developed based on PyTorch, and all training and testing are performed using a single Nvidia RTX 3060 GPU. The batch size is 2, the learning rate is 0.001, and the weight decay is 0.001. 300 epochs are trained. The Adam optimizer is used to optimize the backpropagation network during training. The present application uses pre-trained ImageNet weights to initialize the backbone network, and freezes the parameters of the first two network modules (Conv 1, Layer 1).

[0117] S4: Obtain the image of the region to be picked, and use the trained model to obtain the 2D picking frame and tea bud contour extraction result, complete the tea bud contour extraction and picking point detection.

[0118] As an embodiment, the present application uses a trained lightweight tea bud contour extraction and picking point detection model to detect the region to be picked, obtains a 2D picking frame and tea bud contour extraction result, and completes tea bud contour extraction and picking point detection.

[0119] Based on the combination of field tea growth characteristics and picking requirements, a field famous tea picking robot for tea bud contour extraction and picking point detection is built, calibration experiments and picking experiments are carried out, and the picking performance of each link and the overall picking performance are evaluated.

[0120] The overall structure of the famous tea picking robot is shown in the following Figure 12 The present application proposes an intelligent tea picking robot overall design scheme of a track-type self-propelled mechanism plus a six-axis mechanical arm based on full consideration of tea garden terrain and tea picking technical requirements. The tea picking robot comprises a track-type self-propelled mechanism, a picking mechanical arm, a tea picking end effector, a negative pressure collecting device, and a vision system; wherein the track-type self-propelled mechanism comprises a driving wheel, a DC brushless driving motor, a bearing chamber, a frame, a supporting wheel, a guide wheel, a tensioning device, etc.; the tea picking end effector comprises a stepping motor, a four-link mechanism, a flexible fingertip, a negative pressure hose joint, etc.; the negative pressure collecting device comprises a detachable negative pressure collecting cover and a grommet plastic packaging barrel; the vision system comprises a depth camera and an angle-adjustable depth camera adapter plate.

[0121] Through the research on the tea leaf picking robot control system, a reference can be provided for the whole machine structure design and hardware selection. In order to realize the practical application requirements of the famous and excellent tea single bud precision picking robot, the intelligent tea leaf picking robot is controlled by adopting the scheme of FPGA+ARM, so that the intelligent tea leaf picking robot has the functions of motion, tender bud picking, perception interaction and the like.

[0122] After disassembling the functions of the whole system, the intelligent tea picking robot system is divided into two parts of tea leaf picking control and chassis motion control. The overall structure of the control system of the intelligent tea picking robot and the contents contained in each part are as shown in Figure 13 .

[0123] The camera and the clamp are fixed to the end of the mechanical arm using a customized connector. The flexible clamp is driven by a motor. The I / O control signal is connected to the output port of the mechanical arm controller. The controller is connected with the upper computer through a network cable. The upper computer sends instructions to control the camera to collect images and control the action of the mechanical hand and the flexible clamp. For the eye-hand system, the translation matrix T and the rotation matrix R between the camera coordinate system and the mechanical arm coordinate system are obtained through the eye-hand calibration. Through the eye-hand system, the picking point position and the grabbing route obtained by the recognition algorithm can be converted into the robot base coordinate system, so that the mechanical arm completes the picking task. The connection of the control system is as shown in Figure 14 .

[0124] Finally, please refer to Figure 15 , Figure 15 for the specific results of tea tender bud profile extraction and picking point detection and positioning.

[0125] Please refer to Figure 16 , Figure 16 is a hardware device working schematic diagram of the embodiment of the present application, and the hardware device specifically comprises: a tea profile extraction and picking point intelligent detection device 401, a processor 402 and a storage medium 403.

[0126] The tea profile extraction and picking point intelligent detection device 401 realizes the tea profile extraction and picking point intelligent detection method.

[0127] The processor 402 loads and executes the instructions and data in the storage medium 403 for realizing the tea profile extraction and picking point intelligent detection method.

[0128] The storage medium 403 stores instructions and data; and the storage medium 403 is used for realizing the tea profile extraction and picking point intelligent detection method.

[0129] The present application has the beneficial effects that a lightweight network is constructed, aiming at extracting the outline of tea buds and detecting the picking point, the network solves the problem that the outline of tea buds and the picking point cannot be quickly and accurately extracted when the environment of the tea garden changes, significantly improves the picking efficiency and the automation level, helps to reduce the mispicking and missed picking phenomenon, and further improves the economic benefits.

[0130] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for extracting tea contours and intelligently detecting picking points, characterized by: The method comprises the following steps: S1: Collect RGB original images, depth images, and RGBD images synthesized from depth images of tea tree buds, and construct a training set; S2: Based on the instance segmentation branch and the tea bud picking point detection branch, an end-to-end lightweight tea bud contour extraction and picking point detection model is constructed; S3: using the training set and the back propagation algorithm to train the lightweight tea bud contour extraction and picking point detection model to obtain a trained model; S4: Obtain an image of the area to be picked, use the trained model to obtain the 2D picking frame and tea bud contour extraction results, and complete the tea bud contour extraction and picking point detection; In step S1, the calculation formula for the RGBD image synthesized from the depth image is as follows: Where R, G, B, and D refer to the red, green, blue, and depth values ​​of each pixel, respectively; α is a preset parameter; C1, C2, and C3 are the new values ​​of each pixel in the three channels of the image; then, each pixel value in the new image is normalized from 0 to 255 to obtain an RGBD image; by visual inspection, the pixels of the background and foreground are analyzed to maximize the value of g value, and with the constraint of avoiding excessive color distortion, the parameter α is 0.70; (2) (3) in, g Represents the difference between background and foreground; w 0 represents the ratio of the number of background pixels to the number of pixels in the entire image; w 1 represents the ratio of the number of foreground pixels to the number of pixels in the entire image; μ 、 μ 0. μ 1 represents the average pixel value of the entire image, background, and foreground pixels respectively; The lightweight tea bud contour extraction and picking point detection model in step S2 includes: an instance-based segmentation branch and a tea bud picking point detection branch; Among them, the instance segmentation branch adopts the improved Mask-RCNN model; the tea bud picking point detection branch includes the RGBDF module, FASP module and GFF module; Based on the improved Mask-RCNN model, ResNet and BiP-FPN are used as the backbone network to extract feature maps. For the extracted feature maps, the Region Proposal Network (RPN) is used to predict the Region of Interest (ROI). The ROI is used to crop the feature maps, and finally the tea bud picking point detection branch is used to predict the object category. The input of the FASP module is the feature map output by BiP-FPN, and the output is the grasping candidate objects of different sizes; The input of the RGBDF module is the depth image and the RGBD image synthesized from the depth image, and the output is the reliability score assigned to the depth map; The input of the GFF module is the reliability scores assigned to the grasping candidates and depth maps of objects of different sizes, and the output is the fused high-level semantic features; The RGBDF module consists of three branches: The first branch generates a confidence map from the depth image; it calculates the element-wise addition between the RGBD image and the depth image features, and then multiplies the output with the depth image to enhance the representation of the contour information; In the second branch, the RGBDF module uses multiple convolutions to obtain features of different scales; In the third branch, the RGBDF module calculates the absolute value of the element-wise subtraction between the RGBD image and the depth image features; the output is then multiplied with the depth map; Finally, the RGBDF module connects the multiple feature maps obtained by the three branches and fuses these features through 3×3 convolution operations to obtain low-level RGB-D fusion features. ; In the BiP-FPN module, the multi-scale feature fusion module in the target detector is or Aggregate and fuse multi-level feature information of different resolutions in the form of In the GFF module, the two input features X and Y are first fused, and then the Sigmoid activation function is applied, with the output value between 0 and 1; the weighted average of X and Y is then calculated; and attention weights are assigned to the two different features.

2. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the tea contour extraction and picking point intelligent detection method described in claim 1.

3. A tea contour extraction and picking point intelligent detection device, characterized by: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the tea contour extraction and picking point intelligent detection method described in claim 1.

Images