Winter jujube tree girdling robot system based on binocular vision and working method

Abstract

The invention discloses a winter jujube tree girdling robot system based on binocular vision and a working method, and belongs to the technical field of robots. According to the technical key points, the girdling mechanical arm unit comprises a mechanical arm, an end effector and a mechanical arm control module, and the mechanical arm control module is used for receiving 3D coordinates issued by the computer main control unit to control the mechanical arm and the end effector so as to complete girdling operation; the binocular vision unit is used for acquiring an RGB image and a depth image of a winter jujube tree; the computer main control unit comprises a winter jujube tree girdling trace and pre-girdling position detection model; the winter jujube tree girdling trace and pre-girdling position detection model can identify three-dimensional coordinate information of a girdling trace part and a pre-girdling position in an image according to an RGB image and a depth image obtained by a binocular vision unit; the computer main control unit can send the three-dimensional coordinate information of the pre-girdling position to the girdling mechanical arm unit.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and more specifically, to a binocular vision-based winter jujube tree ring-barking robot system and a working method. Background Art

[0002] Ringing (or girdling) is a crucial measure for improving winter jujube quality and fruit set. However, it relies heavily on manual labor using small tools, and post-ringing tree care is tedious and untimely, resulting in high labor costs. Therefore, in this refined operation, girdling robots will become a key area of intelligent modern agriculture.

[0003] Based on this, the existing ring-peeling machinery is summarized as follows:

[0004] Document 1: Zhanhua County Winter Jujube Research Institute developed a ring-barking device for winter jujube trees in CN206005281U.

[0005] Document 2: Northwest Agriculture and Forestry University developed a jujube tree ring barking knife in CN206517853U.

[0006] Document 3: Shandong University of Technology developed a winter jujube ring-barking machine in CN104381032B.

[0007] From the above, we can see that the existing ring-barking machines have the problems of low degree of automation and limited application scenarios (when facing the main branches interlaced or the ring-barking position is too low, long-term work can easily cause great damage to the body).

[0008] Therefore, improving the intelligence of the ring-peeling robot requires solving the following problems:

[0009] First, how to realize the automation of ring-barking work.

[0010] Second, agricultural robots based on machine vision have been used in harvesting and other processes. However, they have not yet been applied to the girdling process. The difficulty lies in the complex environment of the date orchard. The girdling position is easily affected by factors such as previous girdling marks, tree nodules, branch forks, and foliage. Detecting the girdling position is uncertain and complex, making position detection a key challenge. Summary of the Invention

[0011] The purpose of this application is to address the deficiencies of the above-mentioned prior art and to provide a binocular vision-based winter jujube tree ring-barking robot system and working method.

[0012] The technical solution of this application is as follows:

[0013] A binocular vision-based jujube tree ring-barking robot system comprises: a ring-barking robot arm unit, a binocular vision unit and a computer main control unit; the ring-barking robot arm unit and the binocular vision unit are both bidirectionally electrically connected to the computer main control unit;

[0014] The ring-stripping robot arm unit includes: a robot arm, an end effector and a robot arm control module. The robot arm control module is used to receive 3D coordinates sent by the computer main control unit to control the robot arm and the end effector to complete the ring-stripping and nursing operations.

[0015] Among them, the binocular vision unit is used to obtain the RGB image and depth image of the winter jujube tree;

[0016] Among them, the computer main control unit includes: a winter jujube tree ring-barking mark and pre-ring-barking position detection model; the winter jujube tree ring-barking mark and pre-ring-barking position detection model can identify the ring-barking mark part and the three-dimensional coordinate information of the pre-ring-barking position in the image based on the RGB image and depth image obtained by the binocular vision unit; the computer main control unit can send the three-dimensional coordinate information of the pre-ring-barking position to the ring-barking robotic arm unit.

[0017] Furthermore, it also includes: the crawler chassis unit includes: a crawler chassis and a motion control module, and the motion control module is used to receive control instructions sent by the computer main control unit to drive the crawler chassis to move.

[0018] Furthermore, the binocular camera includes an RGB camera and a depth camera. The RGB camera is used to collect RGB images, and the binocular vision is used to collect depth images. The pixel positions of the RGB image and the depth image correspond one to one, and the resolution and size are the same.

[0019] Furthermore, the ring-barking robot arm unit also includes a care device; the care device includes an atomizing nozzle, which is suitable for fungicide and is used for surface spraying after ring-barking (phloem residual channel).

[0020] Furthermore, the winter jujube tree girdling trace and pre-girdling position detection model is based on the Yolov11 model. The backbone network is changed to the MobileNetV3-Small network, and a coordinate attention (CA) module is added. Some ordinary convolutions in the Head layer are replaced with GhostConv lightweight convolutions. A skeleton extraction algorithm module is added to extract the skeleton and determine whether there is an intersection between the main and side branches, thus obtaining the winter jujube tree girdling trace and pre-girdling position detection model.

[0021] Furthermore, the MobileNetV3-Small network includes: an initial convolutional layer, several inverted residual units, an SE module, a terminal convolutional layer, a global average pooling layer, and a fully connected layer;

[0022] In the initial convolutional layer, the input image is preliminarily processed;

[0023] The inverted residual unit consists of a 1×1 convolution (h-swish activation), a k×k depth-wise separable convolution (stride 1 or 2), and a 1×1 linear projection, and retains the residual connection when the number of input and output channels is the same and the stride is 1;

[0024] The SE module is located in the partial inverted residual unit and adaptively enhances features through global average pooling, two layers of point-by-point convolution (ReLU and h-sigmoid activation) and channel recalibration operations;

[0025] The terminal convolutional layer expands the number of feature map channels and applies h-swish activation;

[0026] The global average pooling layer performs average pooling on the terminal convolution feature map; the fully connected layer is used to output the detection result;

[0027] Furthermore, the Coordinate Attention (CA) module generates two feature maps that encode height and width spatial information respectively by horizontally and vertically global average pooling of the input feature map. After splicing the two feature maps, the channel attention weight is generated through 1×1 convolution and activation function, which is applied to highlight the key areas in the feature map.

[0028] Furthermore, the working method of the skeleton extraction algorithm module is: using the EightNeibor thinning algorithm to extract the skeleton of the input image and extract the intersection points of the main branches and side branches;

[0029] According to the principle of determining the girdling position, if the model has a girdling position that meets the requirements below the intersection, the girdling coordinates are determined;

[0030] If no girdling area is found below the intersection, a girdling position is determined on both the main branch and the side branch above the intersection.

[0031] Furthermore, a binocular vision-based ring-barking method using the aforementioned winter jujube tree ring-barking robot system comprises the following steps:

[0032] Step 1: The winter jujube tree ring-barking robot system moves to the location of the jujube tree to be ring-barked in the orchard;

[0033] Step 2: The binocular vision unit collects RGB images and depth images of the surrounding environment, and sends the collected RGB image information and depth information to the computer main control unit after preprocessing;

[0034] Step 3: The winter jujube tree girdling mark and pre-girdling position detection model of the computer main control unit reads the RGB image collected by the binocular vision unit to extract the branch skeleton and detect the position to be girdled. It then determines the three-dimensional coordinate information of the girdling position in combination with the depth image collected by the binocular vision unit, and finally sends the three-dimensional coordinate information of the girdling position to the picking robot arm unit;

[0035] Step 4: The ring-stripping robot arm unit controls the robot arm and the end effector to perform the operation according to the three-dimensional coordinate information of the ring-stripping position issued by the computer main control unit;

[0036] Step 5: After the ring-barking operation in the area is completed, move to the next ring-barking position.

[0037] Furthermore, step 4 further includes: after the ring-barking operation is completed, the ring-barking robot arm unit performs a care operation on the ring-barking part of the branch: spraying a fungicide (a mixture of carbendazim and seaweed extract) on the surface of the ring-barking part through an atomizing nozzle to form a protective film;

[0038] Furthermore, steps 1 and 5 both include: the crawler chassis unit receives instructions issued by the computer main control unit, and drives the winter jujube tree ring-barking robot system to move to a designated position.

[0039] Furthermore, the crawler chassis unit receives instructions from the chassis motion control module.

[0040] The beneficial effects of this application are:

[0041] The present invention proposes a binocular vision-based winter jujube tree ring-barking robot system, which can be locally deployed on resource-constrained computer equipment (industrial computers). It uses the winter jujube tree ring-barking traces and pre-ringing position detection model to accurately identify the position to be ringed, thereby improving the degree of automation of the ring-barking operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described in detail below with reference to the embodiments in the accompanying drawings, but this does not constitute any limitation to the present invention.

[0043] Figure 1 This is a structural topology diagram of a binocular vision-based jujube tree ring-barking robot system in this application.

[0044] Figure 2 This is a working process diagram of a binocular vision-based jujube tree ring-barking robot system in this application. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to specific embodiments.

[0046] Example 1: A winter jujube tree ring-barking robot system based on binocular vision.

[0047] A binocular vision-based winter jujube tree ring-barking robot system comprises a ring-barking robot arm unit, a crawler chassis unit, a binocular vision unit and a computer main control unit.

[0048] (1) The ring-stripping robot arm unit 100 includes: a robot arm 102, an end effector 103 (i.e., a ring stripper) and a robot arm control module 101. The robot arm control module is used to receive 3D coordinates sent by the computer main control unit to control the robot arm and the end effector to complete the ring-stripping operation.

[0049] (2) The crawler chassis unit 200 includes: a crawler chassis 202 and a motion control module 201. The motion control module is used to receive control instructions issued by the computer main control unit to drive the crawler chassis to move.

[0050] (3) The binocular vision unit 300 includes a binocular stereo vision camera 301 and an image processing module 302; the binocular vision is used to collect image data in the environment (including: RGB images and depth images with a resolution of 1280×720 and consistent size, and the pixel positions of the RGB images and the depth images correspond one to one), and the image processing module is used to obtain the depth information corresponding to each pixel of the depth image, and at the same time pre-process the collected RGB images (reducing highlight and noise) and then send them to the computer main control unit.

[0051] (4) The computer main control unit 400 is used to deploy a model for detecting the girdling marks and pre-girdling positions of winter jujube trees, detect the girdling marks and new girdling areas of winter jujube trees in the image after processing the received RGB image through the model, extract the branch skeleton and detect the pre-girdling position using the EightNeibor thinning algorithm, determine the real 3D coordinate data in combination with the depth information, and send the data to the girdling robot arm unit;

[0052] It should also be noted that the ring-barking robot arm unit may further include a care device 104; the care device includes an atomizing nozzle adapted for fungicide, which is used for spraying the surface of the ring-barking area (the remaining phloem channel).

[0053] like Figure 2 As shown, the system workflow is as follows:

[0054] Step a1: The crawler chassis unit receives the instruction from the computer main control unit and moves to the location of the jujube tree to be ringed in the orchard;

[0055] Step a2: The binocular camera collects depth images and RGB images of the surrounding environment. The image processing module preprocesses the collected RGB images and obtains the depth information of each pixel in the RGB images. The preprocessed RGB images and depth information are sent to the computer main control unit.

[0056] Step a3: The computer main control unit receives the pre-processed RGB image and transmits it to the pre-deployed winter jujube tree ring barking mark and pre-ring barking position detection model. After processing by the winter jujube tree ring barking mark and pre-ring barking position detection model, the ring barking mark and new ring barking area in the RGB image are identified; the branch skeleton and intersection are extracted using the EightNeibor refinement algorithm, and after excluding the ring barking mark and new ring barking area, the area to be ring barked is detected and output to determine the pre-ring barking position, and then the 3D coordinates of the pre-ring barking position are determined in combination with the depth information, and the data is sent to the ring barking robot arm unit.

[0057] Step a4: the ring-peeling robot arm unit controls the robot arm and the end effector to perform ring-peeling according to the coordinates issued by the computer main control unit;

[0058] Step a5: After the ring-barking robot arm unit completes the ring-barking operation, it performs a care operation on the ring-barked branches at the same position; then it receives the next operation coordinate from the computer main control unit, and then repeats step a4.

[0059] Step a6: After the girdling is completed, the crawler chassis unit receives the instruction from the computer main control unit and moves to the next location to be worked.

[0060] In step a3, the computer main control unit identifies the girdling marks through the winter jujube tree girdling marks and pre-girdling position detection model, uses the EightNeibor thinning algorithm to extract the branch skeleton and intersection points, and determines the position to be girdled. The specific process is as follows:

[0061] Step a31: constructing an RGB image dataset, including the collected RGB images of the jujube trees, screening and annotating these RGB images, and marking the ringing marks, the area to be ringed, and the new ringing area with rectangular boxes;

[0062] Step a32: Based on the Yolov11 model, the backbone network was changed to a MobileNetV3-Small network, and a Coordinate Attention (CA) module was added. Some common convolutions in the Head layer were replaced with GhostConv lightweight convolutions. A skeleton extraction algorithm was added to extract the skeleton and determine whether there were main branch intersections. After excluding girdling traces and new girdling areas, the area to be girdled was detected and output to obtain a model for detecting girdling traces and pre-girdling positions of winter jujube trees.

[0063] Step a33: training a model for detecting girdling marks and pre-girdling positions of winter jujube trees, and generating a network file;

[0064] Step a34: deploying the generated network file to the computer main control unit;

[0065] Step a35: transferring the received pre-processed RGB image into the winter jujube tree girdling mark and pre-girdling position detection model;

[0066] Step a36: The winter jujube tree girdling mark and pre-girdling position detection model identifies the jujube tree girdling marks based on the received image, extracts the intersection of the main branch and the side branch using a skeleton extraction algorithm, excludes the girdling marks and the new girdling area, and then detects and outputs the area to be girdled and its two-dimensional coordinates; then, the three-dimensional coordinates of the girdling position are determined based on the internal and external parameters of the depth camera; and the three-dimensional coordinates of the girdling position are also used as the coordinates for the post-girdling care operation;

[0067] Step a37: After identifying the two-dimensional coordinates of the position to be girdled, a skeleton extraction algorithm is used to extract the skeleton of the winter jujube tree including part of the trunk, and the intersection of the trunk, main branch and side branch is detected as a key point; according to the girdling method, the coordinates of the position to be girdled detected by the girdling traces of the winter jujube tree and the pre-girdling position detection model are spatially compared with the key points; if a detection result exists below the intersection of the main branch and the first side branch, it is preferentially selected as the position to be girdled; if no detection result exists below the intersection of the main branch and the first side branch, the main branch and the side branch above this intersection are considered to select the girdling position respectively;

[0068] In step a32, the yolo11_V3Small_CA+Ghost model for detecting girdling marks and pre-girdling positions on winter jujube trees includes a backbone network and a head network. This detection model is based on the YOLOv11_n model, replacing the backbone network with the lightweight MobileNetV3-Small network, enabling deployment with limited computing resources and improving model speed. A coordinate attention module (CA) is added after the 3rd, 7th, and 13th layers of the MobileNetV3-Smal backbone network, enhancing the representation of spatial and channel features and improving the model's detection accuracy and robustness. In the head network, the computationally expensive three-layer traditional convolution is replaced with the lightweight GhostConv convolution, significantly reducing the model's parameter count to just 10.98MB. This significantly improves the model's inference speed and meets the needs of computationally resource-constrained agricultural scenarios.

[0069] The above embodiments are preferred implementation modes of the present invention and are only used to facilitate the explanation of the present invention. They are not intended to limit the present invention in any form. Any person with ordinary knowledge in the technical field can, without departing from the scope of the technical features of the present invention, make partial changes or modifications to the technical contents disclosed in the present invention and make equivalent embodiments without departing from the technical features of the present invention. Such modifications still fall within the scope of the technical features of the present invention.

Claims

1. A binocular vision-based winter jujube tree ring-barking robot system, characterized in that: include: Ringing robot arm unit, binocular vision unit and computer main control unit; The ring-peeling robot arm unit and the binocular vision unit are both bidirectionally electrically connected to the computer main control unit; The ring-peeling robot arm unit includes: a robot arm, an end effector and a robot arm control module. The robot arm control module is used to receive the 3D coordinates sent by the computer main control unit to control the robot arm and the end effector to complete the ring-peeling operation. Among them, the binocular vision unit is used to obtain the RGB image and depth image of the winter jujube tree; Among them, the computer main control unit includes: a winter jujube tree ring-barking mark and pre-ring-barking position detection model; the winter jujube tree ring-barking mark and pre-ring-barking position detection model can identify the ring-barking mark part and the three-dimensional coordinate information of the pre-ring-barking position in the image based on the RGB image and depth image obtained by the binocular vision unit; the computer main control unit can send the three-dimensional coordinate information of the pre-ring-barking position to the ring-barking robotic arm unit.

2. The binocular vision-based jujube tree ring-peeling robot system according to claim 1, characterized in that: Also includes: The crawler chassis unit includes: a crawler chassis and a motion control module. The motion control module is used to receive control instructions sent by the computer main control unit to drive the crawler chassis to move.

3. The binocular vision-based jujube tree ring-peeling robot system according to claim 1, characterized in that: The binocular camera includes an RGB camera and a depth camera. The RGB camera is used to collect RGB images, and the binocular vision is used to collect depth images. The pixel positions of the RGB image and the depth image correspond one to one, and the resolution and size are the same.

4. The binocular vision-based winter jujube tree ring-barking robot system according to claim 1, characterized in that: The ring-barking mechanical arm unit also includes a care device; the care device includes an atomizing nozzle for spraying a fungicide on the surface after ring-barking.

5. The binocular vision-based winter jujube tree ring-barking robot system according to claim 1, characterized in that: The winter jujube tree girdling mark and pre-girdling position detection model is based on the Yolov11 model. The backbone network is changed to the MobileNetV3-Small network, and a coordinate attention CA module is added. Some ordinary convolutions in the Head layer are replaced with GhostConv lightweight convolutions. A skeleton extraction algorithm module is added to extract the skeleton and determine whether there are main and side branch intersections, thus obtaining the winter jujube tree girdling mark and pre-girdling position detection model.

6. The binocular vision-based winter jujube tree ring-barking robot system according to claim 5, characterized in that: The MobileNetV3-Small network includes: an initial convolutional layer, several inverted residual units, a SE module, a terminal convolutional layer, a global average pooling layer, and a fully connected layer; In the initial convolutional layer, the input image is preliminarily processed; The inverted residual unit consists of a 1×1 convolution, a k×k depth-wise separable convolution, and a 1×1 linear projection, and retains the residual connection when the number of input and output channels is the same and the stride is 1; The SE module is located in the partial inverted residual unit and adaptively enhances features through global average pooling, two-layer point-by-point convolution and channel recalibration operations; The terminal convolutional layer expands the number of feature map channels and applies h-swish activation; The global average pooling layer performs average pooling on the terminal convolution feature map; the fully connected layer is used to output the detection result.

7. A binocular vision-based winter jujube tree ring-barking robot system according to claim 5 or 6, characterized in that: The coordinate attention CA module generates two feature maps that encode height and width spatial information respectively by horizontally and vertically global average pooling of the input feature map. After splicing the two feature maps, it generates channel attention weights through 1×1 convolution and activation function, which are applied to highlight the key areas in the feature map.

8. The binocular vision-based jujube tree ring-peeling robot system according to claim 7, characterized in that: The working method of the skeleton extraction algorithm module is: using the EightNeibor thinning algorithm to extract the skeleton of the input image and extract the intersection points of the main branches and side branches; According to the principle of determining the girdling position, if the model has a girdling position that meets the requirements below the intersection, the girdling coordinates are determined; If no girdling area is found below the intersection, a girdling position is determined on both the main branch and the side branch above the intersection.

9. A binocular vision-based ring-barking method, which uses the winter jujube tree ring-barking robot system according to any one of claims 1 to 8, characterized in that: The steps include: Step 1: The jujube tree ring-barking robot system moves to the ring-barking position in the orchard; Step 2: The binocular vision unit collects RGB images and depth images of the surrounding environment, and sends the collected RGB image information and depth information to the computer main control unit after preprocessing; Step 3: The winter jujube tree girdling mark and pre-girdling position detection model of the computer main control unit reads the RGB image collected by the binocular vision unit to extract the branch skeleton and detect the position to be girdled, and then determines the three-dimensional coordinate information of the girdling position in combination with the depth image collected by the binocular vision unit. Finally, the three-dimensional coordinate information of the girdling position is sent to the girdling robot arm unit; Step 4: The ring-stripping robot arm unit controls the robot arm and the end effector to perform the operation according to the three-dimensional coordinate information of the ring-stripping position issued by the computer main control unit; Step 5: After the ring-barking operation in the area is completed, move to the next ring-barking position.

10. A binocular vision-based ring-peeling method according to claim 9, characterized in that: Step 4 also includes: after the ring-barking operation is completed, the ring-barking robot arm unit performs a care operation on the ring-barking part of the branch: spraying a fungicide: a mixture of carbendazim and seaweed extract on the surface of the ring-barking part through an atomizing nozzle to form a protective film; Steps 1 and 5 both include: the crawler chassis unit receives a command from the computer main control unit, driving the winter jujube tree ring-barking robot system to move to a designated position; The crawler chassis unit receives instructions from the chassis motion control module.

Images