Eggplant picking robot system based on binocular vision and picking method thereof

Through binocular vision technology and object detection model, combined with regional growth algorithm, efficient picking of eggplant picking robot system is achieved, solving the problems of network factors and eggplant damage, and improving the picking efficiency and accuracy.

CN118990494AActive Publication Date: 2024-11-22ANHUI UNIVERSITY OF TECHNOLOGY

Patent Information

Application Number
CN202411270994.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-22
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

Existing picking robots are restricted by network factors during picking operations, which affects the picking efficiency. It is difficult to apply to eggplant only by identifying the location of the fruit, which can easily lead to eggplant damage.

Method used

Adopting a binocular vision-based eggplant picking robot system, including a picking robot arm unit, a sports chassis unit, a binocular vision unit and a computer main control unit, depth images and RGB images are collected through a binocular camera, and image processing and object detection model are used to identify them. The eggplant stems and fruit parts are combined with the area growth algorithm to determine the shearing and clamping positions, and accurately locate and pick the operation.

Benefits of technology

Process data locally to accurately identify the eggplant location, reduce the damage rate, and improve the picking efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118990494A_ABST
    Figure CN118990494A_ABST
Patent Text Reader

Abstract

The invention discloses an eggplant picking robot system based on binocular vision and a picking method thereof, and belongs to the technical field of eggplant picking. The eggplant picking robot system comprises a picking mechanical arm unit, a moving chassis unit, a binocular vision unit and a computer main control unit; the picking mechanical arm unit comprises a mechanical arm, a tail end execution unit and a picking mechanical arm control module, and the tail end execution unit comprises a shearing device and a clamping device and is used for picking mature eggplants; the picking mechanical arm control module is used for receiving the 3d coordinates issued by the computer main control unit to control a mechanical arm and a shearing device and a clamping device in the tail end execution unit to conduct picking operation. According to the eggplant picking robot, the position and mature information of each part of the eggplant can be accurately identified and the positions of a shearing point and a clamping point can be accurately positioned on the computer main control unit equipment carried by the picking robot locally by using the eggplant target detection model, so that the eggplant breakage rate in the picking process is greatly reduced, and the picking efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of eggplant picking, and in particular to an eggplant picking robot system based on binocular vision and a picking method thereof. Background Art

[0002] Eggplant is one of the main vegetables in northern my country and plays an important role in agricultural production. In the production cost of eggplant, labor cost accounts for 45%-70% of the total cost of eggplant. In recent years, with the shortage of labor, this cost ratio may increase year by year. Therefore, harvesting robots will become an important direction for the automation and intelligence of modern agriculture.

[0003] In the prior art, the picking robot often needs to send data to the cloud for processing, which requires ensuring that the signal at the picking location is good and that there is a wide area network or local area network for communication. This will cause the picking robot to be restricted by network factors when performing picking operations, thereby affecting the picking efficiency. In addition, the prior art only directly picks the fruit by identifying its position. Since there is a certain uncertainty in the relative position of the eggplant and the eggplant stalk, it is difficult to directly pick the eggplant by only identifying its position, which can easily cause the eggplant to be damaged during the picking process.

[0004] The above problems need to be solved urgently. To this end, the present invention proposes an eggplant picking robot system based on binocular vision. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to solve the problem that the picking robots in the prior art are restricted by network factors when performing picking operations, thereby affecting the picking efficiency, and the problem that only identifying the position of the fruit to directly pick it is difficult to apply to eggplants. An eggplant picking robot system based on binocular vision is provided.

[0006] The present invention solves the above technical problems through the following technical solutions. The present invention includes a picking mechanical arm unit, a motion chassis unit, a binocular vision unit and a computer main control unit;

[0007] The picking robot arm unit includes a robot arm, an end-effector unit and a picking robot arm control module. The end-effector unit includes a shear and a gripper for picking ripe eggplants. The picking robot arm control module is used to receive 3D coordinates sent by the computer main control unit to control the robot arm and the shear and gripper in the end-effector unit to perform picking operations.

[0008] The motion chassis unit includes a motion chassis and a motion control module, and the motion control module is used to receive control instructions sent by the computer main control unit to drive the motion chassis to move;

[0009] The binocular vision unit includes a binocular camera and an image processing module. The binocular camera is used to collect depth images and RGB images of the surrounding environment. The image processing module is used to obtain the depth information of each pixel point of the depth image and pre-process the collected RGB image before sending it to the computer main control unit;

[0010] The computer main control unit is used to deploy the eggplant target detection model. The received RGB image is processed by the eggplant target detection model to identify the stalk and fruit part of the mature eggplant in the image, and the eggplant stalk and fruit area are segmented using the region growing algorithm, and the cutting position and clamping position are determined. The coordinates and width data of the cutting position and the clamping position are determined in combination with the depth information, and the data is sent to the picking robot arm unit.

[0011] Furthermore, the binocular camera includes an RGB camera and a depth camera, wherein the RGB camera is used to collect RGB images, and the depth camera is used to collect depth images, the pixel positions of the RGB image and the depth image correspond one to one, and the resolution and size are the same.

[0012] The present invention also provides a picking method of an eggplant picking robot system based on binocular vision, which is used to pick ripe eggplants using the above-mentioned eggplant picking robot system, and comprises the following steps:

[0013] Step S1: The motion chassis unit receives the instruction issued by the computer main control unit and moves to the location to be picked;

[0014] Step S2: The binocular camera collects a depth image and an RGB image of the surrounding environment, uses an image processing module to preprocess the RGB image, and obtains depth information of each pixel in the depth image, and sends the preprocessed RGB image and the obtained depth information to a computer main control unit;

[0015] Step S3: The computer main control unit transmits the received pre-processed RGB image to the pre-deployed eggplant target detection model, identifies the pedicle and fruit part of the mature eggplant in the RGB image after being processed by the eggplant target detection model, segments the eggplant pedicle and fruit area using the region growing algorithm, determines the cutting position and the clamping position, and then determines the coordinates and width data of the cutting position and the clamping position in combination with the depth information, and sends the data to the picking robot arm unit;

[0016] Step S4: the picking robot arm unit controls the robot arm and the end execution unit to pick the eggplant according to the coordinates sent by the computer main control unit;

[0017] Step S5: The picking robot arm unit places the picked fruit into the designated position reserved on the motion chassis, and receives the coordinates of the next eggplant from the computer main control unit, and then repeats step S4;

[0018] Step S6: After the picking is completed, the motion chassis unit receives the instruction sent by the computer main control unit and moves to the next picking location.

[0019] Furthermore, in step S3, the computer main control unit identifies the pedicle and fruit part of the mature eggplant through the eggplant target detection model, segments the eggplant pedicle and fruit region using the region growing algorithm, and determines the cutting position and the clamping position. The specific process is as follows:

[0020] Step S31: constructing an RGB image dataset, including open source RGB images of eggplants and RGB images of eggplants collected by a picking robot system, screening these RGB images and annotating the screened RGB images, and marking the eggplant stalk and eggplant fruit parts with rectangular frames respectively in a standard manner;

[0021] Step S32: Based on the yolov7_tiny model, the backbone network is modified to the MobileNetV3_small network; the SPPCSPC module in the neck network is replaced with the AIFI module to obtain the eggplant target detection model;

[0022] Step S33: training the eggplant target detection model and generating a network file;

[0023] Step S34: deploying the generated network file on the computer main control unit;

[0024] Step S35: passing the received pre-processed RGB image into the eggplant object detection model;

[0025] Step S36: the eggplant object detection model identifies the eggplants that are ripe and ready for picking based on the received image, and frames the eggplant stalk and fruit part with a rectangular frame;

[0026] Step S37: After the ripe eggplant is identified, the position of the target center is calculated according to the rectangular frame drawn by the eggplant target detection model. The region growing algorithm is used to take the target center position as the initial seed point to segment the foreground and background within the rectangular frame. The foreground image after the eggplant stalk and fruit part are segmented is analyzed, and the eggplant stalk foreground image is selected to find the thinnest part of the root diameter within the set range from the eggplant fruit part as the cutting position. Then, according to the actual physical distance between the shearer and the clamper, the clamping position corresponding to the fruit part is obtained.

[0027] Furthermore, in step S32, the MobileNetV3_small network includes an initial convolution layer, a bottleneck block, an extended convolution layer and a fully connected layer; in the initial convolution layer, preliminary feature extraction and downsampling are performed on the input image; the bottleneck block includes multiple Bneck blocks, and the feature map obtained by the initial convolution layer is first expanded in the Bneck block by an expansion layer composed of 1x1 point-by-point convolution, and then spatial feature extraction is performed through a depth-separable convolution layer, and finally the number of channels is compressed through a compression layer, and processed by multiple Bneck blocks according to the above process; in the extended convolution layer, the feature map obtained by the bottleneck layer is subjected to a dimensionality increase operation through a 1x1 convolution and a maximum pooling layer; in the fully connected layer, the information in the feature map processed by the extended convolution layer is converted into a classification score.

[0028] Furthermore, in step S32, the AIFI module includes a position encoding module, a multi-head attention mechanism module, a normalization layer, and a feedforward layer; the matrix obtained after the feature map passes through the position encoding module is input into the multi-head attention mechanism module, the obtained matrix and the matrix residual at the time of input are connected and then passed through the normalization layer for normalization, and then enters the feedforward layer for processing, and finally the input and output residuals are connected and normalized.

[0029] Furthermore, in step S37, the specific processing process of the region growing algorithm is as follows:

[0030] Step S371: Calculate the position of the target center according to the rectangular frame drawn by the eggplant target detection model, use the target center position as the initial seed point, and use the pixel position where the initial seed point is located as the initial area;

[0031] Step S372: starting from the boundary of the current region, determine whether the adjacent pixel is similar to the current region, and if so, add the pixel to the region;

[0032] Step S373: Update the boundary of the current region and continue to check new adjacent pixels until no new adjacent pixels match and the region stops growing.

[0033] Furthermore, in step S372, the similarity judgment basis is whether the color, light intensity or texture feature of the adjacent pixel is within a set range of the color, light intensity or texture feature difference with the color, light intensity or texture feature of the pixel in the current area.

[0034] Furthermore, in step S37, the mapping relationship between the depth image and the RGB image is constructed through the internal and external parameters of the depth camera and the acquired depth image, and the actual coordinates and actual width data of the shearing position and the clamping position are calculated by the computer main control unit, and the data is sent to the picking robot arm control module. The mapping relationship is:

[0035]

[0036] in, is the spatial coordinate of point P in the depth camera coordinate system, is the plane coordinate of point P in the pixel coordinate system of the depth image, f is the focal length of the depth camera, dx is the actual physical length of the unit pixel in the column direction, and dy is the actual physical length of the unit pixel in the row direction.

[0037] Furthermore, the specific process of the computer main control unit calculating the actual coordinates and actual width data of the shearing position and the clamping position according to the mapping relationship includes the following steps:

[0038] S3701: The pixel coordinates of the shearing position obtained in step S37 are obtained as q(u,v), and the pixel coordinates of the clamping position are j(u,v+d), where d is the pixel distance between the shearer and the clamper;

[0039] S3702: The computer main control unit obtains the depth information Z corresponding to the cutting position and the clamping position according to the pixel coordinates of the cutting position and the clamping position and the depth information received in step S2. Pq and Z Pj ;

[0040] S3703: The pixel coordinates in step S3701 and the depth information in step S3702 are brought into the above mapping relationship to obtain the actual coordinates of the shearing position and the clamping position.

[0041] Compared with the prior art, the present invention has the following advantages: the eggplant picking robot system based on binocular vision and the picking method thereof can use the eggplant target detection model locally on the computer main control unit device carried by the picking robot to accurately identify the position of each part of the eggplant and the maturity information, accurately locate the shearing point and the clamping point, greatly reduce the eggplant breakage rate during the picking process, and improve the picking efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic structural diagram of an eggplant picking robot system based on binocular vision in an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the structure of the eggplant target detection model in an embodiment of the present invention;

[0044] Figure 3 It is a schematic diagram of the structure of the bottleneck block in the MobileNetV3-Small network in an embodiment of the present invention;

[0045] Figure 4It is a schematic diagram of the structure of the AIFI module in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0047] like Figure 1 As shown, this embodiment provides a technical solution: an eggplant picking robot system based on binocular vision, including a picking robot arm unit, a motion chassis unit, a binocular vision unit and a computer main control unit; the picking robot arm unit includes a robot arm, an end execution unit and a picking robot arm control module, the end execution unit includes a shear and a gripper for picking target objects, and the picking robot arm control module is used to receive 3D coordinates issued by the computer main control unit to control the robot arm and the shear and gripper in the end execution unit to perform picking operations; the motion chassis unit includes a motion chassis and a motion control module, and the motion control module is used to receive control instructions issued by the computer main control unit to drive the motion chassis to move; the binocular vision unit includes a binocular camera and an image processing module, and the binocular camera is used to collect surrounding The image data of the environment (including RGB images and depth images with a resolution of 1280x720 and the same size, and the pixel positions of the RGB image and the depth image correspond one to one), the image processing module is used to obtain the depth information corresponding to each pixel of the depth image, and at the same time, the collected RGB image is pre-processed (highlight reduction, noise reduction) and then sent to the computer main control unit; the computer main control unit is used to deploy the eggplant target detection model, and the stalk and fruit part of the mature eggplant in the image are identified according to the received RGB image after being processed by the eggplant target detection model, and the eggplant stalk and fruit area are segmented by the regional growing algorithm, and the cutting position and the clamping position are determined, and then the coordinates and width of the cutting position and the clamping position are determined in combination with the depth information, and the data is sent to the picking robot arm unit.

[0048] The system workflow is as follows:

[0049] Step S1: The motion chassis unit receives the instruction issued by the computer main control unit and moves to the location to be picked;

[0050] Step S2: The binocular camera collects a depth image and an RGB image of the surrounding environment. The image processing module obtains the depth information corresponding to each pixel point according to the acquired depth image, and pre-processes the collected RGB image at the same time, and sends the pre-processed RGB image and the acquired depth information to the computer main control unit.

[0051] Step S3: The computer main control unit receives the preprocessed RGB image and transmits it to the pre-deployed eggplant target detection model. After being processed by the eggplant target detection model, the stalk and fruit of the mature eggplant in the RGB image are identified, and the eggplant stalk and fruit area are segmented using the region growing algorithm. The cutting position and the clamping position are determined, and the coordinates and width of the cutting position and the clamping position are determined in combination with the depth information, and the data is sent to the picking robot arm unit.

[0052] Step S4: the picking robot arm unit controls the robot arm and the end execution unit to pick the eggplant according to the coordinates sent by the computer main control unit;

[0053] Step S5: The picking robot arm unit places the picked fruits into the designated position reserved on the motion chassis, receives the coordinates of the next eggplant from the computer main control unit, and then repeats step S4.

[0054] Step S6: After the picking is completed, the motion chassis unit receives the instruction sent by the computer main control unit and moves to the next picking location.

[0055] In step S3, the computer main control unit identifies the pedicle and fruit part of the mature eggplant through the eggplant target detection model, segments the eggplant pedicle and fruit area using the region growing algorithm, and determines the cutting position and the clamping position. The specific process is as follows:

[0056] Step S31: constructing an RGB image dataset, including open-source RGB images of eggplants and RGB images of eggplants collected by a picking robot, screening these RGB images and annotating the screened RGB images, wherein the screening is to select RGB images containing mature eggplants, and annotating the eggplant stalks and eggplant fruit parts of the selected RGB images with rectangular frames;

[0057] Step S32: Based on the yolov7_tiny model, the backbone network is modified to MobileNetV3_small; the SPPCSPC in the neck network is replaced with the AIFI module to obtain the eggplant target detection model;

[0058] Step S33: training the eggplant target detection model and generating a network file;

[0059] Step S34: deploying the generated network file on the computer main control unit;

[0060] Step S35: passing the received pre-processed RGB image into the eggplant object detection model;

[0061] Step S36: the eggplant object detection model identifies the eggplants that are ripe and ready for picking based on the received image, and frames the eggplant stalk and fruit part with a rectangular frame;

[0062] Step S37: After identifying the ripe eggplants that can be picked, the position of the target center is calculated based on the rectangular frame drawn by the eggplant target detection model. The target center position is used as the initial seed point using the region growing algorithm to segment the foreground and background within the rectangular frame. The foreground image after the eggplant stalk and fruit part are segmented is analyzed, and the thinnest part of the rhizome diameter in the eggplant stalk foreground image at a distance of 1.5 to 2.5 cm from the eggplant fruit part is selected as the cutting position. Then, according to the actual physical distance between the shears and the clamp, the clamping position corresponding to the fruit part is found to ensure that the cutting and clamping operations will not cause damage to the eggplant fruit part.

[0063] In step S32, Figure 2 As shown, it is a structural diagram of the eggplant target detection model in this embodiment. The backbone network is the Backbone network, the neck network is the Neck network, and the head network is the Head network. This target detection model is based on the YOLOv7_tiny model. The backbone network adopts a more lightweight MobileNetV3_small network, which makes the network model easy to deploy on the computer main control unit and improves the network operation speed. In the neck network, the SPPCSPC module is replaced with the AIFI module to improve the performance of the network. The architecture of the MobileNetV3-Small network can be divided into several main parts, including the initial convolution layer, the bottleneck block, the extended convolution layer and the final fully connected layer. The main structure of the bottleneck block is as follows Figure 3 As shown in the figure. The initial convolutional layer and bottleneck block in the MobileNetV3-Small network are used as the backbone network of the eggplant object detection model. In the optimization of the neck network, the original SPPCSPC module is replaced with the AIFI (Attention-based Intrascale Feature Interaction) module, as shown in the figure. Figure 4 As shown, it is used to improve the performance of the network.

[0064] It should be noted that MobileNetV3 is an efficient convolutional neural network architecture specifically designed for mobile and embedded devices, which combines deep separable convolution, linear bottleneck and lightweight attention modules. The core of MobileNetV3 is mainly composed of a series of Bneck modules. The present invention uses the MobileNetV3-Small network. The main difference is the change in the number of channels and the number of times the Bneck module is used.

[0065] In this embodiment, the complete MobileNetV3_small network includes an initial convolution layer, a bottleneck block, an extended convolution layer and a fully connected layer. In the initial convolution layer, preliminary feature extraction and downsampling are performed on the input image; the bottleneck block includes multiple Bneck blocks, and the feature map obtained by the initial convolution layer is first expanded in the Bneck block by an extension layer composed of 1x1 point-by-point convolution, and then spatial feature extraction is performed through a depth-separable convolution layer, and finally the number of channels is compressed through a compression layer, and so on through several Bneck blocks (the number of Bneck blocks in the present invention is 9); in the extended convolution layer, the feature map obtained by the above-mentioned bottleneck layer (i.e., the feature map output after passing through several Bneck blocks) is subjected to dimensionality increase operations through 1x1 convolution and maximum pooling, and finally the information in the feature map is converted into a classification score through a fully connected layer.

[0066] The Bneck module usually includes the following steps: Figure 3 As shown, a 1x1 pointwise convolution (PW) is first performed, followed by a 1x1 depthwise separable convolution (DW). For some Bneck modules, a lightweight attention module is included to adaptively recalibrate channel features. Finally, if the input and output shapes are the same, a residual connection is performed.

[0067] In this embodiment, the main functions of the AIFI module are to reduce computational redundancy and to capture the connections between conceptual entities in the image by applying self-attention operations to high-level features with rich semantic concepts. Figure 2 , 4 As shown in the figure, the feature map S5 finally obtained by the backbone network is passed into the AIFI module after one step of downsampling. The matrix after positional encoding is input into the multi-head attention mechanism. The obtained matrix and the matrix residual at the time of input are connected (Add) and then normalized (Norm). Then it enters the feed forward layer (FFN) for processing. Finally, the input and output residuals are connected and normalized.

[0068] The construction formula of the feedforward layer is as follows:

[0069] FFN(x)=max(0,xW1+b1)W2+b2

[0070] Among them, x is the input matrix, W1 and b1 are the weights and biases of the first layer in the feedforward layer, and W2 and b2 are the weights and biases of the second layer in the feedforward layer.

[0071] The training process of the proposed eggplant target detection network model is as follows: collect an appropriate amount of eggplant pictures, mark the collected eggplant pictures, and the marking format is serial number: 1, part: fruit / stalk, maturity state: mature / unripe. Randomly extract a small part of the labeled eggplant pictures as a validation set, which does not participate in the model training. The remaining pictures are used as training sets for model training. The training set is put into the eggplant target detection network for training. When the model converges, the validation set is put into the target detection network for verification. According to the results of the validation set, choose whether the network needs to supplement pictures for secondary training. If secondary training is required, repeat the above process.

[0072] In step S37, for the proposed region growing algorithm to segment the eggplant pedicle and fruit region, the specific content is as follows: the position of the target center is calculated according to the rectangular frame drawn by the eggplant target detection model, the target center position is used as the initial seed point, and the pixel position where the initial seed point is located is used as the initial region. Starting from the current region boundary, it is determined whether the adjacent pixels are similar to the current region. If similar, the pixel is added to the region. After the new adjacent pixels are added to the region, the boundary of the current region is updated, and the new adjacent pixels are continued to be checked until no new adjacent pixels meet and the region stops growing. The similar judgment basis includes but is not limited to color, light intensity, texture, etc.

[0073] In step S37, the mapping relationship between the depth image and the RGB image is constructed through the internal and external parameters of the depth camera and the collected depth image. The computer main control unit calculates the actual coordinates and actual width of the cutting position and the clamping position, and sends the result to the picking robot arm control module.

[0074] The mapping relationship is:

[0075]

[0076] in, is the spatial coordinate of point P in the depth camera coordinates, is the plane coordinate of point P in the pixel coordinate system of the depth image, f is the focal length of the depth camera, dx is the actual physical length of the unit pixel in the column direction, and dy is the actual physical length of the unit pixel in the row direction.

[0077] More specifically, the specific process of the computer main control unit calculating the actual coordinates and actual width of the shearing position and the clamping position according to the above mapping relationship is as follows:

[0078] 1. The pixel coordinates of the shearing position obtained in step S37 are denoted as q(u,v), and the pixel coordinates of the clamping position are j(u,v+d), where d is the pixel distance between the shearer and the clamper;

[0079] 2. The computer main control unit obtains the depth information Z corresponding to the cutting position and the clamping position according to the pixel coordinates of the cutting position and the clamping position and the depth information received in step S2. Pq and Z Pj ;

[0080] 3. By bringing the pixel coordinates and depth information into the above mapping relationship, the actual coordinates of the shearing position and the clamping position can be obtained.

[0081] In summary, the eggplant picking robot system based on binocular vision in the above-mentioned embodiment can use the eggplant target detection model on the computer main control unit device carried by the picking robot locally to accurately identify the position of each part of the eggplant and the maturity information, and accurately locate the shearing point and the clamping point, which greatly reduces the breakage rate of eggplants during the picking process and improves the picking efficiency.

[0082] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An eggplant picking robot system based on binocular vision, characterized in that: include: Picking robot arm unit, motion chassis unit, binocular vision unit and computer main control unit; The picking robot arm unit includes a robot arm, an end-effector unit and a picking robot arm control module. The end-effector unit includes a shear and a gripper for picking ripe eggplants. The picking robot arm control module is used to receive 3D coordinates sent by the computer main control unit to control the robot arm and the shear and gripper in the end-effector unit to perform picking operations. The motion chassis unit includes a motion chassis and a motion control module, and the motion control module is used to receive control instructions sent by the computer main control unit to drive the motion chassis to move; The binocular vision unit includes a binocular camera and an image processing module. The binocular camera is used to collect depth images and RGB images of the surrounding environment. The image processing module is used to obtain the depth information of each pixel point of the depth image and pre-process the collected RGB image before sending it to the computer main control unit; The computer main control unit is used to deploy the eggplant target detection model. The received RGB image is processed by the eggplant target detection model to identify the stalk and fruit part of the mature eggplant in the image, and the eggplant stalk and fruit area are segmented using the region growing algorithm, and the cutting position and clamping position are determined. The coordinates and width data of the cutting position and the clamping position are determined in combination with the depth information, and the data is sent to the picking robot arm unit.

2. The eggplant picking robot system based on binocular vision according to claim 1 is characterized in that: The binocular camera includes an RGB camera and a depth camera, wherein the RGB camera is used to collect RGB images, and the depth camera is used to collect depth images. The pixel positions of the RGB image and the depth image correspond one to one, and the resolution and size are the same.

3. A picking method of eggplants by a binocular vision-based robot system, characterized in that: The method of picking ripe eggplants by using the eggplant picking robot system as claimed in claim 1 or 2 comprises the following steps: Step S1: The motion chassis unit receives the instruction issued by the computer main control unit and moves to the location to be picked; Step S2: The binocular camera collects a depth image and an RGB image of the surrounding environment, uses an image processing module to preprocess the RGB image, and obtains depth information of each pixel in the depth image, and sends the preprocessed RGB image and the obtained depth information to a computer main control unit; Step S3: The computer main control unit transmits the received pre-processed RGB image to the pre-deployed eggplant target detection model, identifies the pedicle and fruit part of the mature eggplant in the RGB image after being processed by the eggplant target detection model, segments the eggplant pedicle and fruit area using the region growing algorithm, determines the cutting position and the clamping position, and then determines the coordinates and width data of the cutting position and the clamping position in combination with the depth information, and sends the data to the picking robot arm unit; Step S4: the picking robot arm unit controls the robot arm and the end execution unit to pick the eggplant according to the coordinates sent by the computer main control unit; Step S5: The picking robot arm unit places the picked fruit into the designated position reserved on the motion chassis, and receives the coordinates of the next eggplant from the computer main control unit, and then repeats step S4; Step S6: After the picking is completed, the motion chassis unit receives the instruction sent by the computer main control unit and moves to the next picking location.

4. The picking method of the eggplant picking robot system based on binocular vision according to claim 3 is characterized in that: In step S3, the computer main control unit identifies the pedicle and fruit part of the mature eggplant through the eggplant target detection model, segments the eggplant pedicle and fruit area using the region growing algorithm, and determines the cutting position and the clamping position. The specific process is as follows: Step S31: constructing an RGB image dataset, including open source RGB images of eggplants and RGB images of eggplants collected by a picking robot system, screening these RGB images and annotating the screened RGB images, and marking the eggplant stalk and eggplant fruit parts with rectangular frames respectively in a standard manner; Step S32: Based on the yolov7_tiny model, the backbone network is modified to the MobileNetV3_small network; the SPPCSPC module in the neck network is replaced with the AIFI module to obtain the eggplant target detection model; Step S33: training the eggplant target detection model and generating a network file; Step S34: deploying the generated network file on the computer main control unit; Step S35: passing the received pre-processed RGB image into the eggplant object detection model; Step S36: the eggplant object detection model identifies the eggplants that are ripe and ready for picking based on the received image, and frames the eggplant stalk and fruit part with a rectangular frame; Step S37: After the ripe eggplant is identified, the position of the target center is calculated according to the rectangular frame drawn by the eggplant target detection model. The region growing algorithm is used to take the target center position as the initial seed point to segment the foreground and background within the rectangular frame. The foreground image after the eggplant stalk and fruit part are segmented is analyzed, and the eggplant stalk foreground image is selected to find the thinnest part of the root diameter within the set range from the eggplant fruit part as the cutting position. Then, according to the actual physical distance between the shearer and the clamper, the clamping position corresponding to the fruit part is obtained.

5. The picking method of an eggplant picking robot system based on binocular vision according to claim 4, characterized in that: In step S32, the MobileNetV3_small network includes an initial convolution layer, a bottleneck block, an extended convolution layer and a fully connected layer; in the initial convolution layer, preliminary feature extraction and downsampling are performed on the input image; the bottleneck block includes multiple Bneck blocks, and the feature map obtained by the initial convolution layer is first expanded in the Bneck block by an extension layer composed of 1x1 point-by-point convolution, and then spatial feature extraction is performed through a depth-separable convolution layer, and finally the number of channels is compressed through a compression layer, and is processed by multiple Bneck blocks according to the above process; in the extended convolution layer, the feature map obtained by the bottleneck layer is subjected to a dimensionality increase operation through a 1x1 convolution and a maximum pooling layer; in the fully connected layer, the information in the feature map processed by the extended convolution layer is converted into a classification score.

6. The picking method of an eggplant picking robot system based on binocular vision according to claim 4, characterized in that: In step S32, the AIFI module includes a position encoding module, a multi-head attention mechanism module, a normalization layer, and a feedforward layer; the matrix obtained after the feature map passes through the position encoding module is input into the multi-head attention mechanism module, the obtained matrix and the matrix residual at the time of input are connected and then passed through the normalization layer for normalization operation, and then enters the feedforward layer for processing, and finally the input and output residuals are connected and normalized.

7. The picking method of an eggplant picking robot system based on binocular vision according to claim 4, characterized in that: In step S37, the specific processing process of the region growing algorithm is as follows: Step S371: Calculate the position of the target center according to the rectangular frame drawn by the eggplant target detection model, use the target center position as the initial seed point, and use the pixel position where the initial seed point is located as the initial area; Step S372: starting from the boundary of the current region, determine whether the adjacent pixel is similar to the current region, and if so, add the pixel to the region; Step S373: Update the boundary of the current region and continue to check new adjacent pixels until no new adjacent pixels match and the region stops growing.

8. The picking method of the eggplant picking robot system based on binocular vision according to claim 7, characterized in that: In step S372, the similarity judgment is based on whether the color, light intensity or texture feature of the adjacent pixel is within a set range of the color, light intensity or texture feature difference with the color, light intensity or texture feature of the pixel in the current area.

9. The picking method of an eggplant picking robot system based on binocular vision according to claim 4, characterized in that: In step S37, the mapping relationship between the depth image and the RGB image is constructed through the internal and external parameters of the depth camera and the acquired depth image. The actual coordinates and actual width data of the shearing position and the clamping position are calculated by the computer main control unit, and the data is sent to the picking robot arm control module. The mapping relationship is: in, is the spatial coordinate of point P in the depth camera coordinate system, is the plane coordinate of point P in the pixel coordinate system of the depth image, f is the focal length of the depth camera, dx is the actual physical length of the unit pixel in the column direction, and dy is the actual physical length of the unit pixel in the row direction.

10. The picking method of the eggplant picking robot system based on binocular vision according to claim 9, characterized in that: The specific process of calculating the actual coordinates and actual width data of the shearing position and the clamping position by the computer main control unit according to the mapping relationship includes the following steps: S3701: The pixel coordinates of the shearing position obtained in step S37 are obtained as q(u,v), and the pixel coordinates of the clamping position are j(u,v+d), where d is the pixel distance between the shearer and the clamper; S3702: The computer main control unit obtains the depth information Z corresponding to the cutting position and the clamping position according to the pixel coordinates of the cutting position and the clamping position and the depth information received in step S2. Pq and Z Pj ; S3703: The pixel coordinates in step S3701 and the depth information in step S3702 are brought into the above mapping relationship to obtain the actual coordinates of the shearing position and the clamping position.

Citation Information

Patent Citations

  • Method of grabbing shape symmetrical workpiece with high precision by using low precision depth camera

    CN110014426A

  • Mechanical arm dual-mode control method and system based on vision and man-machine cooperation

    CN114419154A

  • Tomato picking method and system of binocular robot based on YOLOv4 algorithm

    CN114846998A

  • Litchi recognition method based on visual algorithm and bionic litchi picking robot

    CN115553132A

  • Method and device for automatic wiring based on visual guidance

    CN115629066A

Cited By

  • Multi-degree-of-freedom double-arm pear fruit picking robot based on agricultural machinery and agricultural technology fusion idea

    CN120660542A

  • Picking mechanical arm operation method and system based on visual feedback

    CN121018601A

  • Picking robot arm operation method and system based on visual feedback

    CN121018601B