Aubergine picking robot system based on binocular vision and picking method thereof

Abstract

The application discloses a kind of eggplant picking robot systems and picking method based on binocular vision, belong to eggplant picking technical field, including picking mechanical arm unit, motion chassis unit, binocular vision unit and computer main control unit;Picking mechanical arm unit includes mechanical arm, end execution unit and picking mechanical arm control module, end execution unit includes shear and gripper, for the picking of mature eggplant, picking mechanical arm control module is used to receive the 3d coordinate control mechanical arm and shear and gripper in end execution unit issued by computer main control unit and carry out picking operation.Picking robot can be carried by local computer main control unit equipment, using eggplant target detection model, can accurately identify the position of each part of eggplant and ripening information, accurately position shear point and clamping point position, greatly reduce the eggplant breakage rate in picking process, improve picking efficiency.

Description

Technical Field

[0001] This invention relates to the field of eggplant harvesting technology, specifically to an eggplant harvesting robot system based on binocular vision and its harvesting method. Background Technology

[0002] Eggplant is one of the main vegetables in northern my country and occupies an important position in agricultural production. Labor costs account for 45%-70% of the total cost of eggplant production. In recent years, with the shortage of labor, this proportion is likely to rise year by year. Therefore, harvesting robots will become an important direction for the automation and intelligentization of modern agriculture.

[0003] In existing technologies, harvesting robots often need to send data to the cloud for processing. This requires a strong signal at the harvesting location, necessitating a wide area network (WAN) or local area network (LAN) for communication. This network factor restricts the harvesting robot's operation, affecting harvesting efficiency. Furthermore, current technologies rely solely on identifying the fruit's location for harvesting. Since the relative position of the eggplant and its stem is somewhat uncertain, this method is unsuitable for eggplants and can easily damage them during harvesting.

[0004] The above problems urgently need to be solved. To address this, the present invention proposes an eggplant harvesting robot system based on binocular vision. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: how to solve the problem that the harvesting robot in the prior art is constrained by network factors when performing harvesting operations, thus affecting the harvesting efficiency, and the problem that it is difficult to use eggplants directly by simply recognizing the position of the fruit. The present invention provides an eggplant harvesting robot system based on binocular vision.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution: the present invention includes a picking robotic arm unit, a motion chassis unit, a binocular vision unit, and a computer main control unit;

[0007] The harvesting robotic arm unit includes a robotic arm, an end effector unit, and a harvesting robotic arm control module. The end effector unit includes a shearer and a gripper for harvesting mature eggplants. The harvesting robotic arm control module receives 3D coordinates from the computer main control unit to control the robotic arm and the shearer and gripper in the end effector unit to perform harvesting operations.

[0008] The motion chassis unit includes a motion chassis and a motion control module. The motion control module is used to receive control commands from the computer main control unit to drive the motion chassis to move.

[0009] The binocular vision unit includes a binocular camera and an image processing module. The binocular camera is used to acquire depth images and RGB images of the surrounding environment. The image processing module is used to acquire depth information of each pixel in the depth image and to preprocess the acquired RGB image before sending it to the computer main control unit.

[0010] The computer main control unit is used to deploy the eggplant target detection model. After processing the received RGB image by the eggplant target detection model, it identifies the stem and fruit parts of the mature eggplant in the image. It uses the region growing algorithm to segment the eggplant stem and fruit areas and determines the cutting position and gripping position. Then, it combines the depth information to determine the coordinates and width data of the cutting position and gripping position, and sends the data to the harvesting robotic arm unit.

[0011] Furthermore, the binocular camera includes an RGB camera and a depth camera. The RGB camera is used to acquire RGB images, and the depth camera is used to acquire depth images. The pixel positions of the RGB images and the depth images correspond one-to-one, and their resolutions and sizes are the same.

[0012] This invention also provides a harvesting method for an eggplant harvesting robot system based on binocular vision, used to harvest mature eggplants using the aforementioned eggplant harvesting robot system, comprising the following steps:

[0013] Step S1: The motion chassis unit receives the instruction from the computer main control unit and moves to the picking location;

[0014] Step S2: The binocular camera acquires depth images and RGB images of the surrounding environment. The image processing module preprocesses the RGB images and obtains the depth information of each pixel in the depth image. The preprocessed RGB images and the obtained depth information are then sent to the computer main control unit.

[0015] Step S3: The computer main control unit receives the pre-processed RGB image and inputs it into the pre-deployed eggplant target detection model. After processing by the eggplant target detection model, the stem and fruit parts of the mature eggplant in the RGB image are identified. The eggplant stem and fruit areas are segmented using the region growing algorithm, and the cutting position and clamping position are determined. Then, the coordinates and width data of the cutting position and clamping position are determined by combining the depth information, and the data is sent to the picking robotic arm unit.

[0016] Step S4: The picking robotic arm unit controls the robotic arm and end effector unit to pick eggplants according to the coordinates issued by the computer main control unit;

[0017] Step S5: The picking robotic arm unit places the picked fruit into the designated position reserved on the motion chassis, and receives the coordinates of the next eggplant from the computer main control unit, and then repeats step S4.

[0018] Step S6: After the harvest is completed, the motion chassis unit receives the instruction from the computer main control unit and moves to the next harvest location.

[0019] Furthermore, in step S3, the computer main control unit identifies the stem and fruit parts of a mature eggplant using an eggplant target detection model, segments the eggplant stem and fruit regions using a region growing algorithm, and determines the cutting and clamping positions. The specific process is as follows:

[0020] Step S31: Construct an RGB image dataset, including open-source RGB images of eggplants and RGB images of eggplants collected by a harvesting robot system. Filter these RGB images and label the filtered RGB images. Standardly, use rectangular boxes to label the eggplant stem and the eggplant fruit.

[0021] Step S32: Based on the yolov7_tiny model, modify the backbone network to the MobileNetV3_small network; replace the SPPCSPC module in the neck network with the AIFI module to obtain the eggplant target detection model;

[0022] Step S33: Train the eggplant target detection model and generate a network file;

[0023] Step S34: Deploy the generated network file on the computer main control unit;

[0024] Step S35: Input the received preprocessed RGB image into the eggplant target detection model;

[0025] Step S36: The eggplant target detection model identifies mature, harvestable eggplants based on the received image and outlines the eggplant stem and fruit with a rectangular frame;

[0026] Step S37: After identifying a mature eggplant, calculate the position of the target center based on the rectangular frame drawn by the eggplant target detection model. Using the region growing algorithm, use the target center position as the initial seed point to segment the foreground and background within the rectangular frame. Analyze the foreground images after segmenting the eggplant stem and fruit parts, select the thinnest part of the stem diameter in the foreground image of the eggplant stem at a set distance from the eggplant fruit part as the cutting position, and then obtain the corresponding clamping position of the fruit part based on the actual physical distance between the cutter and the clamp.

[0027] Furthermore, in step S32, the MobileNetV3_small network includes an initial convolutional layer, a bottleneck block, an extended convolutional layer, and a fully connected layer. In the initial convolutional layer, preliminary feature extraction and downsampling are performed on the input image. The bottleneck block includes multiple Bneck blocks. The feature map obtained from the initial convolutional layer is first extended in the Bneck block by an extension layer consisting of 1x1 pointwise convolutions to expand the number of channels, then spatial features are extracted by a depthwise separable convolutional layer, and finally the number of channels is compressed by a compression layer. This process is repeated through multiple Bneck blocks. In the extended convolutional layer, the feature map obtained from the bottleneck layer is upsized by 1x1 convolutions and max pooling layers. In the fully connected layer, the information in the feature map after processing by the extended convolutional layer is converted into classification scores.

[0028] Furthermore, in step S32, the AIFI module includes a position encoding module, a multi-head attention mechanism module, a normalization layer, and a feedforward layer. The matrix obtained after the feature map passes through the position encoding module is input to the multi-head attention mechanism module. The resulting matrix is ​​concatenated with the residual of the input matrix, and then normalized by the normalization layer. After that, it enters the feedforward layer for processing. Finally, the input and output residuals are concatenated and normalized.

[0029] Furthermore, in step S37, the specific processing procedure of the region growing algorithm is as follows:

[0030] Step S371: Calculate the position of the target center based on the rectangle drawn by the eggplant target detection model, take the target center position as the initial seed point, and take the pixel position where the initial seed point is located as the initial region;

[0031] Step S372: Starting from the current region boundary, determine whether the adjacent pixels are similar to the current region. If they are similar, add the pixel to the region.

[0032] Step S373: Update the boundary of the current region and continue to check for new neighboring pixels until no new neighboring pixels match, at which point the region stops growing.

[0033] Furthermore, in step S372, the similarity judgment criterion is whether the difference between the color, light intensity, or texture features of the adjacent pixel and the color, light intensity, or texture features of the current region pixel is within a set range.

[0034] Furthermore, in step S37, a mapping relationship between the depth image and the RGB image is constructed using the intrinsic and extrinsic parameters of the depth camera and the acquired depth image. The computer main control unit calculates the actual coordinates and actual width data of the shearing position and gripping position, and sends the data to the harvesting robotic arm control module. The mapping relationship is as follows:

[0035]

[0036] in, Let P be the spatial coordinates of point P in the depth camera coordinate system. Let P be the planar coordinates of point P in the pixel coordinate system of the depth image, f be the focal length of the depth camera, dx be the actual physical length per unit pixel in the column direction, and dy be the actual physical length per unit pixel in the row direction.

[0037] Furthermore, the specific process by which the computer control unit calculates the actual coordinates and actual width data of the shearing position and clamping position based on the mapping relationship includes the following steps:

[0038] S3701: Obtain the pixel coordinates of the cutting position obtained in step S37 as q(u,v), then the pixel coordinates of the gripping position are j(u,v+d), where d is the pixel distance between the cutter and the gripper.

[0039] S3702: The computer main control unit obtains the depth information Z corresponding to the cutting position and the gripping position based on the pixel coordinates of the cutting position and the gripping position and the depth information received from step S2. Pq and Z Pj ;

[0040] S3703: By substituting the pixel coordinates from step S3701 and the depth information from step S3702 into the above mapping relationship, the actual coordinates of the cutting position and the gripping position can be obtained.

[0041] Compared with the prior art, the present invention has the following advantages: the eggplant picking robot system and its picking method based on binocular vision can accurately identify the location and ripeness information of various parts of the eggplant and precisely locate the cutting point and clamping point by using the eggplant target detection model on the computer main control unit equipment carried by the picking robot. This greatly reduces the eggplant damage rate during the picking process and improves the picking efficiency. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of the eggplant harvesting robot system based on binocular vision in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the eggplant target detection model in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the bottleneck block structure in the MobileNetV3-Small network in an embodiment of the present invention;

[0045] Figure 4This is a schematic diagram of the AIFI module in an embodiment of the present invention. Detailed Implementation

[0046] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0047] like Figure 1 As shown, this embodiment provides a technical solution: an eggplant harvesting robot system based on binocular vision, including a harvesting robotic arm unit, a motion chassis unit, a binocular vision unit, and a computer main control unit; the harvesting robotic arm unit includes a robotic arm, an end effector unit, and a harvesting robotic arm control module. The end effector unit includes a shearer and a gripper for harvesting the target object. The harvesting robotic arm control module receives 3D coordinates from the computer main control unit to control the robotic arm and the shearer and gripper in the end effector unit to perform harvesting operations; the motion chassis unit includes a motion chassis and a motion control module. The motion control module receives control commands from the computer main control unit to drive the motion chassis to move; the binocular vision unit includes a binocular camera and an image processing module. The binocular camera is used to acquire images of the surrounding environment. The system receives image data (including RGB and depth images with a resolution of 1280x720 and consistent size, with pixel positions corresponding one-to-one). The image processing module acquires depth information corresponding to each pixel in the depth image and preprocesses the acquired RGB images (highlight reduction and noise reduction) before sending them to the main computer control unit. The main computer control unit deploys an eggplant target detection model. Based on the received RGB images, the model identifies the stem and fruit parts of mature eggplants in the image. It uses a region growing algorithm to segment the eggplant stem and fruit regions, determines the cutting and gripping positions, and then combines the depth information to determine the coordinates and width of the cutting and gripping positions. The data is then sent to the harvesting robotic arm unit.

[0048] The system workflow is as follows:

[0049] Step S1: The motion chassis unit receives the instruction from the computer main control unit and moves to the picking location;

[0050] Step S2: The binocular camera acquires depth images and RGB images of the surrounding environment. The image processing module obtains the depth information corresponding to each pixel based on the acquired depth image, and at the same time preprocesses the acquired RGB image. The preprocessed RGB image and the acquired depth information are then sent to the computer main control unit.

[0051] Step S3: The computer main control unit receives the pre-processed RGB image and inputs it into the pre-deployed eggplant target detection model. After processing by the eggplant target detection model, the stem and fruit parts of the mature eggplant in the RGB image are identified. The eggplant stem and fruit areas are segmented using the region growing algorithm, and the cutting position and clamping position are determined. Then, the coordinates and width of the cutting position and clamping position are determined by combining the depth information, and the data is sent to the picking robotic arm unit.

[0052] Step S4: The picking robotic arm unit controls the robotic arm and end effector unit to pick eggplants according to the coordinates issued by the computer main control unit;

[0053] Step S5: The picking robotic arm unit places the picked fruit into the designated position reserved on the motion chassis, and receives the coordinates of the next eggplant from the computer main control unit, and then repeats step S4.

[0054] Step S6: After the harvest is completed, the motion chassis unit receives the instruction from the computer main control unit and moves to the next harvest location.

[0055] In step S3, the computer main control unit identifies the stem and fruit parts of a mature eggplant using the eggplant target detection model, segments the eggplant stem and fruit regions using a region growing algorithm, and determines the cutting and clamping positions. The specific process is as follows:

[0056] Step S31: Construct an RGB image dataset, including open-source RGB images of eggplants and RGB images of eggplants collected by a harvesting robot. Filter these RGB images and label the filtered RGB images. The filtering is to select RGB images containing mature eggplants, and the eggplant stem and eggplant fruit parts are labeled with rectangles on the selected RGB images respectively.

[0057] Step S32: Based on the yolov7_tiny model, modify the backbone network to MobileNetV3_small; replace SPPCSPC in the neck network with the AIFI module to obtain the eggplant target detection model;

[0058] Step S33: Train the eggplant target detection model and generate a network file;

[0059] Step S34: Deploy the generated network file on the computer main control unit;

[0060] Step S35: Input the received preprocessed RGB image into the eggplant target detection model;

[0061] Step S36: The eggplant target detection model identifies mature, harvestable eggplants based on the received image and outlines the eggplant stem and fruit with a rectangular frame;

[0062] Step S37: After identifying mature, harvestable eggplants, calculate the target center position based on the rectangular frame drawn by the eggplant target detection model. Using the region growing algorithm, use the target center position as the initial seed point to segment the foreground and background within the rectangular frame. Analyze the foreground images after segmenting the eggplant stem and fruit parts, and select the thinnest part of the stem diameter in the foreground image of the eggplant stem at a distance of 1.5 to 2.5 cm from the eggplant fruit part as the cutting position. Then, based on the actual physical distance between the cutter and the gripper, find the corresponding gripping position for the fruit part to ensure that the cutting and gripping operations will not damage the eggplant fruit part.

[0063] In step S32, as Figure 2 The diagram shows the structure of the eggplant target detection model in this embodiment. The backbone network is the Backbone network, the Neck network is the Neck network, and the Head network is the Head network. This target detection model is based on the YOLOv7_tiny model. The backbone network uses the more lightweight MobileNetV3_small network, making the network model easier to deploy on the computer's main control unit and improving network speed. In the Neck network, the SPPC module is replaced with the AIFI module to improve network performance. The architecture of the MobileNetV3-Small network can be divided into several main parts, including the initial convolutional layer, the bottleneck block, the extended convolutional layer, and the final fully connected layer. The main structure of the bottleneck block is as follows: Figure 3 As shown, the initial convolutional layers and bottleneck blocks of the MobileNetV3-Small network are used as the backbone network of the eggplant target detection model. In the optimization of the neck network, the original SPPCSPC module is replaced with an AIFI (Attention-based Intrascale Feature Interaction) module, as shown below. Figure 4 As shown, this is used to improve network performance.

[0064] It should be noted that MobileNetV3 is a high-efficiency convolutional neural network architecture specifically designed for mobile and embedded devices. It combines depthwise separable convolutions, linear bottlenecks, and lightweight attention modules. The core of MobileNetV3 mainly consists of a series of Bneck modules. This invention uses the MobileNetV3-Small network, with the main difference being the change in the number of channels and the number of times the Bneck modules are used.

[0065] In this embodiment, the complete MobileNetV3_small network includes an initial convolutional layer, a bottleneck block, an extended convolutional layer, and a fully connected layer. In the initial convolutional layer, preliminary feature extraction and downsampling are performed on the input image. The bottleneck block includes multiple Bneck blocks. The feature map obtained by the initial convolutional layer is first extended in the Bneck block by an extension layer consisting of 1x1 pointwise convolutions to expand the number of channels, then spatial feature extraction is performed by a depthwise separable convolutional layer, and finally the number of channels is compressed by a compression layer. This process is repeated for several Bneck blocks (in this invention, the number of Bneck blocks is 9). In the extended convolutional layer, the feature map obtained by the bottleneck layer (i.e., the feature map output after passing through several Bneck blocks) is upsized by 1x1 convolutions and max pooling, and finally the information in the feature map is converted into a classification score by a fully connected layer.

[0066] The Bneck module typically includes the following steps, such as Figure 3 As shown, a 1x1 pointwise convolution (PW) is performed first, followed by a 1x1 depthwise convolution (DW). For some Bneck modules, a lightweight attention module is included to adaptively recalibrate the channel features. Finally, if the input and output shapes are the same, residual connections are performed.

[0067] In this embodiment, the main functions of the AIFI module are twofold: first, to reduce computational redundancy; and second, to capture the relationships between conceptual entities in the image by applying self-attention operations to high-level features with rich semantic concepts. For example... Figure 2 , 4 As shown, the feature map S5 obtained from the backbone network is downsampled in one step and then fed into the AIFI module. After positional encoding, the matrix is ​​input into the multi-head attention mechanism. The resulting matrix is ​​concatenated with the residual of the input matrix and then normalized. Then it enters the feedforward layer (FFN) for processing. Finally, the input and output residuals are concatenated and normalized.

[0068] The formula for constructing a feedforward layer is as follows:

[0069] FFN(x) = max(0, xW1+b1)W2+b2

[0070] Where x is the input matrix, W1 and b1 are the weights and biases of the first layer in the feedforward layer, and W2 and b2 are the weights and biases of the second layer in the feedforward layer.

[0071] The training process for the proposed eggplant object detection network model is as follows: A suitable number of eggplant images are collected and labeled. The labeling format is: number: 1, location: fruit / stem, maturity status: mature / not mature. A small subset of the labeled eggplant images is randomly selected as the validation set and not used in model training. The remaining images are used as the training set for model training. The training set is then fed into the eggplant object detection network for training. After the model converges, the validation set is fed into the object detection network for validation. Based on the results of the validation set, it is determined whether the network needs additional images for secondary training. If secondary training is required, the above process is repeated.

[0072] In step S37, the proposed region growing algorithm is used to segment the eggplant stem and fruit regions. Specifically, the target center position is calculated based on the bounding box drawn by the eggplant target detection model. This target center position is used as the initial seed point, and the pixel position of the initial seed point is used as the initial region. Starting from the current region boundary, it is determined whether adjacent pixels are similar to the current region. If similar, the pixel is added to the region. After adding new adjacent pixels to the region, the boundary of the current region is updated, and the process of checking for new adjacent pixels continues until no new adjacent pixels match, at which point region growing stops. The similarity criteria include, but are not limited to, color, light intensity, and texture.

[0073] In step S37, the mapping relationship between the depth image and the RGB image is constructed using the intrinsic and extrinsic parameters of the depth camera and the acquired depth image. The computer main control unit calculates the actual coordinates and actual width of the cutting position and gripping position, and sends the results to the picking robot arm control module.

[0074] The mapping relationship is as follows:

[0075]

[0076] in, Let P be the spatial coordinates of point P in the depth camera coordinate system. Let P be the planar coordinates of point P in the pixel coordinate system of the depth image, f be the focal length of the depth camera, dx be the actual physical length per unit pixel in the column direction, and dy be the actual physical length per unit pixel in the row direction.

[0077] More specifically, the process by which the computer control unit calculates the actual coordinates and actual width of the shearing and clamping positions based on the above mapping relationship is as follows:

[0078] 1. Obtain the pixel coordinates of the cutting position obtained in step S37 as q(u,v), then the pixel coordinates of the gripping position are j(u,v+d), where d is the pixel distance between the cutter and the gripper;

[0079] 2. The computer main control unit obtains the depth information Z corresponding to the cutting position and the gripping position based on the pixel coordinates of the cutting position and the gripping position and the depth information received from step S2. Pq and Z Pj ;

[0080] 3. By substituting the pixel coordinates and depth information into the above mapping relationship, the actual coordinates of the cutting position and the clamping position can be obtained.

[0081] In summary, the eggplant harvesting robot system based on binocular vision described in the above embodiments can accurately identify the location and ripeness information of various parts of the eggplant using an eggplant target detection model on the computer main control unit carried by the harvesting robot, and accurately locate the cutting point and gripping point, which greatly reduces the eggplant damage rate during the harvesting process and improves harvesting efficiency.

[0082] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A robot system for harvesting eggplants based on binocular vision, characterized in that, include: The harvesting robotic arm unit, motion chassis unit, binocular vision unit, and computer main control unit; The harvesting robotic arm unit includes a robotic arm, an end effector unit, and a harvesting robotic arm control module. The end effector unit includes a shearer and a gripper for harvesting mature eggplants. The harvesting robotic arm control module receives 3D coordinates from the computer main control unit to control the robotic arm and the shearer and gripper in the end effector unit to perform harvesting operations. The motion chassis unit includes a motion chassis and a motion control module. The motion control module is used to receive control commands from the computer main control unit to drive the motion chassis to move. The binocular vision unit includes a binocular camera and an image processing module. The binocular camera is used to acquire depth images and RGB images of the surrounding environment. The image processing module is used to acquire depth information of each pixel in the depth image and to preprocess the acquired RGB image before sending it to the computer main control unit. The computer main control unit is used to deploy the eggplant target detection model. After processing the received RGB image by the eggplant target detection model, it identifies the stem and fruit parts of the mature eggplant in the image. It uses the region growing algorithm to segment the eggplant stem and fruit areas, and determines the cutting position and gripping position. Then, it combines the depth information to determine the coordinates and width data of the cutting position and gripping position, and sends the data to the harvesting robotic arm unit. The process of harvesting mature eggplants using the aforementioned eggplant-harvesting robot system includes the following steps: Step S1: The motion chassis unit receives the instruction from the computer main control unit and moves to the picking location; Step S2: The binocular camera acquires depth images and RGB images of the surrounding environment. The image processing module preprocesses the RGB images and obtains the depth information of each pixel in the depth image. The preprocessed RGB images and the obtained depth information are then sent to the computer main control unit. Step S3: The computer main control unit receives the pre-processed RGB image and inputs it into the pre-deployed eggplant target detection model. After processing by the eggplant target detection model, the stem and fruit parts of the mature eggplant in the RGB image are identified. The eggplant stem and fruit areas are segmented using the region growing algorithm, and the cutting position and clamping position are determined. Then, the coordinates and width data of the cutting position and clamping position are determined by combining the depth information, and the data is sent to the picking robotic arm unit. Step S4: The picking robotic arm unit controls the robotic arm and end effector unit to pick eggplants according to the coordinates issued by the computer main control unit; Step S5: The picking robotic arm unit places the picked fruit into the designated position reserved on the motion chassis, and receives the coordinates of the next eggplant from the computer main control unit, and then repeats step S4. Step S6: After harvesting is completed, the motion chassis unit receives instructions from the computer main control unit and moves to the next harvesting location; In step S3, the computer main control unit identifies the stem and fruit parts of a mature eggplant using an eggplant target detection model, segments the eggplant stem and fruit regions using a region growing algorithm, and determines the cutting and clamping positions. The specific process is as follows: Step S31: Construct an RGB image dataset, including open-source RGB images of eggplants and RGB images of eggplants collected by a harvesting robot system. Filter these RGB images and label the filtered RGB images. Standardly, use rectangular boxes to label the eggplant stem and the eggplant fruit. Step S32: Based on the yolov7_tiny model, modify the backbone network to the MobileNetV3_small network; replace the SPPCSPC module in the neck network with the AIFI module to obtain the eggplant target detection model; Step S33: Train the eggplant target detection model and generate a network file; Step S34: Deploy the generated network file on the computer main control unit; Step S35: Input the received preprocessed RGB image into the eggplant target detection model; Step S36: The eggplant target detection model identifies mature, harvestable eggplants based on the received image and outlines the eggplant stem and fruit with a rectangular frame; Step S37: After identifying a mature eggplant, calculate the position of the target center based on the rectangle drawn by the eggplant target detection model. Using the region growing algorithm, use the target center position as the initial seed point to segment the foreground and background within the rectangle. Analyze the foreground images after segmenting the eggplant stem and fruit parts, select the thinnest part of the stem diameter in the foreground image of the eggplant stem at a set distance from the eggplant fruit part as the cutting position, and then obtain the corresponding clamping position of the fruit part based on the actual physical distance between the cutter and the clamp. In step S37, the specific processing procedure of the region growing algorithm is as follows: Step S371: Calculate the position of the target center based on the rectangle drawn by the eggplant target detection model, take the target center position as the initial seed point, and take the pixel position where the initial seed point is located as the initial region; Step S372: Starting from the current region boundary, determine whether the adjacent pixels are similar to the current region. If they are similar, add the pixel to the region. Step S373: Update the boundary of the current region, continue to check for new neighboring pixels until no new neighboring pixels match, and the region stops growing; In step S37, a mapping relationship between the depth image and the RGB image is constructed using the intrinsic and extrinsic parameters of the depth camera and the acquired depth image. The computer main control unit calculates the actual coordinates and actual width data of the shearing position and gripping position, and sends the data to the harvesting robotic arm control module. The mapping relationship is as follows: in, Let P be the spatial coordinates of point P in the depth camera coordinate system. Let P be the planar coordinates of point P in the pixel coordinate system of the depth image, f be the focal length of the depth camera, dx be the actual physical length in the column direction per unit pixel, and dy be the actual physical length in the row direction per unit pixel. The specific process by which the computer control unit calculates the actual coordinates and actual width data of the shearing position and clamping position based on the mapping relationship includes the following steps: S3701: Obtain the pixel coordinates of the cutting position obtained in step S37 as q(u,v), then the pixel coordinates of the gripping position are j(u,v+d), where d is the pixel distance between the cutter and the gripper. S3702: The computer main control unit obtains the depth information Z corresponding to the cutting position and the gripping position based on the pixel coordinates of the cutting position and the gripping position and the depth information received from step S2. Pq and Z Pj ; S3703: By substituting the pixel coordinates from step S3701 and the depth information from step S3702 into the above mapping relationship, the actual coordinates of the cutting position and the gripping position can be obtained.

2. The eggplant harvesting robot system based on binocular vision according to claim 1, characterized in that, The binocular camera includes an RGB camera and a depth camera. The RGB camera is used to acquire RGB images, and the depth camera is used to acquire depth images. The pixel positions of the RGB images and the depth images correspond one-to-one, and their resolutions and sizes are the same.

3. The eggplant harvesting robot system based on binocular vision according to claim 2, characterized in that, In step S32, the MobileNetV3_small network includes an initial convolutional layer, a bottleneck block, an extended convolutional layer, and a fully connected layer. In the initial convolutional layer, preliminary feature extraction and downsampling are performed on the input image. The bottleneck block includes multiple Bneck blocks. The feature map obtained from the initial convolutional layer is first expanded in the Bneck block by an extension layer consisting of 1x1 pointwise convolutions, then spatial features are extracted through a depthwise separable convolutional layer, and finally the number of channels is compressed through a compression layer. This process is repeated through multiple Bneck blocks. In the extended convolutional layer, the feature map obtained from the bottleneck layer is upscaled through 1x1 convolutions and max pooling layers. In the fully connected layer, the information in the feature map processed by the extended convolutional layer is converted into classification scores.

4. The eggplant harvesting robot system based on binocular vision according to claim 3, characterized in that, In step S32, the AIFI module includes a position encoding module, a multi-head attention mechanism module, a normalization layer, and a feedforward layer. The matrix obtained after the feature map passes through the position encoding module is input to the multi-head attention mechanism module. The resulting matrix is ​​concatenated with the residual of the input matrix, and then normalized by the normalization layer. After that, it enters the feedforward layer for processing. Finally, the input and output residuals are concatenated and normalized.

5. The eggplant harvesting robot system based on binocular vision according to claim 4, characterized in that, In step S372, the similarity judgment criterion is whether the difference between the color, light intensity, or texture features of the adjacent pixel and the color, light intensity, or texture features of the current region pixel is within a set range.

Images