Automatic tomato picking method and system based on machine vision

Abstract

The invention discloses an automatic tomato picking method and system based on machine vision. The method comprises the following steps: constructing an automatic tomato picking mobile robot model; the method comprises the following steps: acquiring color image data of tomatoes based on an automatic tomato picking mobile robot model, training a YOLOv8 target segmentation model, and constructing an AI model file; acquiring pose data of the tail end of a mechanical arm based on the tomato automatic picking mobile robot model, and performing hand-eye calibration to generate a calibration file; according to the AI model file and the calibration file, image coordinates of the target tomatoes are obtained, path planning is carried out, and automatic picking of the tomatoes is achieved. According to the embodiment of the invention, the position of the tomato can be accurately identified and can be combined with the tail end coordinate of the robot, so that the working efficiency and the obstacle avoidance capability of the robot in a complex environment are improved, and automatic picking of the tomato is realized. The method can be widely applied to the technical field of machine vision image processing.

Description

Technical Field

[0001] The present application relates to the technical field of machine vision image processing, and in particular to a method and system for automatic tomato picking based on machine vision. Background Art

[0002] At present, there are still many difficulties in picking tomatoes in glass greenhouses or greenhouses. Among them, intelligent fruit recognition and positioning are crucial. It is necessary to develop efficient image processing algorithms and new sensors to improve perception functions. In addition, the optimization design of the mechanical body must meet performance requirements and design a lightweight and efficient manipulator. In addition, path planning and motion control technologies also face challenges, and it is necessary to improve the robot's operating efficiency and obstacle avoidance capabilities in complex environments. The design of the end effector must take into account flexibility and protection of fruits and vegetables to avoid damaging plants. Furthermore, image recognition technology must cope with various interference factors in the natural environment to improve recognition rate and speed. Navigation and positioning technology needs to adapt to new cultivation methods and improve path planning and obstacle avoidance capabilities. The flexible structure of the robot must enhance versatility and meet a variety of operational requirements. However, related technologies cannot adapt to operations in complex environments, making it difficult to achieve automated picking.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and system for automatic tomato picking based on machine vision, which can accurately identify the position of tomatoes and combine them with the terminal coordinates of the robot, thereby improving the robot's operating efficiency and obstacle avoidance ability in complex environments and realizing automatic tomato picking.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a method for automatic tomato picking based on machine vision, the method comprising:

[0006] Build a mobile robot model for automatic tomato picking;

[0007] Acquire color image data of tomatoes based on the automatic tomato picking mobile robot model and train the YOLOv8 target segmentation model to build an AI model file;

[0008] Based on the tomato automatic picking mobile robot model, the position and posture data of the end of the robotic arm is obtained and hand-eye calibration is performed to generate a calibration file;

[0009] The image coordinates of the target tomato are obtained according to the AI ​​model file and the calibration file, and path planning is performed to realize automatic picking of tomatoes.

[0010] In some embodiments, the tomato automatic picking mobile robot model includes a mechanical arm, an end effector, an industrial computer, a camera and a light source. The industrial computer and the mechanical arm are wirelessly connected via a preset communication protocol. The first output end of the industrial computer is connected to the camera, the second output end of the industrial computer is connected to the light source, and the output end of the mechanical arm is connected to the end effector, wherein:

[0011] The mechanical arm is used to control the movement of the end effector;

[0012] The end effector is used to pick tomatoes;

[0013] The industrial computer is used to control the operation of the camera and the light source;

[0014] The camera is used to obtain color image data and depth image data of the tomato;

[0015] The light source is used to provide ambient light information.

[0016] In some embodiments, the method of acquiring color image data of tomatoes based on the automatic tomato picking mobile robot model and training the YOLOv8 target segmentation model to construct an AI model file includes:

[0017] Acquiring color image data of tomatoes through a camera of the automatic tomato picking mobile robot model and performing image preprocessing to obtain preprocessed color image data of tomatoes;

[0018] Inputting the preprocessed tomato color image data into the YOLOv8 object segmentation model for training;

[0019] Based on the backbone network module of the YOLOv8 target segmentation model, feature extraction processing is performed on the preprocessed tomato color image data to obtain tomato color feature image data;

[0020] Based on the neck network module of the YOLOv8 target segmentation model, feature fusion is performed on the tomato color feature image data to obtain fused tomato color feature image data;

[0021] Based on the head network module of the YOLOv8 target segmentation model, a non-maximum suppression method is introduced to detect the fused tomato color feature image data, output the tomato color feature image detection result, and generate an AI model file.

[0022] In some embodiments, the acquiring of the position data of the end of the robot arm based on the automatic tomato picking mobile robot model and the hand-eye calibration to generate a calibration file include:

[0023] The end effector of the automatic tomato picking mobile robot model with the calibration plate is moved and photographed by the camera of the automatic tomato picking mobile robot model to obtain the posture data of the end of the robot arm and the posture data of the robot arm base coordinate system;

[0024] Obtaining a transformation matrix between the posture data of the end of the robotic arm and the posture data of the robotic arm base coordinate system;

[0025] A calibration file is generated according to the transformation matrix.

[0026] In some embodiments, the method of acquiring the image coordinates of the target tomato according to the AI ​​model file and the calibration file and performing path planning to realize automatic picking of tomatoes includes:

[0027] Acquiring color image data and depth image data of a target tomato through a camera of the automatic tomato picking mobile robot model;

[0028] Inferring the color image data of the target tomato through the AI ​​model file to obtain the plane coordinates of the target tomato and the regional position of the target tomato;

[0029] According to the regional position of the target tomatoes, a tomato maturity detection process is performed to obtain the regional position of the mature target tomatoes;

[0030] Aligning the position of the mature target tomato region with the depth image data of the target tomato to obtain a depth value of the target tomato;

[0031] The depth value of the target tomato and the plane coordinates of the target tomato are planned by a fast expansion random number algorithm to achieve automatic picking of tomatoes.

[0032] In some embodiments, the inference of the color image data of the target tomato by using the AI ​​model file to obtain the plane coordinates of the target tomato and the regional position of the target tomato includes:

[0033] Performing size adjustment and normalization processing on the color image data of the target tomato through the OpenCV library to obtain normalized color image data of the target tomato;

[0034] Converting the normalized color image data of the target tomato into a multidimensional data matrix and adding a batch dimension to obtain multidimensional color image data of the target tomato;

[0035] The multi-dimensional data of the color image of the target tomato is input into the AI ​​model file for target detection to obtain the plane coordinates of the target tomato and the regional position of the target tomato.

[0036] In some embodiments, the tomato maturity detection process is performed according to the regional position of the target tomato to obtain the regional position of the mature target tomato, including:

[0037] Extracting the target tomato's area position to obtain the green channel area position of the target tomato and determining the total number of pixels of the target tomato;

[0038] Set the grayscale threshold parameter to determine the mature grayscale value range;

[0039] Scanning the green channel area position of the target tomato in a scroll bar manner to obtain the grayscale value of the target tomato;

[0040] Determining a target number of pixels of a target tomato according to the range of mature grayscale values;

[0041] The position of the mature target tomato region is determined according to the ratio of the target pixel number of the target tomato to the total number of pixels of the target tomato.

[0042] In some embodiments, the method of performing path planning on the depth value of the target tomato and the plane coordinates of the target tomato by using a fast expansion random number algorithm to achieve automatic picking of tomatoes includes:

[0043] Combining the depth value of the target tomato with the plane coordinates of the target tomato to generate the spatial coordinates of the target tomato;

[0044] The spatial coordinates of the target tomato are converted into the spatial coordinates of the end effector operation in the robot arm coordinate system by using the calibration file;

[0045] The automatic picking of tomatoes is achieved by planning the path of the spatial coordinates of the end effector operation in the robotic arm coordinate system through a rapid expansion of the random number algorithm.

[0046] In some embodiments, the expression of the fast expansion random number algorithm is specifically as follows:

[0047]

[0048] In the above formula, x r Indicates the random generation of nodes when generating new nodes, x n represents the child node closest to the random point, s represents the step size, x ne Represents the new node that is finally generated.

[0049] To achieve the above object, another aspect of the embodiment of the present application proposes an automatic tomato picking system based on machine vision, the system comprising:

[0050] The first module is used to build a mobile robot model for automatic tomato picking;

[0051] The second module is used to obtain color image data of tomatoes based on the automatic tomato picking mobile robot model and train the YOLOv8 target segmentation model to build an AI model file;

[0052] The third module is used to obtain the position data of the end of the robot arm based on the tomato automatic picking mobile robot model and perform hand-eye calibration to generate a calibration file;

[0053] The fourth module is used to obtain the image coordinates of the target tomato according to the AI ​​model file and the calibration file and perform path planning to achieve automatic picking of tomatoes.

[0054] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for automatic tomato picking based on machine vision. The solution constructs a mobile robot model for automatic tomato picking without the need to design overly complex manipulator equipment. The color image data of tomatoes is further obtained based on the mobile robot model for automatic tomato picking and the YOLOv8 target segmentation model is trained. The position of tomatoes can be accurately identified through the YOLOv8 target segmentation model. The position data of the end of the manipulator arm is obtained based on the mobile robot model for automatic tomato picking and the hand-eye calibration is performed. The position of the tomato can be effectively combined with the end coordinates of the robot to avoid damaging the plant. Finally, the image coordinates of the target tomato are obtained according to the AI ​​model file and the calibration file and the path planning is performed, thereby improving the robot's operating efficiency and obstacle avoidance capability in complex environments and realizing automatic tomato picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a method for automatic tomato picking based on machine vision provided in an embodiment of the present application;

[0056] Figure 2 is a structural schematic diagram of an automatic tomato picking system based on machine vision provided in an embodiment of the present application;

[0057] Figure 3 is a structural schematic diagram of a tomato automatic picking mobile robot model provided in an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of hand-eye calibration provided in an embodiment of the present application;

[0059] Figure 5 It is a schematic diagram of the process of generating AI model files and calibration files provided in an embodiment of the present application;

[0060] Figure 6 This is a simulation schematic diagram of judging the maturity of tomatoes provided in an embodiment of the present application;

[0061] Figure 7 It is a flowchart of automatic tomato picking based on path planning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0063] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0064] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] Reference Figure 1 , Figure 1 A flowchart of a method for automatic tomato picking based on machine vision provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0067] S100, build a mobile robot model for automatic tomato picking;

[0068] It should be noted that, in some embodiments, the automatic tomato picking mobile robot model includes a robotic arm, an end effector, an industrial computer, a camera and a light source. The industrial computer and the robotic arm are wirelessly connected via a preset communication protocol. The first output end of the industrial computer is connected to the camera, the second output end of the industrial computer is connected to the light source, and the output end of the robotic arm is connected to the end effector. The robotic arm is used to control the movement of the end effector; the end effector is used to pick tomatoes; the industrial computer is used to control the operation of the camera and the light source; the camera is used to obtain color image data and depth image data of tomatoes; and the light source is used to provide ambient light information.

[0069] In some specific embodiments, Figure 3 As shown, the camera is an RGBD camera, which can obtain color images, depth images and point clouds, which are used for algorithm processing to extract the posture of the picking point and the posture of obstacles. The robotic arm is used to reach the picking position based on the various posture information of visual feedback, combined with motion control algorithms and strategies. The end effector mainly realizes tomato picking through electrical control, etc. The industrial computer is used for calculation and control of the visual system. Among them, the light source uses neon tubes, which are mainly used to provide sufficient light for the RGBD camera to take pictures when working on cloudy days or at night.

[0070] S200, acquiring color image data of tomatoes based on the automatic tomato picking mobile robot model and training a YOLOv8 target segmentation model to construct an AI model file;

[0071] It should be noted that, in some embodiments, step S200 may include: S210, acquiring tomato color image data through the camera of the tomato automatic picking mobile robot model and performing image preprocessing to obtain preprocessed tomato color image data; S220, inputting the preprocessed tomato color image data into the YOLOv8 target segmentation model for training; S230, based on the backbone network module of the YOLOv8 target segmentation model, performing feature extraction processing on the preprocessed tomato color image data to obtain tomato color feature image data; S240, based on the neck network module of the YOLOv8 target segmentation model, performing feature fusion on the tomato color feature image data to obtain fused tomato color feature image data; S250, based on the head network module of the YOLOv8 target segmentation model, introducing a non-maximum suppression method, detecting the fused tomato color feature image data, outputting the tomato color feature image detection result, and generating an AI model file.

[0072] In some specific embodiments, Figure 5As shown in the figure, first, the device reaches the target operation point and triggers the RGBD camera to take pictures and obtain color images. The same operation method is used to obtain more than 1,000 tomato images. Then, the tomato images are annotated and trained using the YOLOv8 target segmentation model. The process is as follows:

[0073] 1) Dataset preparation: Prepare labeled datasets; perform data preprocessing, using image scaling, normalization, and enhancement to improve the robustness and accuracy of the model.

[0074] 2) Model architecture: YOLOv8 uses an improved YOLO feature extraction network, which usually includes a backbone network (such as CSPNet) for feature extraction and a head network for predicting bounding boxes and category probabilities. Backbone: Responsible for extracting image features, which may use convolutional layers, activation functions, and other layers (such as BatchNorm). Neck: Integrates feature information of different scales through techniques such as feature pyramids.

[0075] 3) Feature extraction: The input image is passed into the backbone network for feature extraction. This process generates multiple feature maps through a series of convolution operations.

[0076] 4) Object Detection: Bounding Box Regression: Predict the bounding box position of each object, usually represented by a coordinate offset. Category Classification: Classify each bounding box to determine whether there is an object inside it and its category.

[0077] 5) Non-maximum suppression (NMS): Since the same object may be detected by multiple bounding boxes, it is necessary to apply the non-maximum suppression algorithm to remove redundant bounding boxes and keep only the most likely box.

[0078] 6) Then, after the training is completed, an AI model file is generated and saved.

[0079] S300, based on the tomato automatic picking mobile robot model, obtaining the position and posture data of the end of the robotic arm and performing hand-eye calibration to generate a calibration file;

[0080] It should be noted that, in some embodiments, step S300 may include: S310, using the camera of the automatic tomato picking mobile robot model to move and shoot the end effector of the automatic tomato picking mobile robot model with a calibration plate to obtain the posture data of the end of the robotic arm and the posture data of the base coordinate system of the robotic arm; S320, obtaining the transformation matrix of the posture data of the end of the robotic arm and the posture data of the base coordinate system of the robotic arm; S330, generating a calibration file according to the transformation matrix.

[0081] In some specific embodiments, Figure 5As shown, the end effector of the robot arm clamps the calibration plate and moves it into the field of view of the RGBD camera. The calibration plate moves once, and the RGBD camera takes a photo once to obtain a color image, and records a set of posture data of the end of the robot arm. The calibration plate moves to multiple positions and angles. The same operation method is used to obtain more than 15 calibration plate images and more than 15 sets of posture data of the end of the robot arm. Then, hand-eye calibration is performed. After the calibration is successful, a calibration file is generated and saved, as shown in the figure. Figure 4 As shown, the process is:

[0082] 1) Use a 7*9 chessboard calibration plate to fix it on the end effector of the robot arm;

[0083] 2) Use an RGBD camera to take 20 sets of calibration plate images in different robot arm postures to obtain the external parameter RT of the chessboard;

[0084] 3) At the same time, record the position of the robot arm base coordinate system at each photo shoot;

[0085] 4) Solution target: Transformation matrix from camera coordinate system to robot base coordinate system base M camera ;

[0086] Specifically, for each group of pictures, there is the following relationship:

[0087]

[0088] Among them, base represents the base coordinate system of the robot arm, end represents the end coordinate system of the robot arm, camera represents the camera coordinate system, and board represents the calibration board coordinate system.

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] We can get a classic system of equations AX = XB, which contains n-1 equations, and the form is transformed:

[0095]

[0096] Then AX=XB can be decomposed into the following two equations:

[0097]

[0098] Solving for this yields:

[0099]

[0100] In the above formula, base M camera Represents a transformation matrix.

[0101] S400, obtaining the image coordinates of the target tomato according to the AI ​​model file and the calibration file and performing path planning to realize automatic picking of tomatoes;

[0102] It should be noted that in some embodiments, Figure 7 As shown, step S400 may include: S410, acquiring color image data and depth image data of a target tomato through a camera of the automatic tomato picking mobile robot model;

[0103] S420, inferring the color image data of the target tomato through the AI ​​model file to obtain the plane coordinates of the target tomato and the regional position of the target tomato;

[0104] Specifically, step S420 also includes: S421, resizing and normalizing the color image data of the target tomato through the OpenCV library to obtain the normalized color image data of the target tomato; S422, converting the normalized color image data of the target tomato into a multidimensional data matrix and adding a batch dimension to obtain the color image multidimensional data of the target tomato; S423, inputting the multidimensional color image data of the target tomato into the AI ​​model file for target detection to obtain the plane coordinates of the target tomato and the regional position of the target tomato.

[0105] In this embodiment, the device runs to the target operation point, triggers the RGBD camera to take pictures and obtain color images, depth images, and point clouds. Then, the AI ​​model is loaded to infer the color image and segment the coordinates of the tomato string in the image (cam x ,cam y ) and tomato bunch area, e.g. Figure 6 As shown. The model inference process is to first load the model, load the pre-trained YOLOv8 model; input data preparation, read the image to be detected, use the OpenCV library; resize the image (usually resized to 640x640) and normalize it to meet the model input requirements. Model inference, convert the pre-processed image to Tensor and add the batch dimension. Use the model for forward propagation to get the prediction result. Parse the result, parse the model output, including bounding box coordinates, confidence score and category information.

[0106] S430, performing tomato maturity detection processing according to the regional position of the target tomato, and obtaining the regional position of the mature target tomato;

[0107] Specifically, step S430 also includes: S431, extracting and processing the regional position of the target tomato to obtain the green channel regional position of the target tomato and determining the total number of pixels of the target tomato; S432, setting the grayscale threshold parameter to determine the mature grayscale value range; S433, scanning the green channel regional position of the target tomato by means of a scroll bar to obtain the grayscale value of the target tomato; S434, determining the target number of pixels of the target tomato according to the mature grayscale value range; S435, determining the mature target tomato regional position according to the ratio of the target number of pixels of the target tomato to the total number of pixels of the target tomato.

[0108] In this embodiment, the maturity of tomatoes is calculated by extracting the green channel image from the color image, designing the gray threshold gray_value parameter on the software interface, and using the scroll bar to drag left and right to adjust. The gray value in the range of [0, gray_value] is characterized as ripe. The total number of pixels with gray value ∈ [0, gray_value] in the tomato string area in the image is N, and the total number of pixels in the tomato string area is M. The value of N / M is calculated to represent the maturity of the tomato, as follows Figure 6 The maturity shown is greater than 90% and is considered as the picking target.

[0109] S440, aligning the position of the mature target tomato region with the depth image data of the target tomato to obtain a depth value of the target tomato;

[0110] In this embodiment, it is determined whether there are tomatoes of OK maturity. If not, it moves to the next position. If so, the depth value of the tomato is obtained by aligning the color image and the depth image (alignment conversion built into the RGBD camera system). z .

[0111] S450, performing path planning on the depth value of the target tomato and the plane coordinates of the target tomato by rapidly expanding the random number algorithm, thereby realizing automatic picking of tomatoes.

[0112] Specifically, step S450 also includes: S451, combining the depth value of the target tomato and the plane coordinates of the target tomato to generate the spatial coordinates of the target tomato; S452, converting the spatial coordinates of the target tomato into the spatial coordinates of the end effector operation in the robot arm coordinate system through the calibration file; S453, performing path planning on the spatial coordinates of the end effector operation in the robot arm coordinate system through the rapid expansion random number algorithm to realize automatic picking of tomatoes.

[0113] In some specific embodiments, the calibration file is called to convert the image coordinates of the tomato (cam x ,cam y ,cam z) is converted to the spatial coordinates of the end effector operation in the robot arm coordinate system (robot x ,robot y ,robot z ).

[0114] Then, the path planning of the robot arm is calculated using the Rapidly Expanding Random Number (RRT) algorithm. RRT is an efficient planning method in multidimensional space. It uses an initial point as the root node and generates a random expansion tree by randomly sampling and adding leaf nodes. The random expansion tree continues to grow until it reaches the destination. In this way, the sample points are continuously randomly expanded until a path connecting the starting point and the end point is generated. The formula used by the RRT algorithm is as follows:

[0115]

[0116] In the above formula, x r Indicates the random generation of nodes when generating new nodes, x n represents the child node closest to the random point, s represents the step size, x ne Represents the new node that is finally generated.

[0117] Finally, the robotic arm moves with the end effector to the target position to pick the tomatoes.

[0118] See also Figure 2 The embodiment of the present application also provides a machine vision-based automatic tomato picking system, which can implement the above-mentioned machine vision-based automatic tomato picking method. The system includes:

[0119] The first module 201 is used to build a mobile robot model for automatic tomato picking;

[0120] The second module 202 is used to obtain color image data of tomatoes based on the tomato automatic picking mobile robot model and train the YOLOv8 object segmentation model to build an AI model file;

[0121] The third module 203 is used to obtain the position and posture data of the end of the robot arm based on the tomato automatic picking mobile robot model and perform hand-eye calibration to generate a calibration file;

[0122] The fourth module 204 is used to obtain the image coordinates of the target tomato according to the AI ​​model file and the calibration file and perform path planning to achieve automatic picking of tomatoes.

[0123] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for automatic tomato picking based on machine vision, characterized in that: The method comprises the following steps: Build a mobile robot model for automatic tomato picking; Acquire color image data of tomatoes based on the automatic tomato picking mobile robot model and train the YOLOv8 target segmentation model to build an AI model file; Based on the tomato automatic picking mobile robot model, the position and posture data of the end of the robotic arm is obtained and hand-eye calibration is performed to generate a calibration file; The image coordinates of the target tomato are obtained according to the AI ​​model file and the calibration file, and path planning is performed to realize automatic picking of tomatoes.

2. The method according to claim 1, characterized in that The tomato automatic picking mobile robot model includes a mechanical arm, an end effector, an industrial computer, a camera and a light source. The industrial computer and the mechanical arm are wirelessly connected via a preset communication protocol. The first output end of the industrial computer is connected to the camera, the second output end of the industrial computer is connected to the light source, and the output end of the mechanical arm is connected to the end effector, wherein: The mechanical arm is used to control the movement of the end effector; The end effector is used to pick tomatoes; The industrial computer is used to control the operation of the camera and the light source; The camera is used to obtain color image data and depth image data of the tomato; The light source is used to provide ambient light information.

3. The method according to claim 1, characterized in that The method of acquiring color image data of tomatoes based on the automatic tomato picking mobile robot model and training the YOLOv8 target segmentation model to construct an AI model file includes: Acquiring color image data of tomatoes through a camera of the automatic tomato picking mobile robot model and performing image preprocessing to obtain preprocessed color image data of tomatoes; Inputting the preprocessed tomato color image data into the YOLOv8 object segmentation model for training; Based on the backbone network module of the YOLOv8 target segmentation model, feature extraction processing is performed on the preprocessed tomato color image data to obtain tomato color feature image data; Based on the neck network module of the YOLOv8 target segmentation model, feature fusion is performed on the tomato color feature image data to obtain fused tomato color feature image data; Based on the head network module of the YOLOv8 target segmentation model, a non-maximum suppression method is introduced to detect the fused tomato color feature image data, output the tomato color feature image detection result, and generate an AI model file.

4. The method according to claim 1, characterized in that: The method of acquiring the position data of the end of the robot arm based on the automatic tomato picking mobile robot model and performing hand-eye calibration to generate a calibration file includes: The end effector of the automatic tomato picking mobile robot model with the calibration plate is moved and photographed by the camera of the automatic tomato picking mobile robot model to obtain the posture data of the end of the robot arm and the posture data of the robot arm base coordinate system; Obtaining a transformation matrix between the posture data of the end of the robotic arm and the posture data of the robotic arm base coordinate system; A calibration file is generated according to the transformation matrix.

5. The method according to claim 1, characterized in that The method of obtaining the image coordinates of the target tomato according to the AI ​​model file and the calibration file and performing path planning to realize automatic picking of tomatoes includes: Acquiring color image data and depth image data of a target tomato through a camera of the automatic tomato picking mobile robot model; Inferring the color image data of the target tomato through the AI ​​model file to obtain the plane coordinates of the target tomato and the regional position of the target tomato; According to the regional position of the target tomatoes, a tomato maturity detection process is performed to obtain the regional position of the mature target tomatoes; Aligning the position of the mature target tomato region with the depth image data of the target tomato to obtain a depth value of the target tomato; The depth value of the target tomato and the plane coordinates of the target tomato are planned by a fast expansion random number algorithm to achieve automatic picking of tomatoes.

6. The method according to claim 5, characterized in that The inference of the color image data of the target tomato by using the AI ​​model file to obtain the plane coordinates of the target tomato and the regional position of the target tomato includes: Performing size adjustment and normalization processing on the color image data of the target tomato through the OpenCV library to obtain normalized color image data of the target tomato; Converting the normalized color image data of the target tomato into a multidimensional data matrix and adding a batch dimension to obtain multidimensional color image data of the target tomato; The multi-dimensional data of the color image of the target tomato is input into the AI ​​model file for target detection to obtain the plane coordinates of the target tomato and the regional position of the target tomato.

7. The method according to claim 5, characterized in that The method of performing tomato maturity detection processing according to the regional position of the target tomato to obtain the regional position of the mature target tomato includes: Extracting the target tomato's area position to obtain the green channel area position of the target tomato and determining the total number of pixels of the target tomato; Set the grayscale threshold parameter to determine the mature grayscale value range; Scanning the green channel area position of the target tomato in a scroll bar manner to obtain the grayscale value of the target tomato; Determining a target number of pixels of a target tomato according to the range of mature grayscale values; The position of the mature target tomato region is determined according to the ratio of the target pixel number of the target tomato to the total number of pixels of the target tomato.

8. The method according to claim 5, characterized in that The method of performing path planning on the depth value of the target tomato and the plane coordinates of the target tomato by using a fast expansion random number algorithm to realize automatic picking of tomatoes includes: Combining the depth value of the target tomato with the plane coordinates of the target tomato to generate the spatial coordinates of the target tomato; The spatial coordinates of the target tomato are converted into the spatial coordinates of the end effector operation in the robot arm coordinate system by using the calibration file; The automatic picking of tomatoes is achieved by planning the path of the spatial coordinates of the end effector operation in the robotic arm coordinate system through a rapid expansion of the random number algorithm.

9. The method according to claim 5, characterized in that The expression of the fast expansion random number algorithm is specifically as follows: In the above formula, x r Indicates the random generation of nodes when generating new nodes, x n represents the child node closest to the random point, s represents the step length, x ne Represents the new node that is finally generated.

10. An automatic tomato picking system based on machine vision, characterized in that: The system comprises: The first module is used to build a mobile robot model for automatic tomato picking; The second module is used to obtain color image data of tomatoes based on the automatic tomato picking mobile robot model and train the YOLOv8 target segmentation model to build an AI model file; The third module is used to obtain the position data of the end of the robot arm based on the tomato automatic picking mobile robot model and perform hand-eye calibration to generate a calibration file; The fourth module is used to obtain the image coordinates of the target tomato according to the AI ​​model file and the calibration file and perform path planning to achieve automatic picking of tomatoes.

Images