Fruit and vegetable picking robot system based on data enhancement and identification precision optimization
By adopting YOLOv5 vision technology and data enhancement method in the fruit and vegetable picking robot system, the problem of insufficient identification and positioning accuracy of fruit and vegetable picking robots in the existing technology in complex environments is solved, and more efficient and accurate picking operations are achieved.
Patent Information
- Application Number
- CN202510435165.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-08
- Publication Date
- 2025-06-20
- Estimated Expiration
- 2045-04-08
AI Technical Summary
Existing fruit and vegetable picking robots have poor accuracy in object recognition and positioning in complex environments, making it difficult to complete efficient decisions and operations in real time.
The fruit and vegetable picking robot system based on YOLOv5 vision technology is adopted, combined with advanced sensors, computer vision, machine learning and robot control technology, and high-precision recognition and positioning of fruits and vegetables through data augmentation and recognition accuracy optimization.
It improves the efficiency and accuracy of the fruit and vegetable picking robot during the picking process, improves the success rate of grabbing in complex environments, and enhances the positioning accuracy.
Smart Images

Figure CN120167229A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robots, and particularly to an intelligent fruit and vegetable picking robot system based on deep learning. More specifically, it refers to a fruit and vegetable picking robot system based on data augmentation and recognition accuracy optimization, which is mainly used for fruit and vegetable picking tasks in agricultural scenarios. Background Art
[0002] With the continuous growth of the global population, the challenges faced by agricultural production have become increasingly severe. To improve agricultural production efficiency and ensure food supply security, the application of automation technology in the agricultural field has become an important development trend. In particular, with the rapid development of visual recognition, machine learning, and artificial intelligence technologies, the application of robots in agriculture has become more extensive and in-depth.
[0003] Currently, fruit and vegetable picking robots have achieved preliminary application results in multiple fields. Such robots generally integrate image processing algorithms, sensors, robotic arms, and grasping tools to achieve precise recognition and picking of fruits and vegetables. The image processing algorithm is responsible for identifying the picking target, and the robotic arm locates and grasps based on the recognition information. However, even though these robots have improved the level of picking automation to a certain extent, there are still many problems related to grasping accuracy in actual applications.
[0004] Current fruit and vegetable picking robots mostly use simple image processing algorithms to identify objects. However, in a complex actual environment, the light changes constantly, the surrounding environment is cluttered, and there are a wide variety of fruits and vegetables. This makes the robot perform poorly in object recognition and positioning accuracy. The grasping process of fruits and vegetables requires extremely high precision. Existing fruit and vegetable picking robots are still in the initial stage of applying efficient computing solutions such as depth estimation, and it is difficult to complete efficient decision-making and operation in real time in a complex environment. In addition, due to cost issues, most large agricultural machines cannot use high-precision cameras for recognition work, resulting in insufficient fruit and vegetable positioning ability. When facing fruits and vegetables of different shapes, sizes, and under different weather conditions, it is difficult to accurately position them, which greatly affects the efficiency and accuracy of subsequent picking, handling, and other operations. Summary of the Invention
[0005] The purpose of the present invention is to provide a fruit and vegetable picking robot system based on data augmentation and recognition accuracy optimization, which solves the problems of poor object recognition and positioning accuracy of existing fruit and vegetable picking robots and difficulty in completing efficient decision-making and operation in real time in a complex environment. The present invention is a multi-functional intelligent fruit and vegetable cultivation system based on YOLOv5 vision technology and robotic arms. Through advanced sensor, computer vision, machine learning, and robot control technologies, it can identify, grasp fruits and vegetables, and improve the efficiency and accuracy of fruit and vegetable picking robots during the picking process.
[0006] The above object of the present invention is achieved by the following technical solutions:
[0007] A fruit and vegetable picking robot system based on data augmentation and recognition accuracy optimization, including
[0008] Visual recognition module: Equipped with a high-resolution camera, it is used to obtain fruit and vegetable images in real time, and uses the YOLOv5 object detection algorithm to identify and locate fruits and vegetables in real time. It can classify and detect fruits and vegetables with high precision and maintain stable performance under complex backgrounds;
[0009] Picking manipulator: Used to perform fruit and vegetable picking operations, including a multi-degree-of-freedom manipulator and a flexible end gripper, which can achieve precise grasping and reduce damage to fruits and vegetables;
[0010] Control module: Connects the visual recognition module and the picking manipulator, and is used to process recognition data, generate picking path planning and manipulator control instructions;
[0011] Mobile module: Used for the autonomous movement of the robot body in the farmland, supports obstacle avoidance function and path planning, and ensures that the robot can efficiently complete multi-point picking tasks.
[0012] The visual recognition module uses the YOLOv5 object detection algorithm to identify and locate fruits and vegetables in real time, which is realized by building an identification and grasping system model, including the following steps:
[0013] Step 1: Image acquisition: Collect target fruit and vegetable images through outdoor shooting as the data set;
[0014] Step 2: Preprocessing: Preprocess the collected images, including denoising and filtering, to improve the image quality;
[0015] Step 3: Expand the data set: Expand the originally collected data set to deal with different influencing factors. The specific method is: under simulated different lighting conditions, expand the data set by adjusting the brightness, contrast, saturation and hue color attributes of the image, and use the method of randomly adjusting parameters to perform color transformation on the image to generate images under different lighting and color conditions;
[0016] Simulate different shooting angles by changing the perspective of the image. In the multi-scale perspective transformation, adjust the parameters of the perspective transformation matrix H i Apply the random scaling factor to the elements of the perspective transformation matrix, and apply multiple perspective transformations at different scales to the same image. If there are N scales, the perspective transformation matrix at the i-th scale is expressed as:
[0017]
[0018] where s x,i and s y,iis the scaling factor at the i-th scale, representing the scaling of the image in the horizontal and vertical directions; b i , d i is the shear parameter, representing the slight inclination of the image along the diagonal direction; c i , f i is the translation parameter, representing the translation of the image in the horizontal and vertical directions; g i , h i is the perspective parameter, representing the change of objects at different distances in the image;
[0019] For each i of each scale, the transformed coordinates (x', y') of each pixel point (x, y) in the image are calculated by the following formula:
[0020]
[0021] The final transformation result needs to be normalized:
[0022]
[0023] Introduce more data diversity during the training process;
[0024] Step 4: Dataset annotation;
[0025] Step 5: Establishment of the recognition model;
[0026] Step 6: Model conversion and deployment;
[0027] Step 7: Target localization;
[0028] Step 8: Depth estimation for three-dimensional world targets;
[0029] Step 9: Optimize lens distortion;
[0030] Step 10: Introduce the weighted average algorithm to improve accuracy
[0031] Simulate the fruit and vegetable images in different weather environments;
[0032] Haze weather simulation, using Gaussian blur to simulate the blurred effect of the image caused by haze; Let the original image be (I original (x, y)), the Gaussian kernel be (G(x, y, σ)), where (σ) controls the degree of blur, and the convolution operation is:
[0033] I fog (x, y) = I original (x, y) * G(x, y, σ)
[0034] Among them, (G(x, y, σ)) is a two-dimensional Gaussian kernel:
[0035]
[0036] Overlay a layer of haze effect with transparency on the blurred image; Let the haze transparency be (α) (ranging from 0 to 1), then the composite image is:
[0037] I'foggy(x,y) = αIfog(x,y) + (1 - α)I original (x,y)
[0038] Among them, (α) controls the concentration of the haze. The closer the value is to 1, the stronger the haze effect;
[0039] Rainy weather simulation, using randomly generated transparent circular water droplets to simulate the effect of raindrops; Set a water droplet layer (R(x,y)), which consists of (n) randomly generated water droplets; The shape of each water droplet is simulated by a Gaussian function, and the size and position of the water droplets are random; For each water droplet, let its position be ((x i ,y i )), the standard deviation is (σ i ), and the generated water droplet shape is:
[0040]
[0041] Among them, (N(x - x i ,y - y i ,σ i )) is a two-dimensional Gaussian distribution function used to simulate the shape of the water droplet;
[0042] To simulate the weak light condition in rainy days, the overall brightness of the image is reduced; Let the brightness adjustment factor be (β) ((0 < β < 1)), then the adjusted rainy day image is:
[0043] I'rainy(x,y) = βIoriginal(x,y) - αR(x,y)
[0044] Among them, (α) is the intensity coefficient of the water droplet layer, which controls the visibility of the raindrops;
[0045] Snowy weather simulation, the distribution of snowflakes is achieved through randomly generated small white dots. Each snowflake is represented as a small Gaussian function, and the position and size are random; Let the snowflake layer be (S(x,y)), which consists of (n) snowflakes. The position of each snowflake is ((x i ,y i )), the standard deviation is (σ i ), then:
[0046]
[0047] Perform a slight contrast adjustment on the image to enhance the contrast between the snowflakes and the background; assuming the brightness adjustment factor is (β), the final snowy-day image is:
[0048] I'snowy(x,y) = β·(Ioriginal(x,y)+αS(x,y))
[0049] where (α) controls the intensity of the snowflakes and (β) controls the overall brightness;
[0050] For the simulation of sunny days, increase the vividness of the image by increasing the brightness, contrast, and color saturation. Assuming the brightness adjustment factor is (β), the adjusted image is:
[0051] I'bright(x,y) = Ioriginal(x,y)+β
[0052] By enhancing the contrast of the image, increase the sense of hierarchy of the image. The contrast factor is (α), usually (α>1), then the enhanced image is:
[0053] I'contrast(x,y) = α·(I'bright(x,y)-μ)+μ
[0054] where (μ) is the average brightness of the original image to ensure that the average brightness of the enhanced image remains unchanged;
[0055] Enhance the color of the image by increasing the color saturation; the process of saturation enhancement is achieved through the following formula:
[0056] i'sunny(x,y) = γ·I'contrast(x,y)
[0057] where (γ) is the saturation factor (usually (γ>1)).
[0058] For the optimization of lens distortion described in Step Nine, the lens error correction method adopts a segmented correction method:
[0059] (1) Region division:
[0060] Divide the lens field of view into a central region and an edge region;
[0061] Central region (r < R threshold ): The distortion is small, and the distortion model is used for correction;
[0062] Edge region (r ≥ R threshold ): The distortion is large, and manual sampling and interpolation correction are adopted;
[0063] (2) Manual sampling:
[0064] Select several sampling points in the 1 / 4 area of the lens field of view, and record the deviation between the uncorrected coordinates and the actual coordinates;
[0065] (3) Interpolation fitting: Use bilinear interpolation or polynomial fitting to generate a correction mapping table;
[0066] Δx = a0 + a1x + a2y + a3xy
[0067] Δy = b0 + b1x + b2y + b3xy
[0068] where a i , b i are fitting coefficients;
[0069] (4) Symmetry optimization: Utilize the symmetry of the rectangular field of view of the lens. Only the correction parameters of the 1 / 4 area need to be calibrated, and the mapping table is reused for the remaining areas through coordinate transformation (x -> -x, y -> -y);
[0070] Edge area:
[0071] x c = x + Δx(r, θ)
[0072] y c = y + Δy(r, θ)
[0073] where Δx(r, θ) and Δy(r, θ) are the correction amounts obtained through interpolation.
[0074] The beneficial effects of the present invention are as follows:
[0075] Compared with the prior art, the present invention has the following remarkable beneficial effects: By innovatively introducing a weather simulation algorithm, the grasping success rate of the robotic arm in complex environments such as rain, fog, and strong light is increased by more than 15%, solving the problem of poor environmental adaptability of traditional vision systems; at the same time, dynamic distortion compensation technology is adopted to improve the positioning accuracy of the monocular vision system from ±2mm to within ±0.5mm, effectively overcoming the positioning deviation caused by lens distortion. These two technological breakthroughs enable the monocular vision grasping system of the present invention to have the high precision and strong robustness of a multi-camera system while maintaining the low-cost advantage, providing a better technical solution for industrial automation. Brief Description of the Drawings
[0076] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and the schematic examples and descriptions of the present invention are used to explain the present invention, and do not constitute an improper limitation to the present invention.
[0077] Figure 1 It is a tomato scale scaling diagram adopted by the present invention;
[0078] Figure 2It is the simulated weather map of the present invention;
[0079] Figure 3 It is the depth information estimation model of the present invention;
[0080] Figure 4 It is the curve graph of the horizontal direction distortion situation of the lens of the present invention;
[0081] Figure 5 It is the curve graph of the vertical direction distortion situation of the lens of the present invention; Detailed implementation manners
[0082] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention. In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.
[0083] See Figures 1 to 5 As shown, the fruit and vegetable picking robot system based on data augmentation and recognition accuracy optimization of the present invention is an intelligent fruit and vegetable picking robot system capable of deep learning, mainly used for fruit and vegetable picking tasks in agricultural scenarios. The present invention has improved the performance of the previous fruit and vegetable manipulator, and can still accurately identify the object position when the monocular camera has low accuracy or the environmental information changes (such as variable lighting and foliage occlusion). It mainly adopts advanced technologies and algorithms, and then through specific data processing, optimizes and integrates the position information to lock the target with high precision and accurately grab. This invention can significantly improve the picking efficiency and ensure the quality of fruits and vegetables at a slightly increased equipment cost, helping the intelligent development of modern agriculture. The present invention is composed of a visual recognition module, a picking manipulator, a control module and a mobile module, and the core technology is to use the YOLOv5 target detection algorithm to perform real-time recognition and positioning of fruits and vegetables. The visual module collects images through a monocular camera and accurately detects the types and positions of fruits and vegetables using the YOLOv5 algorithm; the control module coordinates the manipulator to perform picking actions according to the recognition results, and at the same time plans the moving path of the robot to adapt to the complex farmland environment; the picking manipulator adopts a multi-degree-of-freedom design and is equipped with a flexible gripper, which can complete efficient picking operations while protecting the fruits and vegetables. In addition, the system also supports the dynamic update of the model, and continuously optimizes the recognition performance by adding new fruit and vegetable data. While improving the picking efficiency and reducing the labor cost, the system can meet the needs of different crops and environments, and has broad application prospects.
[0084] The flexible end gripper equipped with the picking robotic arm is made of soft materials and can adjust the grasping force according to the shape and size of fruits and vegetables to avoid damaging the surface of fruits and vegetables. The control module integrates a path planning algorithm, which can generate the optimal picking path according to the terrain of the orchard and the distribution of fruits and vegetables, improving work efficiency. The mobile module adopts a mecanum wheel chassis design and has omnidirectional movement ability, enabling lateral movement, diagonal movement and rotation in place, so as to move flexibly in the orchard or farmland, adapt to complex picking environments and improve operation accuracy. The system supports users to add new fruit and vegetable data sets through annotation tools to optimize the YOLOv5 model and make it adapt to more fruit and vegetable varieties and complex background environments. The entire system can be modularly designed, facilitating users to configure and upgrade according to actual needs, such as replacing the vision camera or the type of robotic arm to adapt to different operation scenarios. The robot body is equipped with a backup battery and a charging module, supporting long-term operation to ensure that the robot can continuously run during the picking process.
[0085] The fruit and vegetable picking robot system based on data augmentation and recognition accuracy optimization of the present invention is used for automatic picking of fruits and vegetables, and includes a mobile module and a picking robotic arm; wherein the picking robotic arm is used for grasping fruits and vegetables, the mobile module is used for controlling the picking robot to travel between fields, and the picking robotic arm is used for separating fruits and vegetables. The mobile module is the support and movement structure of the fruit and vegetable picking robot, and the bottom of the picking robotic arm is fixedly installed on the mobile module; the end of the picking robotic arm is connected to the picking device (flexible end gripper). The picking robotic arm includes a four-axis robotic arm, an actuator and a monocular camera installed at the end of the four-axis robotic arm. The mobile module includes a driving and traveling module, which uses a mecanum wheel chassis to drive the fruit and vegetable picking robot to move. The picking module includes a gripper and a monocular camera, and the gripper can open up to 180 degrees.
[0086] The model construction of the recognition and grasping system of the fruit and vegetable picking robot of the present invention includes the following steps:
[0087] Step 1, Image acquisition: Collect images of target fruits and vegetables through outdoor shooting as the data set;
[0088] Step 2, Preprocessing: Preprocess the collected images, including denoising, filtering, etc., to improve the image quality;
[0089] Step 3, Expand the data set: Expand the originally collected data set in a targeted manner to cope with different influencing factors such as lighting, perspective and weather. The specific method is:
[0090] Under different simulated lighting conditions, the dataset is augmented by adjusting color attributes such as the brightness, contrast, saturation, and hue of the images. By randomly adjusting the parameters, color transformations are applied to the images within a certain range (e.g., brightness from -a to +a, contrast from b to 3b times, saturation from c to 3c times, hue from -g to +g degrees), generating images under different lighting and color conditions.
[0091] Furthermore, different shooting angles are simulated by changing the perspective of the images. Traditional random perspective transformations usually use fixed scaling factors and transformation parameters, which may result in overly similar generated images and fail to fully simulate complex scenarios in the real world. To increase diversity, multi-scale perspective transformation is introduced, such that the generated images vary not only in perspective but also in scale. In multi-scale perspective transformation, the parameters of the perspective transformation matrix H i are adjusted, and random scaling factors are applied to the elements of the perspective transformation matrix. Multiple perspective transformations with different scales are applied to the same image. If there are N scales, the perspective transformation matrix at the i-th scale can be expressed as
[0092]
[0093] where s x,i and s y,i are the scaling factors at the i-th scale, representing the scaling of the image in the horizontal and vertical directions. b i , d i are the shear parameters, representing a slight tilt of the image along the diagonal direction. c i , f i are the translation parameters, representing the translation of the image in the horizontal and vertical directions. g i , h i are the perspective parameters, representing the changes of objects at different distances in the image.
[0094] For each scale i, the transformed coordinates (x’, y’) of each pixel point (x, y) in the image can be calculated by the following formula:
[0095]
[0096] The final transformation result needs to be normalized:
[0097]
[0098] More data diversity is introduced during the training process.
[0099] Furthermore, the fruit and vegetable images are simulated in different weather conditions. First, the haze weather is simulated. The haze weather is usually achieved by blurring the effect and reducing the image clarity. The haze effect is simulated through the following two parts: First, Gaussian blur is used to simulate the blurring effect caused by haze in the image. Gaussian blur is a low-pass filter that can blur the details in the image. Let the original image be (I original (x,y)), the Gaussian kernel be (G(x,y,σ)), where (σ) controls the degree of blurring, and the convolution operation is:
[0100] I fog (x,y) = I original (x,y) * G(x,y,σ)]
[0101] where (G(x,y,σ)) is a two-dimensional Gaussian kernel:
[0102]
[0103] Furthermore, in order to simulate the concentration of haze, a layer of haze effect with transparency is superimposed on the blurred image. Let the haze transparency be (α) (ranging from 0 to 1), then the composite image is:
[0104] I'foggy(x,y) = αIfog(x,y) + (1 - α)I original (x,y)
[0105] where (α) controls the concentration of haze, and the closer the value is to 1, the stronger the haze effect.
[0106] Furthermore, the rainy weather is simulated. The effects of simulating rain include simulating the traces of water droplets and the reduction of overall brightness. It is modeled in the following way: Randomly generated transparent circular water droplets are used to simulate the effect of raindrops. A water droplet layer (R(x,y)) is set, which consists of randomly generated (n) water droplets. The shape of each water droplet is simulated by a Gaussian function, and the size and position of the water droplets are random. For each water droplet, let its position be ((x i ,y i )) and the standard deviation be (σ i ), and the generated water droplet shape is:
[0107]
[0108] where (N(x - x i ,y - y i ,σ i )) is a two-dimensional Gaussian distribution function used to simulate the shape of the water droplet.
[0109] Further, to simulate the dim light conditions on a rainy day, the overall brightness of the image is reduced. The brightness adjustment is achieved through the normalization process of the image. Let the brightness adjustment factor be (β) ((0 < β < 1)), then the adjusted rainy-day image is:
[0110] I'rainy(x,y) = βIoriginal(x,y) - αR(x,y)
[0111] where (α) is the intensity coefficient of the water droplet layer, which controls the visibility of the raindrops.
[0112] Further, for the simulation of snowy weather, the snow simulation is similar to that of rainy days, but the snowflakes are usually smaller and more evenly distributed. The distribution of snowflakes is achieved through randomly generated small white dots. Each snowflake is represented as a small Gaussian function, with random positions and sizes. Let the snowflake layer be (S(x,y)), which consists of (n) snowflakes, and the position of each snowflake is ((x i ,y i ), with a standard deviation of (σ i ), then:
[0113]
[0114] Similar to rainy days, snowy days also reduce the overall brightness. In addition to brightness adjustment, slight contrast adjustment can also be performed on the image to enhance the contrast between the snowflakes and the background. Assuming the brightness adjustment factor is (β), the final snowy-day image is:
[0115] I'snowy(x,y) = β·(Ioriginal(x,y) + αS(x,y))
[0116] where (α) controls the intensity of the snowflakes and (β) controls the overall brightness.
[0117] Further, the simulation of sunny days mainly increases the vividness of the image by enhancing brightness, contrast, and color saturation. To increase the brightness of the image and make it look brighter. Let the brightness adjustment factor be (α), then the adjusted image is:
[0118] I'bright(x,y) = Ioriginal(x,y) + β
[0119] By enhancing the contrast of the image, the sense of hierarchy of the image is increased. Using a simple linear contrast enhancement formula, assuming the contrast factor is (α) (usually α > 1), then the enhanced image is:
[0120] I'contrast(x,y) = α·(I'bright(x,y) - μ) + μ
[0121] Among them, (μ) is the average brightness of the original image, ensuring that the average brightness of the enhanced image remains unchanged.
[0122] To simulate the vivid colors on a sunny day, the colors of the image are enhanced by increasing the color saturation. The process of saturation enhancement is achieved through the following formula:
[0123] I'sunny(x,y) = γ·I'contrast(x,y)
[0124] Among them, (γ) is the saturation factor (usually (γ>1)).
[0125] Step 4: Dataset annotation: Use labelimg to annotate the dataset. Accurately mark the position and category of each fruit and vegetable to ensure the high quality of the dataset for subsequent model training.
[0126] Step 5: Establishment of the recognition model: Use the YOLOV5 model for training. The main hyperparameters during model training are set as follows: Epochs = a, Batch size = b, the initial value of the learning rate η is set to η0 = c and adjusted during the training process; during the training process, the model parameters θ are updated through the backpropagation algorithm and the optimizer to minimize the loss function L(θ) that includes classification error, position error, and confidence error, where the classification error is optimized using the cross-entropy loss function.
[0127] Step 6: Model conversion and deployment: Deploy the trained model to the composite fruit and vegetable picking robot, and convert the Pytorch model to the tensorRT model for deployment.
[0128] Step 7: Target localization: Determine the position of the fruit in the image according to the classification result; when facing unclear localization, use the depth estimation model. Specifically, this model mainly includes the following steps;
[0129] Deploy the trained model, and the monocular camera detects the target object to obtain the bounding box output by YOLOv5. The YOLOv5 model will output the bounding box information of the target object, usually returned in the form of [x1,y1,x2,y2].
[0130] Furthermore, since the coordinates output by YOLOv5 are normalized, they need to be converted to pixel coordinates through the internal parameter matrix in the camera: (x p , y p )
[0131] x p = x1×z1
[0132] y p = y1×z2
[0133] The formula form of the camera intrinsic matrix is as follows:
[0134]
[0135] Among them, f x represents the focal length of the camera in the direction, and f y represents the focal length of the camera in the direction. c x and c y represent the coordinates of the principal point in the image coordinate system and the coordinate respectively. This matrix contains the focal lengths f x , f y and the principal point c x , c y , and is used to convert image coordinates to normalized coordinates in the camera coordinate system:
[0136] x n =(x p -c x ) / f x
[0137] y n =(y p -c y ) / f y
[0138] Step Eight: Depth Estimation of 3D World Objects
[0139] To convert image coordinates to 3D world coordinates, depth information (i.e., the actual distance between the camera and the object in space) needs to be obtained. Methods for estimating depth information through a depth camera, LiDAR, or through object size and camera model are given. Assuming that the camera intrinsics (such as focal length, field of view angle) and the size of the object's border in the image are known, a geometric model as shown in Figure 2 is established (where θ is the camera field of view angle, h is the height of the acquired image, and d is the depth), and the specific process for estimating the actual distance between the object and the camera is as follows:
[0140] 1. Assume that the actual width of the ripe fruit is 2r, the actual visible width of the ripe fruit in the vertical plane where the camera is located is 2h, the depth is d, the field of view angle of the camera is (the field of view angle is the angle formed by the two edges of the maximum range through which the object image can pass through the lens with the vertex of the optical instrument's lens), the maximum side width recognized by the camera is e, the length of the border of the ripe fruit recognized by YOLOv5 is y2, and θ0 is measured.
[0141] 2. According to the principle of similar triangles, the ratio of the actual side width of the ripe fruit to the actual visible width of the camera is equal to the ratio of the side width of the ripe fruit recognized by YOLOV5 to the maximum side width recognized by the camera, that is
[0142]
[0143] In the geometric relationship involving the field of view angle, there is
[0144]
[0145] It can be obtained that
[0146] h = d×tanθ (3)
[0147] 3. Substitute Equation (2) into Equation (1) for derivation: From Equation (1), substituting h = d×tanθ, we can get: Further simplification gives
[0148] Step Nine: Optimize lens distortion: Lens distortion is mainly divided into radial distortion and tangential distortion, and the specific parameters are as follows: Radial distortion: Caused by the curvature of the lens, manifested as stretching or compression at the image edge. The commonly used parameters are k1, k2, k3 (radial distortion coefficients).
[0149] Tangential distortion: Caused by the non - parallelism between the lens and the imaging plane, manifested as tilting or twisting of the image. The commonly used parameters are p1, p2 (tangential distortion coefficients).
[0150] Standard distortion model formula:
[0151] x c = x(1 + k1r 2 + k2r 4 + k3r 6 ) + 2p1xy + p2(r 2 + 2x 2 )
[0152] yx = y(1 + k1r 2 + k2r 4 + k3r 6 ) + 2p2xy + p1(r 2 + 2y 2 )
[0153] where x c , y c are the edge coordinates, is the distance from the pixel point to the center of the lens.
[0154] Lens error correction method: Since the distortion at the lens edge is relatively large, a segmented correction method is adopted:
[0155] 1. Region division:
[0156] Divide the lens field of view into a central region and an edge region.
[0157] Central region (r < R threshold): The distortion is small and is corrected using the above standard distortion model.
[0158] Edge region (r ≥ R threshold ): The distortion is large and is corrected by manual sampling and interpolation.
[0159] 2. Manual sampling:
[0160] Select several sampling points in the 1 / 4 region of the lens field of view (such as the upper right quadrant), and record the deviation between the uncorrected coordinates and the actual coordinates.
[0161] 3. Interpolation fitting:
[0162] Use bilinear interpolation or polynomial fitting to generate a correction mapping table.
[0163] Δx = a0 + a1x + a2y + a3xy
[0164] Δy = b0 + b1x + b2y + b3xy
[0165] where a i , b i are fitting coefficients.
[0166] 4. Symmetry optimization:
[0167] Utilize the symmetry of the lens rectangular field of view. Only the correction parameters for the 1 / 4 region need to be calibrated, and the mapping table is reused for the remaining regions through coordinate transformation (x -> -x, y -> -y).
[0168] Edge region:
[0169] x c = x + Δx(r, θ)
[0170] y c = y + Δy(r, θ)
[0171] where Δx(r, θ) and Δy(r, θ) are the correction amounts obtained through interpolation.
[0172] Step Ten: Introduce a weighted average algorithm to improve accuracy
[0173] Since the dynamic detection results of YOLOV5 may have a certain degree of fluctuation (for example, the detection result coordinates of the same fruit and vegetable may be slightly different in different time frames, and there are multiple detections in the same image), the error of the grasping operation based on a single recognition result is relatively large. The weighted average method can help reduce these small fluctuations when measuring the object coordinates and obtain more stable and accurate results. The specific process is as follows:
[0174] Perform real-time coordinate monitoring and error calculation, and a coordinate processing unit is defined: Calculate the change amount of the coordinates obtained each time with the previous coordinates in real time (Δx = xi x i1 , Δy = y i y i1 ), and the deviation between the current coordinate and the average value of the coordinates measured in the previous 5 times (μ x , μ y ). The calculation results are stored in the cache inside the FPGA. x = x i μ x , Deviation y = y i μ y ).
[0175] Furthermore, adjust the setting and judgment of the error threshold. For fruits and vegetables with different regular shapes, set the threshold of coordinate change amount Δx threshold , Δy threshold , and the deviation threshold By real-time monitoring and judgment, read the coordinate change amount and deviation data calculated by the coordinate processing unit, and compare them with the corresponding thresholds in the system parameter configuration file. Once it is found that the coordinate change amount or deviation obtained in a certain time exceeds the threshold, immediately send a trigger signal to the dynamic weight calibration and weighted average calculation module through an interrupt signal, and the interrupt response time is less than 100 microseconds to ensure timely weight calibration.
[0176] Furthermore, first perform weight adjustment calculation. When receiving the trigger signal, read the current coordinate data and the corresponding confidence level c i output by the YOLOv5 model. Perform weight adjustment according to the preset dynamic weight adjustment formula average , where c
[0177]
[0178] is the average value of the confidence levels measured in the previous 5 times, which is calculated by maintaining a confidence level queue with a length of 5 in the memory. Then perform weighted average recalculation and transmission. Use the adjusted weights and perform weighted average according to the weighted average formula:
[0179] Recalculate the weighted average values (X, Y) of the fruits and vegetables in the x and y directions.
[0179] After completing real-time coordinate monitoring and error calculation, it is also necessary to retain historical data for weighted average optimization of historical grasping error feedback. It is connected to the grasping actuator, coordinate calculation module, and environmental sensor group through a high-speed data bus. After each grasping operation is completed, it automatically obtains and records the grasping time (accurate to milliseconds), fruit and vegetable type identification (determined by comparing the image recognition result with the pre-stored fruit and vegetable type database), light intensity value collected by the environmental light sensor, temperature value collected by the temperature sensor, humidity value collected by the humidity sensor, weighted average coordinates (X, Y) output by the coordinate calculation module, and the actual grasping coordinates (X actual , Y actual ), and calculates the error values in the x and y directions between the two (Error x = XX actual , Error y = YY actual ).
[0180] Furthermore, an error analysis program is set up and triggered by the system's scheduled task every time n grasping tasks are completed. This program extracts error data and related grasping information from the historical database through the database connection interface and analyzes them using the KMeans clustering algorithm and decision tree algorithm. Feature vectors are constructed with fruit and vegetable types, environmental light intensity intervals, temperature intervals, humidity intervals, and the motion state of fruits and vegetables during grasping as feature vectors. The KMeans clustering algorithm is used to preliminarily cluster the error data, and error data with similar features are grouped into one category;
[0181] Furthermore, using the decision tree algorithm, a decision tree model is constructed according to different feature conditions to determine whether the error is mainly caused by unreasonable weight distribution in the coordinate calculation method, environmental interference, the characteristics of fruits and vegetables themselves, or other factors. Analyze the frequency of each type of error and its impact on the grasping accuracy.
[0182] In subsequent grasping tasks, by real-time reading the relevant information (such as fruit and vegetable types, environmental parameters, etc.) of the current grasping task, the weights are adjusted in real time according to the rule library. The adjusted weights are applied to the weighted average calculation in coordinate calculation, and the weighted average is calculated according to the weighted average formula
[0183]
[0184] The weighted average values of the fruits and vegetables in the x and y directions (X, Y) are recalculated and the new weighted average values are transmitted to the grasping actuator in real time to ensure that the grasping operation is carried out based on more accurate coordinates. At the same time, continuously optimize the weight adjustment strategy.
[0185] Example 1:
[0186] The construction of the recognition and grasping system model in this example includes the following steps:
[0187] Step 1: Collect 500 images of mature tomato fruits as the initial dataset for the experiment through a dataset website or by self - shooting.
[0188] Step 2: First, perform data pre - processing for denoising. Use median filtering, mean filtering and other algorithms in Matlab to remove random noise from the images, and perform denoising operations on each collected image. In this embodiment, a monocular camera with a size of 640×480 (mm) is used. Set a suitable filtering window of size 5×5, traverse each pixel point in the image, and effectively remove various noises such as salt - and - pepper noise and Gaussian noise introduced by the shooting environment, equipment, etc., to improve the clarity of the image.
[0189] Step 3: Perform parameter setting and range determination: First, determine the parameter ranges for adjusting the color attributes of the images. The brightness parameter a is set to 50 (indicating that the brightness can be randomly adjusted within the range of - 50 to + 50), the contrast parameter b is set to 1 (i.e., the contrast can randomly vary between 1 - 3 times), the saturation parameter c is set to 1 (meaning that the saturation can randomly change between 1 - 3 times), and the hue parameter g is set to 30 (the hue can be randomly adjusted within the range of - 30 to + 30 degrees). These parameter ranges can basically cover the possible illumination and color difference situations.
[0190] Furthermore, for the original images, use the image - processing library in Python to randomly adjust the brightness, contrast, saturation and hue according to the set parameter ranges to simulate images under different illumination conditions. Apply perspective transformations of multiple different scales to each image in the adjusted dataset. Considering data diversity and computing resources, initially start from a relatively small N, set it to 5, and quickly verify the effectiveness without consuming too many resources.
[0191] For each scale i, generate a random perspective transformation matrix:
[0192]
[0193] where s x,i and s y,i are the scaling factors at the i - th scale, representing the scaling of the image in the horizontal and vertical directions.
[0194] s x,i = rand(μ(0.5, 1.5))s y,i = rand(μ(0.5, 1.5))
[0195] b i ,d i are the shear parameters at the i - th scale, representing a slight tilt of the image along the diagonal direction.
[0196] b i = rand(μ(-0.1, 0.1))d i = rand(μ(-0.1, 0.1))
[0197] c i , f i are translation parameters, representing the translation of the image in the horizontal and vertical directions.
[0198] c i = rand(μ(-100, 100))s y,i = rand(μ(-100, 100))
[0199] g i , h i are perspective parameters, representing the changes of objects at different distances in the image.
[0200] g i = rand(μ(-0.001, 0.001))h i = rand(μ(-0.001, 0.001))
[0201] For each pixel point (x, y) in the image, its transformed coordinates (x', y') are calculated through the following formula:
[0202]
[0203] The final transformation result needs to be normalized:
[0204]
[0205] The transformed coordinates (x', y') may not be integers. Interpolation is performed on them, and bilinear interpolation is used to calculate the value of each pixel point in the transformed image. The transformed image I k,i (x', y') = BilinearInterpolation(I k , x, y), and multi-scale perspective transformation is performed according to the gradually increasing value of N until N = 10. The transformed image I k,i is saved as a new training sample.
[0206] Furthermore, the influences of different weather conditions are simulated, including haze, rainy days, and snowy days.
[0207] For smoggy weather, the Gaussian blur function G(x, y) is used to simulate the blur effect caused by smog, where the standard deviation (σ) controls the degree of blur; subsequently, a smog layer with transparency α (α is set to 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 in sequence) is superimposed on the blurred image, and the composite image formula is
[0208] I fog =(1 - α)I + αI haze
[0209] where I haze represents the smog effect layer, and α controls the smog concentration.
[0210] For the rainy day effect, a water droplet layer (R(x, y)) with a random distribution is generated, and the overall brightness is reduced. The brightness adjustment formula
[0211] I'rainy(x, y) = βIoriginal(x, y) - αR(x, y)
[0212] where β (β is taken as 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 in sequence) is the brightness adjustment factor, and at the same time, the water droplet layer intensity coefficient α is added.
[0213] The snow day simulation is similar to the rainy day, but the snowflakes are smaller and evenly distributed. The final snow day image is
[0214] I′snowy(x, y) = β·(Ioriginal(x, y) + αS(x, y))
[0215] where β (the value is adjusted in the same way as the rainy day simulation) and α control the brightness and snowflake intensity respectively.
[0216] Step 4: Annotate the dataset, and use labelimg to annotate the dataset. Accurately mark the position and category of each fruit and vegetable.
[0217] Step 5: Use the YOLOV5 model for training. The main hyperparameters during model training are set as follows: Epochs = 1000, Batch size = 16, the initial value of the learning rate η is set to η0 = c and adjusted during the training process; during the training process, the model parameters θ are updated through the backpropagation algorithm and the optimizer to minimize the loss function L(θ) that includes classification error, position error, and confidence error, where the classification error is optimized using the cross-entropy loss function.
[0218] Step 6: Model conversion and deployment: Deploy the trained model to the composite fruit and vegetable picking robot, and convert the Pytorch model to the tensorRT model for deployment.
[0219] Step 7: Deploy the trained model. The monocular camera detects the target object to obtain the bounding box [x1, y1, x2, y2] output by YOLOv5. Further, it is converted into pixel coordinates through the internal parameter matrix in the camera: (x p , y p )
[0220] x p = x1 × z1
[0221] y p = y1 × z2
[0222] The formula form of the camera internal parameter matrix is:
[0223]
[0224] where f x represents the focal length of the camera in the direction, f y represents the focal length of the camera in the direction, c x and c y represent the coordinates of the principal point in the image coordinate system and the coordinates respectively. This matrix contains the focal lengths f x , f y and the principal point c x , c y , and is used to convert the image coordinates into the normalized coordinates in the camera coordinate system:
[0225] x n = (x p - c x ) / f x
[0226] y n = (y p - c y ) / f y
[0227] Step 8: Depth estimation for three-dimensional world targets
[0228] To convert the image coordinates into three-dimensional world coordinates, it is necessary to obtain the depth information (i.e., the actual distance between the camera and the object in space). Methods for estimating the depth information through a depth camera, LiDAR, or through the object size and camera model are given. Assuming that the camera internal parameters (such as focal length, field of view angle) and the size of the object's bounding box in the image are known, a geometric model as shown in Figure 2 is established (where θ is the camera field of view angle, h is the height of the acquired image, and d is the depth), and the specific process for estimating the actual distance between the object and the camera is as follows:
[0229] 1. Let the actual width of the ripe fruit be 2r, the actual visible width of the ripe fruit in the vertical plane where the camera is located be 2h, the depth be d, the field of view angle of the camera be (the field of view angle is the angle formed by the two edges of the maximum range through which the image of the measured target can pass through the lens with the vertex of the lens of the optical instrument), the maximum side width recognized by the camera be e, the length of the border of the ripe fruit recognized by YOLOv5 be y2, and the experimentally measured θ0 be obtained.
[0230] 2. According to the principle of similar triangles, the proportional relationship between the actual side width of the ripe fruit and the actual visible width of the camera is equal to the proportional relationship between the side width of the ripe fruit recognized by YOLOV5 and the maximum side width recognized by the camera, that is
[0231]
[0232] In the geometric relationship involving the field of view angle, there is
[0233]
[0234] It can be obtained that
[0235] h = d×tanθ (3)
[0236] 3. Substitute (2) into (1) for derivation: From equation (1), substituting h = d×tanθ, we can get: After further simplification, we can get
[0237] Step Nine: Lens parameter setting. Set the radial distortion coefficients k1 = 0.12, k2 = -0.03, k3 = 0.01, p1 = 0.001, p2 = 0.001 according to the standard checkerboard calibration method
[0238] The image resolution is 1920*1080, take W = 1920 pixels, H = 1080 pixels
[0239] Take 70% of the diagonal length
[0240] R threshold = 0.7*d = 1500 pixels
[0241] Furthermore, perform manual sampling and interpolation fitting. The sampling area is selected in the upper right quadrant (1 / 4 area) of the lens field of view, and 20 evenly distributed sampling points are selected to ensure coverage of the typical distortion positions in the edge area.
[0242] Sampling point data recording: Record the uncorrected coordinates (x raw , y raw ) and the actual coordinates (x real , y real)Calculate the deviation Δx = x real - x raw Δy = y real - y raw . Generate a mapping table using bilinear interpolation:
[0243] Δx = a0 + a1x + a2y + a3xy
[0244] Δy = b0 + b1x + b2y + b3xy
[0245] The range of fitting coefficients is:
[0246] a0, b0 ∈ [-5.0, 5.0] a1, b1 ∈ [-0.1, 0.1] a2, b2 ∈ [-0.1, 0.1] a3, b3 ∈ [-0.01, 0.01]
[0247] Furthermore, for symmetry optimization, symmetrically map the correction mapping table in the upper - right quadrant to the other three quadrants. Coordinate transformation in the lower - left quadrant: x' = -x, y' = -y, coordinate change in the upper - left quadrant: x' = -x, y' = y, coordinate transformation in the lower - right quadrant: x' = x, y' = -y
[0248] Furthermore, first perform a region judgment on each pixel point (x, y) and calculate its distance to the center
[0249]
[0250] When r < R threshold , use the standard distortion model for correction:
[0251] x c = x(1 + k1r 2 + k2r 4 + k3r 6 ) + 2p1xy + p2(r 2 + 2x 2 )
[0252] y c = y(1 + k1r 2 + k2r 4 + k3r 6 ) + 2p2xy + p1(r 2 + 2y 2 )
[0253] If r ≥ R threshold , read Δx, Δy from the mapping table and correct:
[0254] x c = x + Δx
[0255] y c = y + Δy
[0256] where Δx(r, θ) and Δy(r, θ) are correction amounts obtained by interpolation.
[0257] Step Ten: After the above processing is completed, collect the relevant data (x1, x2, y1, y2) of the observed target object N times. Among them, the relevant coordinate values of the i-th measurement are z i (z i = x 1i , x 2i , y 1i , y 2i ). In this embodiment, N = 10 is taken. After the vision system of the composite fruit and vegetable picking robot is started, for the same target fruit and vegetable, 10 detections are performed through the YOLOV5 model to obtain the coordinate information of the tomato in the camera coordinate system obtained by each detection. Each coordinate contains two components, x and y (temporarily not considering the z-axis coordinate corresponding to the depth information, which will be processed separately later). For each coordinate component (x1, x2, y1, y2) in these 10 measurements, calculate its mean μ and standard deviation σ respectively. The mean can be calculated using the conventional arithmetic mean calculation formula. For the coordinate component, μ x = (x1 + x2 +... + x 10 ) / 10; the standard deviation σ is calculated according to the standard deviation calculation formula in statistics.
[0258] Furthermore, traverse each coordinate point obtained from these 10 measurements. For each coordinate point, when its distance from the corresponding mean in the x coordinate or y coordinate exceeds 3 times the standard deviation (i.e., |x i - μ x | > 3σ x or |y i - μ y | > 3σ γ ), then this coordinate point is considered an outlier and is discarded; otherwise, retain this coordinate point as a valid data point for subsequent weighted average calculation. In this way, abnormal measurement values caused by factors such as detection errors and environmental interference are removed, ensuring the data quality of the weighted average calculation.
[0259] Furthermore, for the remaining valid coordinate data after outlier removal, assign weights according to the confidence of each recognition result. In the detection results output by the YOLOV5 model, each detection box has a corresponding confidence value, indicating the reliability of the detection box containing the target tomato. Denote the confidence of the i-th recognition as c i . First, normalize the confidence of all valid data points to ensure that the sum of the weights is 1. The specific calculation method is w i = c i / (c1 + c2 +... + c10 ) In this way, the higher the confidence level of the measurement result, the greater the proportion it occupies in the weighted average calculation, making more reasonable use of the reliability differences of different detection results.
[0260] Further, perform weighted average calculation. According to the following weighted average calculation formula, calculate the weighted averages X and Y of the target fruit and vegetable in the x and y coordinate directions respectively
[0261] Weighted average
[0262] Weighted average
[0263] where x i and y i are the x and y coordinate values of the i-th valid data point, w i is the corresponding weight, and n is the number of valid data points (the number of remaining data points after outlier removal). Through the above calculation, a more stable and accurate coordinate position of the fruit and vegetable after weighted average processing in the camera coordinate system is obtained.
[0264] Real-time calculate the change amount (Δx = x i x i1 , Δy = y i y i1 ) between the coordinates obtained each time and the previous coordinates, as well as the deviation (Deviation x , Deviation y ) between the current coordinate and the average value (μ x , μ y ) of the previous 5 measurement coordinates (Deviation x = x i μ x , Deviation y = y i μ y ). The calculation results are stored in the internal high-speed cache of the FPGA.
[0265] Further, adjust the setting and judgment of the error threshold. For small and regularly shaped fruits and vegetables [type 1], set the coordinate change amount threshold Δx threshold = 2 pixels, Δy threshold = 2 pixels, and the deviation threshold pixels, pixels; for large and irregularly shaped fruits and vegetables [type 2], set the coordinate change amount threshold to Δx threshold = 5 pixels, Δy threshold = 5 pixels, and set the deviation threshold to pixels, pixels.
[0266] Through real-time monitoring and judgment, the error threshold setting and judgment module reads the coordinate change and deviation data calculated by the coordinate processing unit in real time, and compares it with the corresponding threshold in the system parameter configuration file. Once it is found that the coordinate change or deviation obtained at a certain time exceeds the threshold, a trigger signal is immediately sent to the dynamic weight calibration and weighted average calculation module through an interrupt signal. The interrupt response time is less than 100 microseconds to ensure timely weight calibration.
[0267] Dynamic weight calibration and weighted average calculation: First, perform weight adjustment calculation. When a trigger signal is received, read the current coordinate data and the corresponding confidence c output by the YOLOv5 model. i . According to the preset dynamic weight adjustment formula Weight adjustment is performed, where c average is the average of the confidences of the previous five measurements, calculated by maintaining a confidence queue of length 5 in memory. Then the weighted average is recalculated and transmitted, using the adjusted weights and weighted average according to the weighted average formula:
[0268]
[0269] Recalculate the weighted average of the fruits and vegetables in the x and y directions (X, Y).
[0270] After completing real-time coordinate monitoring and error calculation, it is also necessary to retain historical data and perform weighted average optimization of historical grasping error feedback. It is connected to the grasping actuator, coordinate calculation module and environmental sensor group through a high-speed data bus. Within 100 milliseconds after each grasping operation is completed, it automatically obtains and records the grasping time (accurate to milliseconds), fruit and vegetable type identification (determined by comparing the image recognition result with the pre-stored fruit and vegetable type database), the light intensity value collected by the ambient light sensor, the temperature value collected by the temperature sensor, the humidity value collected by the humidity sensor, the weighted average coordinates (X, Y) output by the coordinate calculation module, and the actual grasping coordinates (X actual , Y actual ) and calculate the error values (Error x =XX actual , Error y =YY actual ).
[0271] Furthermore, an error analysis program is set up, which is triggered by the system timer task after completing 100 grab tasks. The program extracts error data and related grab information from the historical database through the database connection interface, and uses the KMeans clustering algorithm and decision tree algorithm for analysis.
[0272] Construct feature vectors with fruit and vegetable types (represented by unique codes, 001 for tomatoes, 002 for bananas), environmental light intensity intervals (divided into multiple intervals such as 0 - 200 lux, 201 - 500 lux, 501 - 1000 lux, etc.), temperature intervals (40°C - 0°C, 0°C - 20°C, 20°C - 40°C, etc.), humidity intervals (0% - 30% RH, 31% - 60% RH, 61% - 100% RH), and the motion state of the fruit and vegetable during grasping (judged by the displacement sensor data before the robotic arm grasps, 0 for stationary, 1 for low-speed movement, 2 for high-speed movement) as feature vectors.
[0273] Use the KMeans clustering algorithm to preliminarily cluster the error data, grouping error data with similar features into one category; then use the decision tree algorithm to construct a decision tree model according to different feature conditions to determine whether the error is mainly caused by unreasonable weight distribution in the coordinate calculation method (by analyzing the historical weight distribution data and error correlation to determine that the weights of certain coordinate points are too high or too low), environmental interference (when the light intensity is lower than a specific value, the image noise increases, resulting in coordinate recognition deviation, determined by comparing the error frequencies and magnitudes at different light intensities), the characteristics of the fruit and vegetable itself (the smooth and reflective surface leads to inaccurate contour detection, judged according to the fruit and vegetable type and error characteristics), or other factors (the camera lens is dirty, determined by regularly cleaning the camera and comparing the error data before and after cleaning). Analyze the frequency of each type of error occurrence and its impact on the grasping accuracy.
[0274] In subsequent grasping tasks, by real-time reading the relevant information of the current grasping task (fruit and vegetable type, environmental parameters, etc.), the weights are adjusted in real time according to the rule base. The adjusted weights are applied to the weighted average calculation in the coordinate calculation, and the weighted average is calculated according to the weighted average formula
[0275]
[0276] Recalculate the weighted average values (X, Y) of the fruit and vegetable in the x and y directions and transmit the new weighted average values to the grasping execution mechanism in real time to ensure that the grasping operation is based on more accurate coordinates. At the same time, continuously optimize the weight adjustment strategy.
[0277] Convert to the world coordinate system and obtain the external parameter matrix [R|T] of the camera (including the rotation matrix R and the translation vector T). This external parameter matrix is usually obtained through prior calibration work such as camera calibration and is stored in the robot's vision system configuration file or in a parameter storage area that can be obtained in real time. Using the external parameter matrix [R|T] of the camera, convert the coordinates (X, Y) (the result obtained from the above weighted average calculation) in the camera coordinate system to the coordinates (X_w, Y_w) in the world coordinate system through the corresponding coordinate transformation formula. The specific coordinate transformation calculation process is carried out according to the mathematical principles of spatial coordinate transformation, such as through matrix multiplication and other operations, so as to convert the coordinate position of the fruit and vegetable to the world coordinate system corresponding to the actual working space of the robot, facilitating the subsequent manipulator to perform accurate operations based on this coordinate.
[0278] Determine the grasping position
[0279] After completing the above coordinate transformation, combined with the depth information estimation (the depth information of the target fruit and vegetable can be obtained through the existing depth sensor in the robot vision system, or the corresponding depth value Z_w can be obtained through an image-based depth estimation algorithm), determine the complete three-dimensional coordinate position (X_w, Y_w, Z_w) of the target fruit and vegetable in the world coordinate system. According to factors such as the type, shape of the fruit and vegetable, and the grasping characteristics of the manipulator, comprehensively consider and determine the most suitable grasping position. For example, for spherical fruits and vegetables, the grasping position may be selected near its center position; for long and strip-shaped fruits and vegetables, the grasping position may be selected at the position of its center of gravity, etc., providing accurate target position information for the subsequent precise grasping operation of the manipulator.
[0280] Overall detection and processing loop
[0281] For the same target fruit and vegetable, continuously process the 10 detection results according to the above weighted average algorithm, including removing outliers, calculating the weighted average coordinates, converting the coordinate system, and determining the grasping position, etc., to ensure that the target position information on which the manipulator performs the grasping action each time is optimized and as accurate as possible, improving the accuracy and success rate of the entire picking process.
[0282] The above is only a preferred example of the present invention and is not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made to the present invention shall be included within the protection scope of the present invention.
Claims
1. A fruit and vegetable picking robot system based on data enhancement and recognition accuracy optimization, characterized in that: include Visual recognition module: Equipped with a high-resolution camera, it is used to obtain fruit and vegetable images in real time, and uses the YOLOv5 target detection algorithm to identify and locate fruits and vegetables in real time. It can classify fruits and vegetables with high precision and detect them in real time, and can maintain stable performance under complex backgrounds; Harvesting robot: used to perform fruit and vegetable picking operations, including a multi-degree-of-freedom robot and a flexible end gripper, which can achieve precise grasping and reduce damage to fruits and vegetables; Control module: connects the visual recognition module and the picking robot arm to process the recognition data and generate picking path planning and robot arm control instructions; Mobile module: used for autonomous movement of the robot body in the farmland, supports obstacle avoidance and path planning, and ensures that the robot can efficiently complete multi-point picking tasks.
2. The fruit and vegetable picking robot system based on data enhancement and recognition accuracy optimization according to claim 1 is characterized in that: The visual recognition module uses the YOLOv5 target detection algorithm to perform real-time recognition and positioning of fruits and vegetables, which is achieved by building a recognition and grasping system model, including the following steps: Step 1: Image acquisition: Collect target fruit and vegetable images as a data set through outdoor shooting; Step 2: Preprocessing: Preprocess the collected images, including denoising and filtering, to improve image quality; Step 3: Expand the dataset: Expand the dataset in a targeted manner to deal with different influencing factors. The specific method is as follows: expand the dataset by adjusting the brightness, contrast, saturation and hue color attributes of the image under simulated different lighting conditions, and transform the color of the image by randomly adjusting the parameters to generate images under different lighting and color conditions; By changing the viewing angle of the image to simulate different shooting angles, in the multi-scale perspective transformation, the perspective transformation matrix H is adjusted i Parameters, apply random scaling factors to the perspective transformation matrix elements, apply multiple perspective transformations of different scales to the same image, there are N scales, then the perspective transformation matrix at the i-th scale is expressed as: where s x,i and y,i is the scaling factor at the i-th scale, indicating the scaling of the image in the horizontal and vertical directions; b i , d i is the shear parameter, indicating a slight tilt of the image along the diagonal direction; c i , f i is the translation parameter, which indicates the translation of the image in the horizontal and vertical directions; g i ,h i It is the perspective parameter, which simulates the changes of objects at different distances in the image; For each scale i, the transformed coordinates (x', y') of each pixel (x, y) in the image are calculated using the following formula: The final transformation result needs to be normalized: Introducing more data diversity during training; Step 4: Dataset annotation; Step 5: Establish recognition model; Step 6: Model conversion and deployment; Step 7: Target positioning; Step 8: Depth estimation of the three-dimensional world target; Step 9: Optimize lens distortion; Step 10: Introduce weighted average algorithm to improve accuracy.
3. The fruit and vegetable picking robot system based on data enhancement and recognition accuracy optimization according to claim 2 is characterized in that: Use fruit and vegetable images to simulate different weather conditions; Haze weather simulation, using Gaussian blur to simulate the blur effect of the image caused by haze; let the original image be I original (x, y), the Gaussian kernel is G(x, y, σ), where σ controls the degree of blur, and the convolution operation is: I fog (x,y)=I original (x,y)*G(x,y,σ) Where G(x,y,σ) is a two-dimensional Gaussian kernel: A layer of haze effect with transparency is superimposed on the blurred image; assuming that the haze transparency is α between 0 and 1, the composite image is: I'foggy(x,y)=αIfog(x,y)+(1-α)I original (x,y) Among them, α controls the concentration of haze. The closer the value is to 1, the stronger the haze effect is. Rainy weather simulation uses randomly generated transparent circular water drops to simulate the effect of raindrops; set a water drop layer R(x,y), which is composed of n randomly generated water drops; the shape of each water drop is simulated by a Gaussian function, and the size and position of the water drop are random; for each water drop, set its position to (x i ,y i ), with a standard deviation of σ i , the resulting drop shape is: Among them, N(xx i ,yy i ,σ i ) is a two-dimensional Gaussian distribution function used to simulate the shape of water droplets; In order to simulate the weak light on a rainy day, the brightness of the image is reduced overall; let the brightness adjustment factor be β, 0<β<1, then the adjusted rainy day image is: I'rainy(x,y)=βIoriginal(x,y)-αR(x,y) Among them, α is the intensity coefficient of the water drop layer, which controls the visibility of raindrops; Snowy weather simulation, the distribution of snowflakes is achieved by randomly generated small white dots. Each snowflake is represented by a small Gaussian function, and its position and size are random; let the snowflake layer be S(x,y), which consists of n snowflakes, and the position of each snowflake is (x i ,y i ), with a standard deviation of σ i ,but: A slight contrast adjustment is performed on the image to enhance the contrast between the snowflakes and the background; assuming the brightness adjustment factor is β, the final snowy image is: I'snowy(x,y)=β·(Ioriginal(x,y)+αS(x,y)) Among them, α controls the intensity of the snowflakes, and β controls the overall brightness; The simulation of a sunny day increases the vividness of the image by improving the brightness, contrast and color saturation. If the brightness adjustment factor is β, the adjusted image is: I'bright(x,y)=Ioriginal(x,y)+β By enhancing the contrast of the image, the layering of the image is increased. The contrast factor is α, α>1, then the enhanced image is: I'contrast(x,y)=α·(I'bright(x,y)-μ)+μ Among them, μ is the average brightness of the original image, ensuring that the average brightness of the enhanced image remains unchanged; The color of the image is enhanced by increasing the color saturation of the image; the saturation enhancement process is achieved by the following formula: I'sunny(x,y)=γ·I'contrast(x,y) Where γ is the saturation factor, γ>1.
4. The fruit and vegetable picking robot system based on data enhancement and recognition accuracy optimization according to claim 2 is characterized in that: The lens distortion optimization described in step nine and the lens error correction method adopts a segmented correction method: (1) Regional division: Divide the lens field of view into a central area and a peripheral area; Central area <R threshold : Small distortion, using distortion model correction; Edge area r ≥ R threshold : The distortion is large, and manual sampling and interpolation correction are used; (2) Manual sampling: Select several sampling points in the 1 / 4 area of the lens field of view and record the deviation between the uncorrected coordinates and the actual coordinates; (3) Interpolation fitting: Generate a correction mapping table using bilinear interpolation or polynomial fitting; Δx=a0+a1x+a2y+a3xy Δy=b0+b1x+b2y+b3xy where a i ,b i is the fitting coefficient; (4) Symmetry optimization: By utilizing the symmetry of the rectangular field of view of the lens, only the correction parameters of 1 / 4 of the area need to be calibrated, and the remaining area reuses the mapping table through coordinate transformation (x->-x, y->-y); Edge area: x c =x+Δx(r,θ) y c =y+Δy(r,θ) Where Δx(r,θ) and Δy(r,θ) are the corrections obtained by interpolation.
Citation Information
Patent Citations
Picking robot and picking method for kiwi fruits
CN103749094A
Control method and control system of intelligent fruit picking robot
CN115299245A
Intelligent picking robot
CN116097981A
Sprout tea harvesting machine and tea harvesting method based on multi modal recognition by using infrared ray spectrum
JP2024169326A
Cited By
Intelligent mushroom harvesting robot
CN120982354A
Robot image recognition capability testing and evaluating method
CN121438076A
