Fruit and vegetable picking robot system based on data enhancement and recognition accuracy optimization

Abstract

The present application relates to a kind of based on data enhancement and identification precision optimization fruit and vegetable picking robot system, belong to robot field.By vision recognition module, mechanical arm module, control module and mobile module are composed, using YOLOv5 target detection algorithm is carried out real-time identification and positioning to fruit and vegetable.Visual module is collected image by monocular camera, and utilize YOLOv5 algorithm accurately detects the type and location of fruit and vegetable;Control module according to identification result, coordinate mechanical arm and execute picking action, while planning the moving path of robot to adapt to complex farmland environment;Mechanical arm module uses multi-degree of freedom design and carries flexible gripper, can complete efficient picking operation under the premise of protecting fruit and vegetable.In addition, the system also supports the dynamic update of model, continuously optimizes identification performance by adding new fruit and vegetable data.The system can meet the needs of different crops and environment while improving picking efficiency and reducing labor cost, has wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of robotics, and particularly to an intelligent fruit and vegetable harvesting robot system based on deep learning, especially a fruit and vegetable harvesting robot system based on data augmentation and optimized recognition accuracy, which is mainly used for fruit and vegetable harvesting tasks in agricultural scenarios. Background Technology

[0002] With the continued growth of the global population, agricultural production faces increasingly severe challenges. To improve agricultural production efficiency and ensure food security, the application of automation technology in agriculture has become an important development trend. In particular, the rapid development of visual recognition, machine learning, and artificial intelligence technologies has led to the increasingly widespread and in-depth application of robots in agriculture.

[0003] Currently, fruit and vegetable harvesting robots have achieved initial application results in several fields. These robots generally integrate image processing algorithms, sensors, robotic arms, and grasping tools to achieve accurate identification and harvesting of fruits and vegetables. Image processing algorithms are responsible for identifying the harvesting target, and the robotic arm locates and grasps based on the identification information. However, even though these robots have improved the level of harvesting automation to a certain extent, many problems related to grasping accuracy still exist in practical applications.

[0004] Currently, most fruit and vegetable harvesting robots rely on simple image processing algorithms to identify objects. However, in complex real-world environments, with constantly changing lighting, cluttered surroundings, and a wide variety of fruits and vegetables, these robots perform poorly in object recognition and positioning accuracy. The process of grasping fruits and vegetables demands extremely high precision. Existing fruit and vegetable harvesting robots are still in the early stages of applying efficient computational solutions such as depth estimation, making it difficult to make efficient decisions and operations in real time in complex environments. Furthermore, most large agricultural machines cannot use high-precision cameras for recognition due to cost constraints, resulting in insufficient fruit and vegetable positioning capabilities. When faced with fruits and vegetables of varying shapes and sizes, or under different weather conditions, they struggle to locate them accurately, which significantly impacts the efficiency and accuracy of subsequent harvesting and handling operations. Summary of the Invention

[0005] The purpose of this invention is to provide a fruit and vegetable harvesting robot system based on data augmentation and optimized recognition accuracy, solving the problems of poor object recognition and positioning accuracy in existing fruit and vegetable harvesting robots, and difficulty in making efficient decisions and operations in real time in complex environments. This invention is a multifunctional intelligent fruit and vegetable cultivation system based on YOLOv5 vision technology and a robotic arm. Through advanced sensors, computer vision, machine learning, and robot control technology, it identifies and grasps fruits and vegetables, improving the efficiency and accuracy of the fruit and vegetable harvesting robot during the harvesting process.

[0006] The above-mentioned objective of the present invention is achieved through the following technical solution:

[0007] A fruit and vegetable harvesting robot system based on data augmentation and optimized recognition accuracy, including

[0008] Visual recognition module: Equipped with a high-resolution camera, it is used to acquire fruit and vegetable images in real time and to identify and locate fruits and vegetables in real time using the YOLOv5 target detection algorithm. It can perform high-precision classification and real-time detection of fruits and vegetables and maintain stable performance in complex backgrounds.

[0009] Harvesting robotic arms: used to perform fruit and vegetable harvesting operations, including multi-degree-of-freedom robotic arms and flexible end grippers, which can achieve precise grasping and reduce damage to fruits and vegetables;

[0010] Control module: Connects the vision recognition module and the picking robotic arm, and is used to process recognition data, generate picking path planning and robotic arm control instructions;

[0011] Mobile module: Used for the autonomous movement of the robot body in the farmland, supporting obstacle avoidance and path planning to ensure that the robot can efficiently complete multi-point harvesting tasks.

[0012] The visual recognition module uses the YOLOv5 object detection algorithm to identify and locate fruits and vegetables in real time. This is achieved by building a recognition and grasping system model, including the following steps:

[0013] Step 1: Image Acquisition: Collect images of the target fruits and vegetables as a dataset through outdoor photography;

[0014] Step 2, Preprocessing: The acquired images are preprocessed, including noise reduction and filtering, to improve image quality;

[0015] Step 3: Expand the dataset: Expand the dataset in a targeted manner to cope with different influencing factors. The specific method is as follows: expand the dataset by simulating different lighting conditions and adjusting the brightness, contrast, saturation and hue of the image. Use random parameter adjustment to perform color transformation on the image to generate images under different lighting and color conditions.

[0016] By changing the image's perspective to simulate different shooting angles, the perspective transformation matrix H is adjusted in multi-scale perspective transformation. i The parameters are used to apply random scaling factors to the elements of the perspective transformation matrix. If multiple perspective transformations at different scales are applied to the same image, with N scales, then the perspective transformation matrix at the i-th scale is expressed as:

[0017]

[0018] Where s x,i and s y,ib is the scaling factor at the i-th scale, representing the scaling of the image in the horizontal and vertical directions; i d i This is the clipping parameter, representing a slight tilt of the image along the diagonal direction; c i f i These are translation parameters, representing the horizontal and vertical translation of the image; g i h i These are perspective parameters, representing the changes in an image when simulating objects at different distances.

[0019] For each scale i, the transformed coordinates (x', y') of each pixel (x, y) in the image are calculated using the following formula:

[0020]

[0021] The final transformation result needs to be normalized:

[0022]

[0023] Introduce more data diversity during the training process;

[0024] Step 4: Dataset labeling;

[0025] Step 5: Establish the recognition model;

[0026] Step Six: Model Conversion and Deployment;

[0027] Step 7: Target Positioning;

[0028] Step 8: Depth estimation for targets in the three-dimensional world;

[0029] Step 9: Optimize lens distortion;

[0030] Step 10: Introduce a weighted average algorithm to improve accuracy.

[0031] Images of fruits and vegetables are used to simulate different weather conditions.

[0032] The smog simulation uses Gaussian blur to simulate the blurring effect caused by smog in an image; let the original image be (I original (x,y)), the Gaussian kernel is (G(x,y,σ)), where (σ) controls the degree of blurring, and the convolution operation is:

[0033] I fog (x,y)=I original (x,y)*G(x,y,σ)

[0034] Where (G(x,y,σ)) is a two-dimensional Gaussian kernel:

[0035]

[0036] A haze effect with transparency is superimposed on a blurred image; let the haze transparency be (α) (between 0 and 1), then the composite image is:

[0037] I'foggy(x,y)=αIfog(x,y)+(1-α)I original (x,y)

[0038] Among them, (α) controls the concentration of smog, and the closer the value is to 1, the stronger the smog effect;

[0039] Rainy weather simulation uses randomly generated transparent circular water droplets to simulate the effect of raindrops; a water droplet layer (R(x,y)) is set, consisting of (n) randomly generated water droplets; the shape of each water droplet is simulated using a Gaussian function, and the size and position of the water droplets are random; for each water droplet, its position is set to ((x...y)... i ,y i The standard deviation is (σ). i The resulting water droplets are shaped as follows:

[0040]

[0041] Among them, (N(xx) i yy i ,σ i )) is a two-dimensional Gaussian distribution function used to simulate the shape of a water droplet;

[0042] To simulate the low light conditions of a rainy day, the overall brightness of the image is reduced; let the brightness adjustment factor be (β) ((0<β<1)), then the adjusted rainy day image is:

[0043] I'rainy(x,y)=βIoriginal(x,y)-αR(x,y)

[0044] Where (α) is the intensity coefficient of the water droplet layer, which controls the visibility of the raindrops;

[0045] In the snow weather simulation, the distribution of snowflakes is achieved by randomly generated small white dots. Each snowflake is represented as a small Gaussian function, with its position and size being random. Let the snowflake layer be (S(x,y)), which consists of (n) snowflakes, and the position of each snowflake is ((x,y)). i ,y i The standard deviation is (σ). i ),but:

[0046]

[0047] A slight contrast adjustment is made to the image to enhance the contrast between the snowflakes and the background; assuming the brightness adjustment factor is (β), the final snowy image is as follows:

[0048] I'snowy(x,y)=β·(Ioriginal(x,y)+αS(x,y))

[0049] Among them, (α) controls the intensity of the snowflake, and (β) controls the overall brightness;

[0050] To simulate a sunny day, the image's vibrancy is increased by enhancing brightness, contrast, and color saturation. Let the brightness adjustment factor be (β). The adjusted image would look like this:

[0051] I'bright(x,y)=Ioriginal(x,y)+β

[0052] By enhancing the contrast of an image, increasing its sense of depth, and using a contrast factor of (α), typically (α>1), the enhanced image would look like this:

[0053] I'contrast(x,y)=α·(I'bright(x,y)-μ)+μ

[0054] Where (μ) is the average brightness of the original image, ensuring that the average brightness of the enhanced image remains unchanged;

[0055] The colors of an image are enhanced by increasing its color saturation; the process of saturation enhancement is achieved using the following formula:

[0056] i'sunny(x,y)=γ·I'contrast(x,y)

[0057] Where (γ) is the saturation factor (usually (γ>1)).

[0058] Step nine describes a method for optimizing lens distortion and correcting lens errors, employing a segmented correction approach.

[0059] (1) Regional division:

[0060] Divide the lens's field of view into a central region and an edge region;

[0061] Central area (r) <R threshold ): The distortion is small, and a distortion model is used for correction;

[0062] Edge region (r≥R) threshold Large distortion was corrected using manual sampling and interpolation.

[0063] (2) Manual sampling:

[0064] Several sampling points were selected in 1 / 4 of the lens's field of view, and the deviation between the uncorrected coordinates and the actual coordinates was recorded.

[0065] (3) Interpolation fitting: Use bilinear interpolation or polynomial fitting to generate a corrected mapping table;

[0066] Δx=a0+a1x+a2y+a3xy

[0067] Δy=b0+b1x+b2y+b3xy

[0068] Where a i ,b i These are the fitting coefficients;

[0069] (4) Symmetry optimization: By utilizing the symmetry of the rectangular field of view of the lens, only the correction parameters of 1 / 4 of the region need to be calibrated, and the mapping table is reused in the remaining regions through coordinate transformation (x->-x, y->-y);

[0070] Edge area:

[0071] x c =x + Δx(r,θ)

[0072] y c =y + Δy(r,θ)

[0073] Where Δx(r,θ) and Δy(r,θ) are correction values ​​obtained through interpolation.

[0074] The beneficial effects of this invention are as follows:

[0075] Compared with existing technologies, this invention has the following significant advantages: By innovatively introducing a weather simulation algorithm, the success rate of the robotic arm's grasping in complex environments such as rain, fog, and strong light is increased by more than 15%, solving the problem of poor environmental adaptability of traditional vision systems; at the same time, by employing dynamic distortion compensation technology, the positioning accuracy of the monocular vision system is improved from ±2mm to within ±0.5mm, effectively overcoming the positioning deviation caused by lens distortion. These two technological breakthroughs enable the monocular vision grasping system of this invention to maintain its low-cost advantage while possessing the high precision and robustness of multi-camera systems, providing a superior technical solution for industrial automation. Attached Figure Description

[0076] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate the invention and are used to explain it, but do not constitute an undue limitation of the invention.

[0077] Figure 1 This is a scaled-down image of a tomato used in this invention;

[0078] Figure 2This is a simulated weather map for the present invention;

[0079] Figure 3 This is the depth information estimation model of the present invention;

[0080] Figure 4 This is a curve showing the horizontal distortion of the lens in this invention;

[0081] Figure 5 This is a curve showing the vertical distortion of the lens in this invention; Detailed Implementation

[0082] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0083] See Figures 1 to 5 As shown, the fruit and vegetable harvesting robot system of this invention, based on data augmentation and optimized recognition accuracy, is an intelligent fruit and vegetable harvesting robot system capable of deep learning, primarily used for fruit and vegetable harvesting tasks in agricultural scenarios. This invention improves the performance of previous fruit and vegetable robotic arms, enabling accurate object positioning even when the accuracy of a single-lens camera is low or environmental information changes (such as fluctuating lighting or foliage obstruction). It primarily employs advanced technologies and algorithms, and through specific data processing, optimizes and integrates positional information to lock onto targets with high precision and accurately grasp them. This invention can significantly improve harvesting efficiency and ensure fruit and vegetable quality with a slight increase in equipment cost, contributing to the intelligent development of modern agriculture. This invention consists of a visual recognition module, a harvesting robotic arm, a control module, and a movement module, with the core technology being the use of the YOLOv5 target detection algorithm for real-time identification and positioning of fruits and vegetables. The vision module acquires images via a monocular camera and uses the YOLOv5 algorithm to accurately detect the type and location of fruits and vegetables. Based on the recognition results, the control module coordinates the robotic arm to perform the harvesting action, while simultaneously planning the robot's movement path to adapt to complex farmland environments. The harvesting robotic arm employs a multi-degree-of-freedom design and is equipped with flexible grippers, enabling efficient harvesting while protecting the fruits and vegetables. Furthermore, the system supports dynamic model updates, continuously optimizing recognition performance by incorporating new fruit and vegetable data. This system improves harvesting efficiency and reduces labor costs while meeting the needs of different crops and environments, demonstrating broad application prospects.

[0084] The harvesting robotic arm is equipped with flexible end effectors made of soft materials, which can adjust the gripping force according to the shape and size of fruits and vegetables to avoid surface damage. The control module integrates a path planning algorithm, which can generate the optimal harvesting path based on the orchard terrain and the distribution of fruits and vegetables, improving work efficiency. The mobility module adopts a wheel-based chassis design, which has omnidirectional mobility, enabling lateral movement, diagonal movement, and in-situ rotation, allowing for flexible movement within orchards or farmlands, adapting to complex harvesting environments, and improving operational accuracy. The system supports users adding new fruit and vegetable datasets through annotation tools to optimize the YOLOv5 model, making it adaptable to more fruit and vegetable varieties and complex background environments. The entire system is modularly designed, allowing users to easily upgrade configurations according to actual needs, such as changing the vision camera or robotic arm type to adapt to different operating scenarios. The robot body is equipped with a backup battery and charging module, supporting long-term operation and ensuring continuous operation during harvesting.

[0085] This invention relates to a fruit and vegetable harvesting robot system based on data augmentation and optimized recognition accuracy, used for automated harvesting of fruits and vegetables. The system includes a mobile module and a harvesting robotic arm. The harvesting robotic arm is used to grasp the fruits and vegetables, the mobile module controls the robot's movement in the field, and the harvesting robotic arm is used to separate the fruits and vegetables. The mobile module serves as the support and movement structure for the fruit and vegetable harvesting robot, with the bottom of the harvesting robotic arm fixedly mounted on it. The end of the harvesting robotic arm is connected to a harvesting device (flexible end effector gripper). The harvesting robotic arm includes a four-axis robotic arm, an actuator, and a monocular camera mounted at the end of the four-axis robotic arm. The mobile module includes a drive module that uses a wheeled chassis to drive the fruit and vegetable harvesting robot. The harvesting module includes a gripper and a monocular camera; the gripper can open up to 180 degrees.

[0086] The model construction of the identification and grasping system of the fruit and vegetable harvesting robot of the present invention includes the following steps:

[0087] Step 1: Image Acquisition: Collect images of the target fruits and vegetables outdoors as a dataset;

[0088] Step 2, Preprocessing: The acquired images are preprocessed, including noise reduction and filtering, to improve image quality;

[0089] Step 3: Expand the dataset: Expand the original dataset to accommodate different influencing factors such as lighting, viewing angle, and weather. The specific methods are as follows:

[0090] To simulate different lighting conditions, the dataset is expanded by adjusting the color attributes of the image, such as brightness, contrast, saturation, and hue. By randomly adjusting the parameters, the image is color-transformed within a certain range (e.g., brightness -a to +a, contrast b-3b times, saturation c-3c times, hue -g to +g degrees), generating images under different lighting and color conditions.

[0091] Furthermore, different shooting angles are simulated by changing the image's perspective. Traditional random perspective transformations typically use fixed scaling factors and transformation parameters, which can lead to generated images that are too similar and fail to adequately simulate complex scenes in the real world. To increase diversity, multi-scale perspective transformation is introduced, so that the generated images vary not only in perspective but also in scale. In multi-scale perspective transformation, the perspective transformation matrix H is adjusted... i The parameters are used to apply random scaling factors to the elements of the perspective transformation matrix. If multiple perspective transformations at different scales are applied to the same image, with N scales, then the perspective transformation matrix at the i-th scale can be expressed as:

[0092]

[0093] Where s x,i and s y,i is the scaling factor at the i-th scale, representing the scaling of the image in the horizontal and vertical directions. i d i This is the clipping parameter, representing a slight tilt of the image along the diagonal direction. i f i This is the translation parameter, representing the translation of the image in the horizontal and vertical directions. g i h i It is a perspective parameter that represents the changes in an image when simulating objects at different distances.

[0094] For each scale i, the transformed coordinates (x', y') of each pixel (x, y) in the image can be calculated using the following formula:

[0095]

[0096] The final transformation result needs to be normalized:

[0097]

[0098] Introduce more data diversity during the training process.

[0099] Furthermore, the fruit and vegetable images are simulated under different weather conditions. First, a hazy weather simulation is performed, which is typically achieved through blurring and reducing image sharpness. The hazy effect is simulated in two parts: first, Gaussian blur is used to simulate the blurring effect caused by haze. Gaussian blur is a low-pass filter that can blur details in an image. Let the original image be (I... original (x,y)), the Gaussian kernel is (G(x,y,σ)), where (σ) controls the degree of blurring, and the convolution operation is:

[0100] I fog (x,y)=I original (x,y)*G(x,y,σ)]

[0101] Where (G(x,y,σ)) is a two-dimensional Gaussian kernel:

[0102]

[0103] Furthermore, to simulate the concentration of smog, a layer of smog effect with transparency is superimposed on the blurred image. Let the smog transparency be (α) (between 0 and 1), then the composite image is:

[0104] I'foggy(x,y)=αIfog(x,y)+(1-α)I original (x,y)

[0105] Among them, (α) controls the concentration of smog, and the closer the value is to 1, the stronger the smog effect.

[0106] Furthermore, rainy weather simulation is implemented, with effects including simulating water droplet traces and a reduction in overall brightness. Modeling is performed as follows: randomly generated transparent circular water droplets are used to simulate the effect of raindrops. A water droplet layer (R(x,y)) is defined, consisting of (n) randomly generated water droplets. The shape of each water droplet is simulated using a Gaussian function, and the size and position of the droplets are randomized. For each water droplet, its position is set to ((x...y)...y). i ,y i The standard deviation is (σ). i The resulting water droplets are shaped as follows:

[0107]

[0108] Among them, (N(xx) i yy i ,σ i )) is a two-dimensional Gaussian distribution function used to simulate the shape of a water droplet.

[0109] Furthermore, to simulate the low light conditions of a rainy day, the overall brightness of the image is reduced. Brightness adjustment is achieved through image normalization. Let the brightness adjustment factor be (β) ((0<β<1)), then the adjusted rainy day image is:

[0110] I'rainy(x,y)=βIoriginal(x,y)-αR(x,y)

[0111] Here, (α) is the intensity coefficient of the water droplet layer, which controls the visibility of the raindrops.

[0112] Furthermore, the snow weather simulation is similar to the rain weather simulation, but the snowflakes are typically smaller and more evenly distributed. The distribution of snowflakes is achieved through randomly generated small white dots. Each snowflake is represented as a small Gaussian function, with its position and size being random. Let the snowflake layer be (S(x,y)), which consists of (n) snowflakes, each with a position ((x,y)). i ,y i The standard deviation is (σ). i ),but:

[0113]

[0114] Similar to rainy days, snowy days also reduce overall brightness. In addition to brightness adjustments, slight contrast adjustments can be made to enhance the contrast between the snowflakes and the background. Assuming the brightness adjustment factor is (β), the final snowy image would look like this:

[0115] I'snowy(x,y)=β·(Ioriginal(x,y)+αS(x,y))

[0116] Among them, (α) controls the intensity of the snowflakes, and (β) controls the overall brightness.

[0117] Furthermore, the simulation of a sunny day primarily enhances the vibrancy of the image by increasing brightness, contrast, and color saturation. To increase the image's brightness and make it appear more luminous, let the brightness adjustment factor be (α). Then, the adjusted image is:

[0118] I'bright(x,y)=Ioriginal(x,y)+β

[0119] By enhancing the contrast of an image, the sense of depth and dimension is increased. Using a simple linear contrast enhancement formula, assuming a contrast factor of α (usually α>1), the enhanced image would be:

[0120] I'contrast(x,y)=α·(I'bright(x,y)-μ)+μ

[0121] Where (μ) is the average brightness of the original image, ensuring that the average brightness of the enhanced image remains unchanged.

[0122] To simulate the vibrant colors of a sunny day, the image's colors are enhanced by increasing its color saturation. This saturation enhancement is achieved using the following formula:

[0123] I'sunny(x,y)=γ·I'contrast(x,y)

[0124] Where (γ) is the saturation factor (usually (γ>1)).

[0125] Step 4: Dataset Labeling: Use labelimg to label the dataset. Accurately label the location and category of each fruit and vegetable to ensure that the dataset used for subsequent model training is of high quality.

[0126] Step 5: Model Building: The YOLOv5 model is used for training. The main hyperparameters during model training are set as follows: Epochs = a, Batch size = b, and the initial learning rate η is set to η0 = c and adjusted during training. The training process updates the model parameters θ through the backpropagation algorithm and optimizer to minimize the loss function L(θ) which includes classification error, location error and confidence error. The classification error is optimized using the cross-entropy loss function.

[0127] Step 6, Model Conversion and Deployment: Deploy the trained model onto the composite fruit and vegetable harvesting robot, and convert the PyTorch model into a TensorRT model for deployment.

[0128] Step 7: Target Localization: Based on the classification results, determine the location of the fruit in the image; when the localization is unclear, a depth estimation model is used. Specifically, this model mainly includes the following steps;

[0129] The trained model is deployed, and a monocular camera detects the target object, obtaining the bounding box output by YOLOv5. The YOLOv5 model will output the bounding box information of the target object, usually returned in the form of [x1,y1,x2,y2].

[0130] Furthermore, since the coordinates output by YOLOv5 are normalized, they need to be converted to pixel coordinates using the intrinsic parameter matrix in the camera: (x p y p )

[0131] x p =x1×z1

[0132] y p =y1×z2

[0133] The formula for the camera intrinsic parameter matrix is ​​as follows:

[0134]

[0135] Among them, f x f represents the focal length of the camera in the direction of movement. y c represents the focal length of the camera in the direction of movement. x and c y These represent the coordinates of the principal point in the image coordinate system, respectively. This matrix contains the focal length f. x ,f y and principal point c x ,c y This is used to convert image coordinates to normalized coordinates in the camera coordinate system.

[0136] x n =(x p -c x ) / f x

[0137] y n =(y p -c y ) / f y

[0138] Step 8: Depth estimation for 3D world targets

[0139] To convert image coordinates to 3D world coordinates, depth information (i.e., the actual distance between the camera and the object in space) is needed. Methods for estimating depth information using a depth camera, LiDAR, or based on object size and a camera model are presented. Assuming camera intrinsic parameters (such as focal length and field of view) and the object's bounding box size in the image are known, a method is established as follows: Figure 2 The specific process for estimating the actual distance between the object and the camera using the shown geometric model (where θ is the camera's field of view, h is the height of the captured image, and d is the depth) is as follows:

[0140] 1. Let the actual width of the ripe fruit be 2r, the actual width visible to the camera on the vertical plane where the ripe fruit is located be 2h, the depth be d, the field of view of the camera be (the field of view is the angle formed by the two edges of the maximum range through which the image of the target object can be seen, with the lens of the optical instrument as the vertex), the maximum side width recognized by the camera be e, the length of the ripe fruit frame recognized by YOLOv5 be y2, and θ0 be measured.

[0141] 2. Based on the principle of similar triangles, the ratio of the actual side width of the ripe fruit to the actual visible width of the camera is equal to the ratio of the side width of the ripe fruit recognized by YOLOv5 to the maximum side width recognized by the camera.

[0142]

[0143] In geometric relationships involving the field of view angle, there is

[0144]

[0145] achievable

[0146] h=d×tanθ(3)

[0147] 3. Substituting equation (2) into equation (1) for derivation: From equation (1), substituting h=d×tanθ, we get: Further simplification yields

[0148] Step 9: Optimize Lens Distortion: Lens distortion is mainly divided into radial distortion and tangential distortion. Specific parameters are as follows: Radial Distortion: Caused by lens curvature, it manifests as stretching or compression of image edges. Commonly used parameters are k1, k2, and k3 (radial distortion coefficients).

[0149] Tangential distortion: caused by the lens not being parallel to the imaging plane, manifesting as image tilt or distortion. Commonly used parameters are p1 and p2 (tangential distortion coefficients).

[0150] Standard distortion model formula:

[0151] x c =x(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0152] yx=y(1+k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 )

[0153] Where x c y c For edge coordinates, This represents the distance from a pixel to the center of the lens.

[0154] Lens error correction methods: Due to significant distortion at the lens edges, a segmented correction method is used.

[0155] 1. Regional division:

[0156] The lens field of view is divided into a central region and an edge region.

[0157] Central area (r) <R threshold): The distortion is relatively small, and the above standard distortion model is used for correction.

[0158] Edge region (r≥R) threshold ): The distortion is large, so manual sampling and interpolation are used for correction.

[0159] 2. Manual sampling:

[0160] Select several sampling points in one-quarter of the lens's field of view (such as the upper right quadrant) and record the deviation between the uncorrected coordinates and the actual coordinates.

[0161] 3. Interpolation fitting:

[0162] Use bilinear interpolation or polynomial fitting to generate a corrected mapping table.

[0163] Δx=a0+a1x+a2y+a3xy

[0164] Δy=b0+b1x+b2y+b3xy

[0165] Where a i ,b i represents the fitting coefficient.

[0166] 4. Symmetry optimization:

[0167] By taking advantage of the symmetry of the lens's rectangular field of view, only the correction parameters of 1 / 4 of the region need to be calibrated, and the mapping table for the remaining region can be reused through coordinate transformation (x->-x, y->-y).

[0168] Edge area:

[0169] x c =x + Δx(r,θ)

[0170] y c =y + Δy(r,θ)

[0171] Where Δx(r,θ) and Δy(r,θ) are correction values ​​obtained through interpolation.

[0172] Step 10: Introduce a weighted average algorithm to improve accuracy.

[0173] Because YOLOv5's dynamic detection results may fluctuate (for example, the coordinates of the same fruit or vegetable may differ slightly in different time frames, or there may be multiple detections in the same image), relying on a single recognition result for grasping operations results in a relatively large error. The weighted average method can help reduce these small fluctuations when measuring object coordinates, yielding more stable and accurate results. The specific process is as follows:

[0174] Real-time coordinate monitoring and error calculation are performed, and a coordinate processing unit is defined: This unit calculates in real-time the change (Δx = x) between each acquired coordinate and the previous coordinate.i x i1 Δy=y i y i1 ), and the average of the current coordinates and the coordinates of the previous 5 measurements (μ). x μ y Deviation x =x i μ x Deviation y =y i μ y The calculation results are stored in the cache inside the FPGA.

[0175] Furthermore, the setting and judgment of the error threshold are adjusted. For fruits and vegetables with different regular shapes, a threshold Δx for coordinate change is set. threshold Δy threshold Deviation threshold Through real-time monitoring and judgment, the coordinate change and deviation data calculated by the coordinate processing unit are read and compared with the corresponding thresholds in the system parameter configuration file. Once it is found that the acquired coordinate change or deviation exceeds the threshold, a trigger signal is immediately sent to the dynamic weight calibration and weighted average calculation module via an interrupt signal. The interrupt response time is less than 100 microseconds to ensure timely weight calibration.

[0176] Furthermore, weight adjustment calculations are performed first. When a trigger signal is received, the current coordinate data and the corresponding confidence level c output by the YOLOv5 model are read. i According to the preset dynamic weight adjustment formula Perform weight adjustments, where c average The confidence level is calculated as the average of the previous 5 measurements by maintaining a confidence level queue of length 5 in memory. Then, a weighted average is recalculated and transmitted, using adjusted weights, and weighted according to the weighted average formula:

[0177]

[0178] Recalculate the weighted average (X, Y) of fruits and vegetables in the x and y directions.

[0179] After completing real-time coordinate monitoring and error calculation, historical data needs to be retained for weighted average optimization of historical grasping error feedback. This is achieved via a high-speed data bus connected to the grasping execution mechanism, coordinate calculation module, and environmental sensor group. After each grasping operation, the system automatically acquires and records the grasping time (accurate to milliseconds), fruit and vegetable type identification (determined by comparing image recognition results with a pre-stored fruit and vegetable type database), light intensity value collected by the ambient light sensor, temperature value collected by the temperature sensor, humidity value collected by the humidity sensor, weighted average coordinates (X, Y) output by the coordinate calculation module, and actual grasping coordinates (X, Y). actual Y actual ), and calculate the error values ​​between the two in the x and y directions. x =XX actual Error y =YY actual ).

[0180] Furthermore, an error analysis program is set up, triggered by a system timer after every n crawling tasks. This program extracts error data and related crawling information from the historical database via a database connection interface, and analyzes it using KMeans clustering and decision tree algorithms. Feature vectors are constructed, using fruit and vegetable type, ambient light intensity range, temperature and humidity range, and the movement state of the fruits and vegetables during crawling as feature vectors. The KMeans clustering algorithm is used to initially cluster the error data, grouping error data with similar characteristics into one category.

[0181] Furthermore, using the decision tree algorithm, decision tree models are constructed based on different feature conditions to determine whether the error is mainly caused by unreasonable weight allocation in the coordinate calculation method, environmental interference, the inherent characteristics of fruits and vegetables, or other factors. The frequency of each type of error and its impact on the grasping accuracy are analyzed.

[0182] In subsequent crawling tasks, relevant information (fruit and vegetable types, environmental parameters, etc.) is read in real time, and the weights are adjusted in real time according to the rule base. The adjusted weights are then used in the weighted average calculation of the coordinates, and the weighted average is calculated according to the weighted average formula.

[0183]

[0184] The weighted average (X, Y) of the fruits and vegetables in the x and y directions is recalculated, and the new weighted average is transmitted to the grasping execution mechanism in real time to ensure that the grasping operation is based on more accurate coordinates. At the same time, the weight adjustment strategy is continuously optimized.

[0185] Example 1:

[0186] The model building for the recognition and capture system in this embodiment includes the following steps:

[0187] Step 1: Collect 500 images of ripe tomatoes using dataset websites or by taking your own photos as the initial dataset for the experiment.

[0188] Step Two: First, preprocess the data to remove noise. In MATLAB, algorithms such as median filtering and mean filtering are used to remove random noise from the images. Noise removal is performed on each acquired image. This embodiment uses a 640×480 (mm) monocular camera. A suitable 5×5 filtering window is set, and each pixel in the image is traversed to effectively remove various types of noise, such as salt-and-pepper noise and Gaussian noise, introduced by factors such as the shooting environment and equipment, thus improving image clarity.

[0189] Step 3: Parameter Setting and Range Determination: First, determine the parameter range used to adjust the image's color attributes. Set the brightness parameter a to 50 (meaning the brightness can be randomly adjusted within the range of -50 to +50), the contrast parameter b to 1 (meaning the contrast can randomly vary between 1 and 3 times), the saturation parameter c to 1 (meaning the saturation can randomly vary between 1 and 3 times), and the hue parameter g to 30 (the hue can be randomly adjusted between -30 and +30 degrees). These parameter ranges can basically cover possible lighting and color differences.

[0190] Furthermore, for the original images, brightness, contrast, saturation, and hue were randomly adjusted according to a set parameter range using an image processing library in Python to simulate images under different lighting conditions. Multiple perspective transformations at different scales were applied to each image in the adjusted dataset. Considering data diversity and computational resources, an initial attempt was made starting with a relatively small N, set to 5, which quickly verified its effectiveness without excessive resource consumption.

[0191] For each scale i, generate a perspective transformation random matrix:

[0192]

[0193] Where s x,i and s y,i It is the scaling factor at the i-th scale. It represents the scaling of the image in the horizontal and vertical directions.

[0194] s x,i = rand(μ(0.5,1.5))s y,i =rand(μ(0.5,1.5))

[0195] b i d i is the shearing parameter at the i-th scale, representing a slight tilt of the image along the diagonal direction.

[0196] b i =rand(μ(-0.1,0.1))d i =rand(μ(-0.1,0.1))

[0197] c i f i These are translation parameters, representing the translation of the image in the horizontal and vertical directions.

[0198] c i =rand(μ(-100,100))s y,i =rand(μ(-100,100))

[0199] g i h i It is a perspective parameter that represents the changes in an image when simulating objects at different distances.

[0200] g i =rand(μ(-0.001,0.001))h i =rand(μ(-0.001,0.001))

[0201] For each pixel (x, y) in the image, its transformed coordinates (x', y') are calculated using the following formula:

[0202]

[0203] The final transformation result needs to be normalized.

[0204]

[0205] The transformed coordinates (x', y') may not be integers. Therefore, interpolation is performed on them using bilinear interpolation to calculate the value of each pixel in the transformed image. The transformed image I k,i (x',y')=BilinearInterpolation(I k The image (x, y) is transformed using a multi-scale perspective transformation with progressively increasing values ​​of N until N = 10. The transformed image I... k,i Save as a new training sample.

[0206] Furthermore, the effects of different weather conditions are simulated, including smog, rain, and snow.

[0207] For hazy weather, a Gaussian blur function G(x,y) is used to simulate the blurring effect caused by haze, where the standard deviation (σ) controls the degree of blurring. Then, a haze layer with transparency α (α is set sequentially to 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9) is superimposed on the blurred image. The formula for synthesizing the image is as follows:

[0208] I fog = (1-α)I+αI haze

[0209] Among them I haze This represents the smog effect layer, with α controlling the smog concentration.

[0210] The rain effect is achieved by generating a randomly distributed layer of water droplets (R(x,y)) and reducing the overall brightness. The brightness adjustment formula is as follows:

[0211] I'rainy(x,y)=βIoriginal(x,y)-αR(x,y)

[0212] Where β (β takes values ​​of 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 respectively) is a brightness adjustment factor, and the water droplet layer intensity coefficient α is also added.

[0213] The snow simulation is similar to the rain simulation, but the snowflakes are smaller and more evenly distributed, resulting in the final snow scene image.

[0214] I'snowy(x,y)=β·(Ioriginal(x,y)+αS(x,y))

[0215] β (adjusted in the same way as the rain simulation) and α control the brightness and snow intensity, respectively.

[0216] Step 4: Label the dataset using the labelimg tool. Accurately label the location and category of each fruit and vegetable.

[0217] Step 5: Train the YOLOv5 model. The main hyperparameters during model training are set as follows: Epochs = 1000, Batch size = 16, and the initial learning rate η is set to η0 = c and adjusted during training. The training process updates the model parameters θ through the backpropagation algorithm and optimizer to minimize the loss function L(θ) which includes classification error, location error and confidence error. The classification error is optimized using the cross-entropy loss function.

[0218] Step 6, Model Conversion and Deployment: Deploy the trained model onto the composite fruit and vegetable harvesting robot, and convert the PyTorch model into a TensorRT model for deployment.

[0219] Step 7: Deploy the trained model. Use a monocular camera to detect the target object and obtain the bounding box [x1, y1, x2, y2] output by YOLOv5. Further, convert this bounding box to pixel coordinates using the camera's intrinsic parameter matrix: (x... p y p )

[0220] x p =x1×z1

[0221] y p =y1×z2

[0222] The formula for the camera intrinsic parameter matrix is ​​as follows:

[0223]

[0224] Among them, f x f represents the focal length of the camera in the direction of movement. y c represents the focal length of the camera in the direction of movement. x and c y These represent the coordinates of the principal point in the image coordinate system, respectively. This matrix contains the focal length f. x ,f y and principal point c x ,c y This is used to convert image coordinates to normalized coordinates in the camera coordinate system.

[0225] x n =(x p -c x ) / f x

[0226] y n =(y p -c y ) / f y

[0227] Step 8: Depth estimation for 3D world targets

[0228] To convert image coordinates to 3D world coordinates, depth information (i.e., the actual distance between the camera and the object in space) is needed. Methods for estimating depth information using a depth camera, LiDAR, or based on object size and a camera model are presented. Assuming camera intrinsic parameters (such as focal length and field of view) and the object's bounding box size in the image are known, a method is established as follows: Figure 2 The specific process for estimating the actual distance between the object and the camera using the shown geometric model (where θ is the camera's field of view, h is the height of the captured image, and d is the depth) is as follows:

[0229] 1. Let the actual width of the ripe fruit be 2r, the actual width visible to the camera on the vertical plane where the ripe fruit is located be 2h, the depth be d, the field of view of the camera be (the field of view is the angle formed by the two edges of the maximum range through which the image of the target object can pass through the lens, with the lens as the vertex), the maximum width recognized by the camera be e, the length of the ripe fruit frame recognized by YOLOv5 be y2, and the experimental measurement be θ0.

[0230] 2. Based on the principle of similar triangles, the ratio of the actual side width of the ripe fruit to the actual visible width of the camera is equal to the ratio of the side width of the ripe fruit recognized by YOLOv5 to the maximum side width recognized by the camera.

[0231]

[0232] In geometric relationships involving the field of view angle, there is

[0233]

[0234] achievable

[0235] h=d×tanθ (3)

[0236] 3. Substituting (2) into (1) for derivation: From equation (1), substituting h=d×tanθ, we can obtain: Further simplification yields

[0237] Step Nine: Lens parameter settings. According to the standard checkerboard calibration method, set the radial distortion coefficients as follows: k1 = 0.12, k2 = -0.03, k3 = 0.01, p1 = 0.001, p2 = 0.001.

[0238] The image resolution is 1920*1080, with W = 1920 pixels and H = 1080 pixels.

[0239] Take 70% of the diagonal length

[0240] R threshold =0.7 * d = 1500 pixels

[0241] Furthermore, manual sampling and interpolation fitting were performed. The sampling area was selected in the upper right quadrant (1 / 4 region) of the lens field of view, and 20 evenly distributed sampling points were selected to ensure the coverage of typical distortion positions in the edge area.

[0242] Sampling point data recording: Record the uncorrected coordinates (x, y) of each sampling point. raw ,y raw ) and actual coordinates (x) real ,y realCalculate the deviation Δx = x real -x raw Δy=y real -y raw Generate the mapping table using bilinear interpolation:

[0243] Δx=a0+a1x+a2y+a3xy

[0244] Δy=b0+b1x+b2y+b3xy

[0245] The range of fitting coefficients is:

[0246] a0,b0∈[-5.0,5.0]a1,b1∈[-0.1,0.1]a2,b2∈[-0.1,0.1]a3,b3∈[-0.01,0.01]

[0247] Furthermore, symmetry optimization involves symmetrically mapping the corrected mapping table in the upper right quadrant to the other three quadrants. The coordinate transformations are as follows: lower left quadrant: x' = -x, y' = -y; upper left quadrant: x' = -x, y' = y; lower right quadrant: x' = x, y' = -y.

[0248] Furthermore, for each pixel (x, y), a region determination is performed, and its distance to the center is calculated.

[0249]

[0250] When r <R threshold Correction using a standard distortion model:

[0251] x c =x(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0252] y c =y(1+k1r) 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 )

[0253] If r≥R threshold Read Δx and Δy from the mapping table and correct them:

[0254] x c =x + Δx

[0255] y c =y+Δy

[0256] Where Δx(r,θ) and Δy(r,θ) are correction values ​​obtained through interpolation.

[0257] Step 10: After completing the above processing, collect relevant data (x1, x2, y1, y2) from N observations of the target object, where the relevant coordinate value of the i-th measurement is z. i (z i =x 1i x 2i y 1i y 2i In this embodiment, N=10. After the vision system of the composite fruit and vegetable harvesting robot is activated, outlier identification and removal are performed 10 times using the YOLOv5 model for the same target fruit or vegetable. The coordinate information of the tomato in the camera coordinate system is obtained for each detection. Each coordinate contains two components, x and y (the z-axis coordinate corresponding to the depth information is not considered for now and will be processed separately later). For each coordinate component (x1, x2, y1, y2) in these 10 measurements, its mean μ and standard deviation σ are calculated. The mean can be calculated using the conventional arithmetic mean formula. For coordinate components, μ... x = (x1+x2+...+x) 10 ) / 10; the standard deviation σ is calculated according to the standard deviation calculation formula in statistics.

[0258] Furthermore, for each coordinate point obtained from these 10 measurements, when its distance from the corresponding mean in the x-coordinate or y-coordinate exceeds 3 times the standard deviation (i.e., |x... i -μ x |>3σ x or |y i -μ y |>3σ γ If the coordinate point is not found to be an outlier, it is considered an outlier and discarded; otherwise, it is retained as a valid data point for subsequent weighted average calculations. This method removes outlier measurements that may be caused by detection errors, environmental interference, or other factors, thus ensuring the quality of the data used in the weighted average calculation.

[0259] Furthermore, for the remaining valid coordinate data after outlier removal, weights are assigned based on the confidence level of each recognition result. In the detection results output by the YOLOv5 model, each detection box has a corresponding confidence value, indicating the reliability to which the detection box contains the target tomato. The confidence level of the i-th recognition is denoted as c. i First, normalize the confidence scores of all valid data points to ensure that the sum of the weights is 1. The specific calculation method is w. i =c i / (c1+c2+...+c10 In this way, the measurement results with higher confidence levels have a greater weight in the weighted average calculation, making more reasonable use of the reliability differences between different test results.

[0260] Further, a weighted average is calculated. Based on the following weighted average calculation formula, the weighted average values ​​X and Y of the target fruits and vegetables in the x and y coordinate directions are calculated respectively.

[0261] Weighted average

[0262] Weighted average

[0263] Where, x i and y i w are the x and y coordinates of the i-th valid data point. i Here, represents the corresponding weight, and n is the number of valid data points (the number of data points remaining after outlier removal). Through the above calculations, a more stable and accurate coordinate position of fruits and vegetables is obtained in the camera coordinate system after weighted averaging.

[0264] Real-time calculation of the change in coordinates between each acquisition and the previous acquisition (Δx = x) i x i1 Δy=y i y i1 ), and the average of the current coordinates and the coordinates of the previous 5 measurements (μ). x μ y Deviation x =x i μ x Deviation y =y i μ y The calculation results are stored in the cache inside the FPGA.

[0265] Furthermore, the setting and judgment of the error threshold are adjusted. For small, regularly shaped fruits and vegetables [Category 1], a coordinate change threshold Δx is set. threshold = 2 pixels, Δy threshold =2 pixels, deviation threshold Pixels Pixels; for large, irregularly shaped fruits and vegetables [Category 2], the coordinate change threshold is set to Δx. threshold = 5 pixels, Δy threshold =5 pixels, deviation threshold set to Pixels Pixel.

[0266] Through real-time monitoring and judgment, the error threshold setting and judgment module reads the coordinate change and deviation data calculated by the coordinate processing unit in real time and compares them with the corresponding thresholds in the system parameter configuration file. Once it is found that the acquired coordinate change or deviation exceeds the threshold, a trigger signal is immediately sent to the dynamic weight calibration and weighted average calculation module through an interrupt signal. The interrupt response time is less than 100 microseconds to ensure timely weight calibration.

[0267] Dynamic weight calibration and weighted average calculation: First, weight adjustment calculation is performed. When a trigger signal is received, the current coordinate data and the corresponding confidence score c output by the YOLOv5 model are read. i According to the preset dynamic weight adjustment formula Perform weight adjustments, where c average The confidence level is calculated as the average of the previous 5 measurements by maintaining a confidence level queue of length 5 in memory. Then, a weighted average is recalculated and transmitted, using adjusted weights, and weighted according to the weighted average formula:

[0268]

[0269] Recalculate the weighted average (X, Y) of fruits and vegetables in the x and y directions.

[0270] After completing real-time coordinate monitoring and error calculation, historical data needs to be retained for weighted average optimization of historical grasping error feedback. This is achieved via a high-speed data bus connected to the grasping execution mechanism, coordinate calculation module, and environmental sensor group. Within 100 milliseconds after each grasping operation, the system automatically acquires and records the grasping time (accurate to milliseconds), fruit and vegetable type identification (determined by comparing image recognition results with a pre-stored fruit and vegetable type database), light intensity value collected by the ambient light sensor, temperature value collected by the temperature sensor, humidity value collected by the humidity sensor, weighted average coordinates (X, Y) output by the coordinate calculation module, and actual grasping coordinates (X, Y). actual Y actual ), and calculate the error values ​​between the two in the x and y directions. x =XX actual Error y =YY actual ).

[0271] Furthermore, an error analysis program is set up, which is triggered by a system timer after every 100 crawling tasks are completed. This program extracts error data and related crawling information from the historical database through a database connection interface, and performs analysis using the KMeans clustering algorithm and the decision tree algorithm.

[0272] A feature vector is constructed using the following features: fruit and vegetable type (represented by a unique code, such as 001 for tomatoes and 002 for bananas), ambient light intensity range (divided into multiple ranges such as 0-200 lux, 20-1500 lux, 50-1000 lux), temperature range (40℃-0℃, 0℃-20℃, 20℃-40℃, etc.), humidity range (0%-30%RH, 31%-60%RH, 61%-100%RH), and the movement state of the fruit and vegetable during grasping (determined by displacement sensor data before the robotic arm grasps, with 0 for stationary, 1 for low-speed movement, and 2 for high-speed movement).

[0273] The KMeans clustering algorithm was used to initially cluster the error data, grouping error data with similar characteristics into one category. Then, a decision tree algorithm was used to construct a decision tree model based on different feature conditions to determine whether the errors were primarily caused by unreasonable weight allocation in the coordinate calculation method (by analyzing historical weight allocation data and error correlation to determine if certain coordinate points were overweighted or underweighted), environmental interference (increased image noise leading to coordinate recognition deviation when light intensity is below a certain value, determined by comparing the error frequency and magnitude under different light intensities), the inherent characteristics of fruits and vegetables (smooth, reflective surfaces causing inaccurate contour detection, determined based on fruit and vegetable types and error characteristics), or other factors (dirty camera lens, determined by regularly cleaning the camera and comparing error data before and after cleaning). The frequency of each type of error and its impact on grasping accuracy were analyzed.

[0274] In subsequent crawling tasks, relevant information (fruit and vegetable types, environmental parameters, etc.) is read in real time, and the weights are adjusted in real time according to the rule base. The adjusted weights are then used in the weighted average calculation of the coordinates, and the weighted average is calculated according to the weighted average formula.

[0275]

[0276] The weighted average (X, Y) of the fruits and vegetables in the x and y directions is recalculated, and the new weighted average is transmitted to the grasping execution mechanism in real time to ensure that the grasping operation is based on more accurate coordinates. At the same time, the weight adjustment strategy is continuously optimized.

[0277] The system is converted to the world coordinate system to obtain the camera's extrinsic parameter matrix [R|T] (containing the rotation matrix R and translation vector T). This extrinsic parameter matrix is ​​typically obtained through pre-calibration work such as camera calibration and is stored in the robot's vision system configuration file or a parameter storage area that can be obtained in real time. Using the camera's extrinsic parameter matrix [R|T], the coordinates (X, Y) in the camera coordinate system (the result obtained from the weighted average calculation above) are converted to coordinates (X_w, Y_w) in the world coordinate system using the corresponding coordinate transformation formula. The specific coordinate transformation calculation process follows the mathematical principles of spatial coordinate transformation, such as through matrix multiplication, thereby transforming the coordinate position of the fruits and vegetables to the world coordinate system corresponding to the robot's actual workspace, facilitating accurate operation by the robotic arm based on these coordinates.

[0278] Determine the grab location

[0279] After completing the coordinate transformation described above, the target fruit and vegetable's complete three-dimensional coordinate position (X_w, Y_w, Z_w) is determined by combining depth information estimation (which can be obtained from the depth sensor in the robot vision system or by obtaining the corresponding depth value Z_w through an image-based depth estimation algorithm). Based on factors such as the type and shape of the fruit and vegetable, and the robotic arm's grasping characteristics, the most suitable grasping position is comprehensively considered and determined. For example, for spherical fruits and vegetables, the grasping position might be chosen near their center; for long, thin fruits and vegetables, the grasping position might be chosen at their center of gravity, etc., providing accurate target position information for the subsequent precise grasping operation of the robotic arm.

[0280] Overall detection and processing cycle

[0281] For the same target fruit and vegetable, the above weighted average algorithm is continuously applied to process the results of 10 detections. This includes removing outliers, calculating the weighted average coordinates, transforming the coordinate system, and determining the grasping position. This ensures that the target position information used by the robotic arm to perform the grasping action each time is optimized and as accurate as possible, thereby improving the accuracy and success rate of the entire harvesting process.

[0282] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made to the present invention should be included within the scope of protection of the present invention.

Claims

1. A fruit and vegetable harvesting robot system based on data augmentation and optimized recognition accuracy, characterized in that: include Visual recognition module: Equipped with a high-resolution camera, it is used to acquire fruit and vegetable images in real time and to identify and locate fruits and vegetables in real time using the YOLOv5 target detection algorithm. It can perform high-precision classification and real-time detection of fruits and vegetables and maintain stable performance in complex backgrounds. Harvesting robotic arms: used to perform fruit and vegetable harvesting operations, including multi-degree-of-freedom robotic arms and flexible end grippers, which can achieve precise grasping and reduce damage to fruits and vegetables; Control module: Connects the vision recognition module and the picking robotic arm, and is used to process recognition data, generate picking path planning and robotic arm control instructions; Mobile module: Used for the autonomous movement of the robot body in the farmland, supporting obstacle avoidance and path planning to ensure that the robot can efficiently complete multi-point harvesting tasks; The visual recognition module uses the YOLOv5 object detection algorithm to identify and locate fruits and vegetables in real time. This is achieved by building a recognition and grasping system model, including the following steps: Step 1: Image Acquisition: Collect images of the target fruits and vegetables as a dataset through outdoor photography; Step 2, Preprocessing: The acquired images are preprocessed, including noise reduction and filtering, to improve image quality; Step 3: Expand the dataset: Expand the dataset in a targeted manner to cope with different influencing factors. The specific method is as follows: expand the dataset by simulating different lighting conditions and adjusting the brightness, contrast, saturation and hue of the image. Use random parameter adjustment to perform color transformation on the image to generate images under different lighting and color conditions. By changing the image's perspective to simulate different shooting angles, the perspective transformation matrix is ​​adjusted in multi-scale perspective transformation. The parameters are used to apply random scaling factors to the elements of the perspective transformation matrix. Multiple perspective transformations at different scales are applied to the same image, with N scales. Then the... The perspective transformation matrix at each scale is represented as follows: ； in and It is the first The scaling factor at each scale represents the scaling of the image in the horizontal and vertical directions; , It is a clipping parameter that indicates a slight tilt of the image along the diagonal direction; , These are translation parameters, representing the translation of the image in the horizontal and vertical directions; , These are perspective parameters, representing the changes in an image when simulating objects at different distances. For each scale i, the transformed coordinates (x', y') of each pixel (x, y) in the image are calculated using the following formula: ； The final transformation result needs to be normalized: = = ； Introduce more data diversity during the training process; Step 4: Dataset labeling; Step 5: Establish the recognition model; Step Six: Model Conversion and Deployment; Step 7: Target Positioning; Step 8: Depth estimation for targets in the three-dimensional world; Step 9: Optimize lens distortion; Step 10: Introduce a weighted average algorithm to improve accuracy.

2. The fruit and vegetable harvesting robot system based on data augmentation and recognition accuracy optimization according to claim 1, characterized in that: Images of fruits and vegetables are used to simulate different weather conditions. This simulation of haze weather uses Gaussian blur to mimic the blurring effect caused by haze on an image; let the original image be... Gaussian kernel is ,in To control the degree of blur, the convolution operation is as follows: ； in, It is a two-dimensional Gaussian kernel: ； Overlay a haze effect with transparency onto the blurred image; let the haze transparency be... If the value is between 0 and 1, the synthesized image is: ； in, The closer the concentration of smog is to 1, the stronger the smog effect. The rain weather simulation uses randomly generated transparent circular water droplets to simulate the effect of raindrops. A water droplet layer R(x, y) is defined, consisting of n randomly generated water droplets. The shape of each water droplet is simulated using a Gaussian function, and the size and position of the droplets are randomized. For each water droplet, its position is set to... The standard deviation is The generated water droplets are shaped as follows: ； in, It is a two-dimensional Gaussian distribution function used to simulate the shape of a water droplet; To simulate low light conditions on a rainy day, the overall brightness of the image is reduced; let the brightness adjustment factor be... The adjusted rainy day image is as follows: ； in, It is the intensity coefficient of the water droplet layer, which controls the visibility of the raindrops; In the snow weather simulation, the distribution of snowflakes is achieved by randomly generated small white dots. Each snowflake is represented as a small Gaussian function, with its position and size being random. Let the snowflake layer be S(x, y), which consists of n snowflakes, and the position of each snowflake is... The standard deviation is ,but: ； Slightly adjust the contrast of the image to enhance the contrast between the snowflakes and the background; assume the brightness adjustment factor is [value missing]. The final image of the snowy day would then be: ； in, Controlling the intensity of snowflakes, Control the overall brightness; The simulation of a sunny day increases the vibrancy of the image by improving brightness, contrast, and color saturation. The brightness adjustment factor is set to... The adjusted image is: ； By enhancing image contrast, the sense of depth in the image is increased; the contrast factor is... , The enhanced image is: ； in, It represents the average brightness of the original image, ensuring that the average brightness of the enhanced image remains unchanged. The colors of an image are enhanced by increasing its color saturation; the process of saturation enhancement is achieved using the following formula: ； in, It is the saturation factor. .

3. The fruit and vegetable harvesting robot system based on data augmentation and recognition accuracy optimization according to claim 1, characterized in that: Step nine describes a method for optimizing lens distortion and correcting lens errors, employing a segmented correction approach. (1) Regional division: Divide the lens's field of view into a central region and an edge region; Central area The distortion is small, so a distortion model is used for correction. Edge area Large distortion was corrected using manual sampling and interpolation. (2) Manual sampling: Several sampling points were selected in 1 / 4 of the lens's field of view, and the deviation between the uncorrected coordinates and the actual coordinates was recorded. (3) Interpolation fitting: Use bilinear interpolation or polynomial fitting to generate a corrected mapping table; ； ； in , These are the fitting coefficients; (4) Symmetry optimization: By utilizing the symmetry of the rectangular field of view of the lens, only the correction parameters of 1 / 4 of the region need to be calibrated, and the mapping table is reused in the remaining regions through coordinate transformation (x->-x, y->-y); Edge area: ； ； in and This is the correction amount obtained through interpolation.

Images