A fruit posture adaptive picking method, system and device based on binocular vision

Abstract

This invention relates to the field of intelligent agricultural robot technology, specifically to a fruit posture adaptation harvesting method, system, and equipment based on binocular vision. It avoids the high breakage rate problem caused by existing technologies that rely on fixed postures or simple path planning for harvesting. By accurately sensing the fruit's growth posture, dynamically calculating the target posture of the end effector, and adjusting it in real time, it can adapt to fruits in different growth stages without human intervention, reducing labor intensity and costs. It ensures precise alignment between the fruit and the harvesting direction of the end effector, achieving a harvesting success rate of over 95%, avoiding the poor adaptability of fixed-posture harvesting. Simultaneously, during harvesting, the harvesting force is applied along the physiological stress direction of the fruit stem, without lateral compression, pulling, or reciprocating vibration, thereby reducing the fresh fruit breakage rate to below 3%, meeting practical operational needs.

Description

Technical Field

[0001] This invention relates to the field of intelligent agricultural robot technology, specifically to a fruit posture adaptation harvesting method, system, and equipment based on binocular vision. Background Technology

[0002] Goji berries are an important economic crop with wide applications in food, medicine, and health products, and market demand continues to grow. Most goji berry cultivation areas are located in the Northwest, in regions with strong ultraviolet radiation and intense sunlight. Moreover, the large-scale goji berry cultivation areas make the labor intensity of workers during the goji berry harvesting process extremely high.

[0003] Goji berry harvesting primarily employs traditional mechanical methods such as vibration, brushing, vibration-brushing combination, and air wave harvesting. However, these methods suffer from high impurity rates and fresh fruit breakage rates. In some scenarios, single-fruit harvesting robots are being tested to obtain the spatial coordinates of the fruit. However, the end effector uses a fixed clamping posture (such as vertical or horizontal clamping) or a simple straight-line approach, which easily leads to collisions with surrounding branches and adjacent fruits, causing additional damage and reducing harvesting success rates. When the fruit is in a random growth state, such as tilting or drooping, the harvesting direction deviates from the fruit's physiological growth direction, generating lateral shear force or compressive stress, resulting in fruit stalk breakage and peel damage. Therefore, both types of technologies have significant drawbacks, making it difficult to balance harvesting efficiency with the integrity of fresh fruit. Summary of the Invention

[0004] The purpose of this invention is to provide a fruit posture adaptation harvesting method, system and equipment based on binocular vision, to solve the technical problem of high fruit harvesting damage rate in the prior art.

[0005] The solution of the present invention to the above-mentioned technical problems is as follows: A fruit posture adaptation harvesting method based on binocular vision includes the following steps: S1. Acquire a pair of synchronized images containing the target fruit using a binocular camera, perform stereo matching and 3D reconstruction on the synchronized image pair, and generate 3D point cloud data containing the fruit. S2. Identify the fruits in the synchronized image pair and determine the two-dimensional coordinates of the center point of the corresponding fruit in the image. Two-dimensional coordinates of the fruit stem attachment point ; S3. Based on the two-dimensional coordinates of the center point of the fruit. Two-dimensional coordinates of the fruit stem attachment point Based on the aforementioned 3D point cloud data, the 3D coordinates of the fruit's center point in the binocular camera coordinate system are determined. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system ; S4. Based on the three-dimensional coordinates of the center point of the fruit in the binocular camera coordinate system. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Determine the fruit's growth posture vector ; S5. Vectorize the fruit's growth posture. Transform from camera coordinate system to robot arm base coordinate system and based on the robotic arm's coordinate system Reverse direction vector As an end effector, it is positioned in the direction of approach to the target at the picking point; S6. Define the preset approach axis direction of the end effector tool coordinate system as a unit vector. Calculation makes Rotate to Alignment required rotation matrix and the rotation matrix Converted into the end effector angle used to control the robotic arm; S7. Calculate the three-dimensional coordinates of the fruit stem attachment point in the binocular camera coordinate system. Transform to three-dimensional coordinates in the robot arm's base coordinate system Based on the three-dimensional coordinates of the fruit stem attachment point in the robotic arm's base coordinate system Calculate the target position of the end effector tool center point ; S8. Based on the obtained target position The end effector's attitude angle is used to control the robotic arm to move the end effector to the target pose. S9. Control the end effector to harvest the fruit.

[0006] Further specifying, the acquisition of synchronized image pairs containing the target fruit specifically refers to: The robotic arm moves the binocular camera to the attitude observation position to determine whether there is fruit. If there is, it acquires a synchronized image pair containing the target fruit; if not, it adjusts the attitude observation position and repeats this step. The stereo matching and 3D reconstruction of the synchronized image pairs includes the following steps: Use the OpenCV library to perform Gaussian filtering to denoise the synchronized image pairs; Image distortion correction is performed on synchronized image pairs based on pre-calibrated intrinsic and extrinsic parameters; Run the SGBM stereo matching algorithm, set the matching window size and disparity search range, and generate a pixel-level disparity map; The 3D depth information of each pixel is calculated using a parallax-depth conversion model; the 3D depth information is then used to reconstruct 3D point cloud data containing the fruit.

[0007] Further specifying, the identification of fruits in the synchronized image pair includes the following steps: Based on the pre-trained YOLOv8 instance segmentation model, target detection is performed on the synchronous image pairs after image distortion correction to select the fruits to be picked. The growth posture vector .

[0008] Further specifying, the vector of the fruit's growth posture Transform from camera coordinate system to robot arm base coordinate system Specifically:

[0009] in, and These are the rotation matrix and translation vector from the camera coordinate system to the tool coordinate system, obtained from hand-eye calibration, respectively. and Let the rotation matrix and translation vector be the distance from the tool coordinate system to the robot arm base coordinate system; The end effector is in the target approach direction at the picking point. .

[0010] Further specifying, the calculation makes Rotate to Alignment required rotation matrix Includes the following steps: Calculate the axis of rotation ; Calculate the rotation angle ; Constructing a rotation axis antisymmetric matrix ; Calculate the rotation matrix using Rodriguez's formula. ,in It is a 3-order identity matrix; The rotation matrix Specifically, this is converted into the end effector angle for controlling the robotic arm's attitude angle:

[0011]

[0012]

[0013] in, Rotation matrix No. Line number Column elements, , and These are the rotation angles around the X-axis, Y-axis, and Z-axis of the tool coordinate system, respectively.

[0014] Further defining the target position of the center point of the end effector tool. , To safely approach the distance.

[0015] Further specifying, controlling the end effector to harvest fruit specifically includes the following steps: The end effector performs the picking action; Determine whether the end effector has picked the fruit. If yes, the process ends; otherwise, control the end effector to pick the fruit again.

[0016] Furthermore, the fruit posture adaptation and harvesting method based on binocular vision also includes the following steps: The robotic arm is controlled to move the end effector, which has been holding the fruit, to the predetermined collection point; The end effector is controlled to perform a release action, releasing the fruit into the collection container to complete a single harvesting operation.

[0017] A binocular vision-based fruit posture adaptation harvesting system, used to implement the aforementioned binocular vision-based fruit posture adaptation harvesting method, includes: The image acquisition and processing module is used to acquire a pair of synchronized images containing the target fruit, perform stereo matching and three-dimensional reconstruction on the synchronized image pair, and generate three-dimensional point cloud data containing the fruit. The target recognition and pose feature extraction module is used to receive and identify fruits in synchronized image pairs, and determine the two-dimensional coordinates of the center point of the corresponding fruit in the image. Two-dimensional coordinates of the fruit stem attachment point Used to determine the two-dimensional coordinates of the center point of the fruit. Two-dimensional coordinates of the fruit stem attachment point Based on the aforementioned 3D point cloud data, the 3D coordinates of the fruit's center point in the binocular camera coordinate system are determined. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Used to determine the three-dimensional coordinates of the fruit's center point in the stereo camera coordinate system. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Determine the fruit's growth posture vector ; The coordinate transformation and attitude correction calculation module converts the fruit's growth attitude vector... Transform from camera coordinate system to robot arm base coordinate system and based on the robotic arm's coordinate system Reverse direction vector The target approach direction of the end effector at the picking point; the preset approach axis direction used to define the tool coordinate system of the end effector is a unit vector. Calculation makes Rotate to Alignment required rotation matrix and the rotation matrix Converted into the end effector angle for controlling the robotic arm; used to determine the three-dimensional coordinates of the fruit stalk attachment point in the binocular camera coordinate system. Transform to three-dimensional coordinates in the robot arm's base coordinate system Based on the three-dimensional coordinates of the fruit stem attachment point in the robotic arm's base coordinate system Calculate the target position of the end effector tool center point ; The robotic arm motion control module is used to control the motion based on the obtained target position. The end effector's attitude angle is used to control the robotic arm to move the end effector to the target pose. An end effector control module is used to control the end effector to complete fruit picking.

[0018] A fruit posture adaptation picking device based on binocular vision, based on the above-mentioned fruit posture adaptation picking system based on binocular vision, includes a robotic arm, a binocular camera, an end effector and a host computer. The robotic arm, the binocular camera and the end effector are all communicatively connected to the host computer. The end effector and the binocular camera are both located at the end of the robotic arm. The binocular camera is located above the end effector and faces the picking direction. The host computer is used to execute the image acquisition and processing module, the target recognition and posture feature extraction module, the coordinate transformation and posture correction calculation module, the robotic arm motion control module, and the end effector control module.

[0019] The beneficial effects of this invention are as follows: 1. This invention avoids the problem of high breakage rates caused by existing technologies that rely on fixed postures or simple path planning for harvesting. By accurately sensing the fruit's growth posture, dynamically calculating the target posture of the end effector and adjusting it in real time, it can adapt to fruits in different growth stages without manual intervention, reducing labor intensity and operating costs. It ensures precise alignment between the fruit and the harvesting direction of the end effector, achieving a harvesting success rate of over 95% and avoiding the poor adaptability of fixed posture harvesting. At the same time, during harvesting, the harvesting force is applied along the physiological stress direction of the fruit stalk, without lateral compression, pulling, or reciprocating vibration, thereby reducing the fresh fruit breakage rate to below 3% and meeting the needs of actual operations.

[0020] 2. This invention consists of a system composed of five major software modules: image acquisition and processing, target recognition and posture feature extraction, coordinate transformation and posture correction calculation, robotic arm motion control, and end effector control. It collaboratively completes the entire closed-loop operation from data acquisition to instruction execution, making the system stable and responsive in real time, improving the quality and success rate of harvesting, and effectively improving harvesting efficiency with its fully automated and all-weather operation capabilities. Attached Figure Description

[0021] Figure 1 This is a step diagram of the fruit posture adaptation and harvesting method based on binocular vision according to the present invention. Figure 2 This is a diagram illustrating the key points and spatial orientation vector detection effect of wolfberry fruit according to the present invention. Figure 3 This is a schematic diagram illustrating the principle of reverse vector alignment between the end effector posture and the fruit growth posture of the present invention. Figure 4 This is a schematic diagram of the fruit posture adaptation and harvesting method based on binocular vision according to the present invention. Figure 5 This is a diagram of the fruit posture adaptation and harvesting system based on binocular vision according to the present invention. Figure 6 This is a diagram illustrating the working state of the fruit posture adaptation harvesting device based on binocular vision according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0025] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] It should be noted that this invention is for fruits with random growth postures such as upright, tilted, and drooping, especially fruits with thin skin, abundant juice, and fragile stems, such as goji berries.

[0027] Vibratory harvesters use high-frequency vibration to drive the fruit branches to resonate, causing the goji berries to detach from the branches. Although the harvesting efficiency can reach 100-150 kg / h, its core drawback is that the vibration energy is indiscriminately transmitted to the fruit branches, flower buds, and fruits. This not only easily causes branch breakage and flower bud drop, directly affecting the yield of the next season, but also leads to violent collisions between fruits and branches, and between fruits themselves. At the same time, it carries a large number of leaves, twigs, and other impurities (the impurity ratio can reach 15%-20%), which increases the subsequent sorting costs and results in a fresh fruit damage rate as high as 20%-40%. Although air-wave harvesters can reduce mechanical damage to the fruit branches, the method of using high-pressure airflow to impact the fruit and cause it to fall to the ground, resulting in secondary collisions, is not effective in harvesting low-hanging, densely growing fruits, and the problem of impurity entrainment is also prominent.

[0028] The comb-type harvester uses rotating comb teeth to "scrape" the fruit off the branches. This method causes more direct mechanical damage to the fruit: the hard contact between the comb teeth and the fruit skin can easily scratch the skin (which is only 0.1-0.2mm thick) and squeeze the pulp (which has a water content of over 85%), resulting in a fresh fruit breakage rate as high as 30%-50%, significantly reducing the marketable yield. In addition, the comb teeth can easily get tangled in the branches, causing damage to the branches and further limiting its application scenarios. Although the vibration-comb combination balances efficiency and damage to some extent, it is still essentially a "forced separation" logic and has not solved the problem of incompatible contact between the fruit and the mechanical parts. The breakage rate and impurity entrainment rate remain at a high level of 15%-25%.

[0029] While single-fruit harvesting robots have broken through the traditional mechanical logic of "mass forced detachment" and can locate individual fruits through visual recognition, they still have three major weaknesses: First, they lack the ability to perceive the fruit's growth posture, only acquiring the spatial coordinates of the fruit and failing to identify the growth orientation of the "stem-fruit body." The end effector uses a fixed clamping posture (such as vertical or horizontal clamping) or a simple straight-line approach. When the fruit is in a random growth state such as tilting or drooping, the harvesting direction deviates from the physiological growth direction of the fruit, generating lateral shear force or compressive stress, leading to stem breakage and peel damage, with a breakage rate as high as 15%-25%. Second, the path planning lacks flexibility, failing to dynamically adjust the movement trajectory based on the fruit's posture, making it prone to collisions with surrounding branches and adjacent fruits, causing additional damage and reducing the harvesting success rate. Third, the recognition and positioning accuracy is limited, with insufficient differentiation between mature, green, and semi-ripe fruits, and difficulty in accurately locating the stem attachment point, resulting in clamping the entire fruit body during harvesting, further exacerbating the risk of breakage.

[0030] Therefore, the common deficiency of existing technologies lies in their failure to address the core requirement of "harvesting direction matching growth posture" for goji berries, which are characterized by "thin skin, abundant juice, fragile fruit stems, and random growth posture." Whether it's the "forced detachment" of traditional machinery or the "position-guided harvesting" of single-fruit robots, neither can fundamentally prevent damage to the fruit from mismatched external forces, resulting in a persistently high rate of fresh fruit breakage and failing to meet the needs of the rapidly developing goji berry industry. Therefore, developing an efficient, low-damage goji berry harvesting equipment, system, and method has significant practical importance and market value.

[0031] Example 1 refer to Figure 1 This invention provides a fruit posture adaptation harvesting method based on binocular vision, comprising the following steps: S1. Acquire a pair of synchronized images containing the target fruit using a binocular camera, perform stereo matching and 3D reconstruction on the synchronized image pairs, and generate 3D point cloud data containing the fruit. S2. Identify the fruits in the synchronized image pair and determine the two-dimensional coordinates of the center point of the corresponding fruit in the image. Two-dimensional coordinates of the fruit stem attachment point ; S3. Based on the two-dimensional coordinates of the center point of the fruit. Two-dimensional coordinates of the fruit stem attachment point By combining 3D point cloud data, the 3D coordinates of the center point of the fruit in the binocular camera coordinate system were determined. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system ; S4. Based on the three-dimensional coordinates of the center point of the fruit in the binocular camera coordinate system. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Determine the fruit's growth posture vector ; S5. Vectorize the fruit's growth posture. Transform from camera coordinate system to robot arm base coordinate system and based on the robotic arm's coordinate system Reverse direction vector As an end effector, it is positioned in the direction of approach to the target at the picking point; S6. Define the preset approach axis direction of the end effector tool coordinate system as a unit vector. Calculation makes Rotate to Alignment required rotation matrix and the rotation matrix Converted into the end effector angle used to control the robotic arm; S7. Calculate the three-dimensional coordinates of the fruit stem attachment point in the binocular camera coordinate system. Transform to three-dimensional coordinates in the robot arm's base coordinate system Based on the three-dimensional coordinates of the fruit stem attachment point in the robotic arm's base coordinate system Calculate the target position of the end effector tool center point ; S8. Based on the obtained target location The end effector's attitude angle is used to control the robotic arm to move the end effector to the target pose. S9. Control the end effector to harvest the fruit.

[0032] To further explain, acquiring synchronized image pairs containing the target fruit specifically involves: After the robotic arm moves the binocular camera to the recognition posture, it uses the binocular camera to capture images of the current position to determine whether there is fruit. If so, it acquires a synchronized image pair containing the target fruit; otherwise, it needs to adjust the posture observation position and repeat this step until there is fruit, at which point it will begin preparations for picking.

[0033] To further explain, stereo matching and 3D reconstruction of synchronized image pairs includes the following steps: The OpenCV library is used to perform 5×5 window Gaussian filtering for noise reduction on the synchronized image pairs. Based on pre-calibrated intrinsic parameters (focal length) and principal point coordinates ) and external parameters (baseline distance) Image distortion correction is performed on synchronized image pairs using rotation and translation matrices. Run the SGBM stereo matching algorithm, set the matching window size to 11×11 and the disparity search range to 0-64 pixels, and generate a pixel-level disparity map; Through the parallax-depth conversion model Calculate the 3D depth information for each pixel. The difference is the horizontal coordinate of the same pixel; using 3D depth information, 3D point cloud data containing the fruit is reconstructed, where invalid points with depth values ​​<200mm or >1200mm are filtered out.

[0034] The pre-calibrated intrinsic and extrinsic parameters are determined using a checkerboard calibration method. The black and white checkerboard calibration board has 12×9 interior corner points (12 horizontal and 9 vertical), with each checkerboard square having a physical size of 15mm×15mm (precise dimensions, no distortion). The program calls functions in OpenCV, taking the world coordinates of the interior corner points, their corresponding pixel coordinates, and the image size (1280, 720) as input, and outputting the intrinsic parameter matrix and distortion coefficients. The intrinsic parameter matrix contains... , Principal point coordinates , The matrix form is The distortion coefficients include four radial distortion coefficients. , , and and two tangential distortion coefficients and radial distortion , Tangential distortion: , ,in and The original, distortion-free, normalized coordinates. Finally, the radial and tangential distortions are superimposed to obtain the total distortion result, thus completing the image distortion correction for the synchronized image pair.

[0035] To further explain, identifying the fruit in a synchronized image pair includes the following steps: Based on the pre-trained YOLOv8 instance segmentation model, target detection is performed on the color synchronized image pairs after image distortion correction to select the fruits to be picked.

[0036] Among these criteria, the fruits to be picked need to be determined based on standards such as color or size to avoid picking unripe fruits, semi-ripe fruits, and impurities, with a confidence level of ≥0.6, thereby improving the reliability of picking.

[0037] refer to Figure 2The training dataset for the YOLOv8 model uses images of goji berries taken in the field. The camera used is the same binocular camera as the system. The images are manually labeled with LabelMe (only mature goji berries are labeled) in JSON format. Goji berries are labeled with rectangles and the stems are labeled with key points.

[0038] To further explain, in the image coordinate system {P}, the stem point... The center point of the fruit Based on the pre-calibrated camera parameters and the pixel-level depth information calculated by the SGBM stereo matching algorithm. and These two points can be transformed from the image coordinate system {P} to the camera coordinate system {C}:

[0039]

[0040] Obtain the center point of the fruit , fruity point ; Growth posture vector This vector represents the real-time growth posture of the fruit.

[0041] To further explain, the growth posture vector of the fruit... Transform from camera coordinate system to robot arm base coordinate system Specifically: Load the pre-calibrated hand-eye transformation matrix (rotation matrix) +Translation vector ), to grow posture vector Transform from the camera coordinate system {C} to the robotic arm end-effector coordinate system {T}. ,in Let be a rotation matrix. For translation vectors:

[0042]

[0043] The coordinates of two key points of the target fruit in the tool coordinate system {T} are obtained, and then the coordinates of the two key points are transformed to the base coordinate system {B} of the robotic arm. The transformation process is based on the rotation matrix between the tool coordinate system {T} and the robotic arm base coordinate system {B}, which is determined by the six-point method. Translation vector Perform the conversion:

[0044]

[0045] Calculate and normalize the fruit pose vector:

[0046] Right now,

[0047] in, and Let be the rotation matrix and translation vector from the tool coordinate system to the robot arm base coordinate system.

[0048] refer to Figure 3 ,Pick The opposite direction vector =- That is, the picking direction vector from the center point of the fruit body to the attachment point of the fruit stem in the base coordinate system, defined as follows: The clamping direction of the end effector is preset to the negative X-axis of the tool coordinate system.

[0049] To further explain, the calculation makes Rotate to Alignment required rotation matrix Includes the following steps: Calculate the current direction Rotate to the target direction Required: Rotation axis Rotation angle ; make , ,but ; ; Constructing a rotation axis antisymmetric matrix ; Calculate the current direction using the Rodriguez formula. Rotate to the target direction rotation matrix ,in It is a 3-order identity matrix; Solving the rotation matrix According to "X-axis correspondence" Y-axis corresponding and the corresponding Z-axis The correspondence between "" and "" will be used to rotate the matrix. Specifically, this is converted into the end effector angle used to control the robotic arm's attitude angle:

[0050]

[0051]

[0052] in, Rotation matrix No. Line number Column elements, , and These represent the rotation angles around the X, Y, and Z axes of the tool coordinate system, respectively.

[0053] After obtaining the end-effector posture angle, it is necessary to check whether it is within the allowable range of the robotic arm joint movement. If so, the robotic arm motion planning is successful and continues; otherwise, the robotic arm recognition posture is readjusted and a new synchronous image pair containing the target fruit is acquired.

[0054] To further explain, to ensure a smooth approach from the fruit's tail, the target position of the end effector tool center point (TCP) is set behind the stem point (along...). (Direction) A preset safe approach distance (For example, at 10mm), the target position of the end effector tool center point. .

[0055] refer to Figure 4 To further explain, controlling the end effector to harvest fruit specifically includes the following steps: Based on the calculated three Euler angles The control end effector moves smoothly in the opposite direction of the fruit's growth posture in the adapted posture, and triggers the picking action when it reaches the vicinity of the fruit stem attachment point, thus completing the harvesting of fresh wolfberry fruit. After harvesting, it is necessary to determine whether the end effector has harvested the fruit. If so, the process ends; otherwise, the end effector is controlled to harvest the fruit again.

[0056] To further explain, the fruit posture adaptation harvesting method based on binocular vision also includes the following steps: The robotic arm is controlled to move the end effector, which has already gripped the fruit, to the designated collection point; The end effector is controlled to perform a release action, releasing the fruit into the collection container to complete a single harvesting operation; This process is repeated until all the fruit to be picked has been harvested.

[0057] By using binocular vision to perceive the fruit's growth posture in real time, and combining it with the Rodriguez formula, the end effector is precisely aligned with the picking direction (alignment accuracy ≤2°). The picking force is applied along the physiological stress direction of the fruit stalk, without lateral squeezing or pulling. Actual measurements show that the fresh fruit breakage rate is reduced to below 3%, far lower than the 15%-40% of existing mechanical picking and the 15%-25% of traditional single-fruit robots. For random growth postures of goji berries, such as upright, tilted, and drooping, the system dynamically calculates the target posture of the end effector and adjusts it in real time. It can adapt to fruits in different growth states without human intervention, with a picking success rate of ≥95%. In the single-fruit precision picking mode, the stable efficiency reaches 10 fruits / minute. Although the efficiency per minute is far lower than the speed of manual single-fruit picking, the equipment supports 24-hour continuous operation without fatigue or rest requirements. It can stably complete about 14,400 picking tasks per day (24 hours), meeting operational needs.

[0058] Example 2 refer to Figure 5 This embodiment provides a fruit posture adaptation harvesting system based on binocular vision, used to implement the fruit posture adaptation harvesting method based on binocular vision described in Embodiment 1, including: The image acquisition and processing module is used to acquire synchronous image pairs containing the target fruit, perform stereo matching and 3D reconstruction on the synchronous image pairs, and generate 3D point cloud data containing the fruit. The target recognition and pose feature extraction module is used to receive and identify fruits in synchronized image pairs, and determine the two-dimensional coordinates of the center point of the corresponding fruit in the image. Two-dimensional coordinates of the fruit stem attachment point Used to determine the two-dimensional coordinates of the center point of the fruit. Two-dimensional coordinates of the fruit stem attachment point By combining 3D point cloud data, the 3D coordinates of the center point of the fruit in the binocular camera coordinate system were determined. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Used to determine the three-dimensional coordinates of the fruit's center point in the stereo camera coordinate system. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Determine the fruit's growth posture vector ; The coordinate transformation and attitude correction calculation module converts the fruit's growth attitude vector... Transform from camera coordinate system to robot arm base coordinate system and based on the robotic arm's coordinate system Reverse direction vector This serves as the target approach direction for the end effector at the picking point; the preset approach axis direction used to define the tool coordinate system of the end effector is a unit vector. Calculation makes Rotate to Alignment required rotation matrix and the rotation matrix Converted to the end effector attitude angle for controlling the robotic arm; used to determine the three-dimensional coordinates of the fruit stalk attachment point in the binocular camera coordinate system. Transform to three-dimensional coordinates in the robot arm's base coordinate system Based on the three-dimensional coordinates of the fruit stem attachment point in the robotic arm's base coordinate system Calculate the target position of the end effector tool center point ; The robotic arm motion control module is used to control the motion based on the obtained target position. The end effector's attitude angle is used to control the robotic arm to move the end effector to the target pose. The end effector control module is used to control the end effector to complete fruit picking.

[0059] Example 3 refer to Figure 6 Based on Embodiment 2, this embodiment provides a fruit posture adaptation picking device based on binocular vision, including a robotic arm, a binocular camera, an end effector, a host computer, a power supply, a base, and a support. The robotic arm is connected to the base via the support, and the base can be movable. The power supply is located on the base and provides power to the robotic arm, the binocular camera, the end effector, the host computer, and the base. The robotic arm, the binocular camera, and the end effector are all communicatively connected to the host computer. The end effector and the binocular camera are both located at the end of the robotic arm, with the binocular camera positioned above the end effector and facing the picking direction.

[0060] The host computer is used to execute the image acquisition and processing module, the target recognition and posture feature extraction module, the coordinate transformation and posture correction calculation module, the robotic arm motion control module, and the end effector control module.

[0061] The image acquisition and processing module receives synchronized image pairs transmitted from the binocular camera via a USB interface, and outputs denoised 3D point cloud data and corrected color images to the target recognition and pose feature extraction module.

[0062] The target recognition and pose feature extraction module receives the 3D point cloud data and corrected color image output by the image acquisition and processing module, and calculates the features based on the attachment point of the fruit stem. Pointing to the center of the fruit Three-dimensional space vector It is then output to the coordinate transformation and attitude correction solution module.

[0063] The coordinate transformation and attitude correction solution module receives three-dimensional spatial vectors. Processing yields the growth posture vector in the base coordinate system and the obtained target pose ( , , The output is sent to the robotic arm motion control module.

[0064] The robotic arm motion control module calls the "posture parameter configuration interface" of the secondary development of the robotic arm to set the target posture ( , , The input parameter is passed in; the secondary development of the "Cartesian space precision motion interface" is called, and the position parameter is used to drive the robotic arm to move the end effector and complete the posture adjustment. The real-time status feedback interface is called to monitor the return status code of the motion control interface. When the "execution completed" status code is returned, it is determined that the posture adjustment is completed and the picking action is triggered.

[0065] The end effector control module receives the picking trigger signal from the robotic arm motion control module and sends opening and closing control commands to the lower-level computer (Arduino UnoR3 control board) via serial port. After the commands are parsed by the Arduino UnoR3 control board, a precise PWM control signal is output to the servo driver PCA9685 through the I2C interface to drive the servo to rotate, which drives the linkage transmission structure to make the two arc-shaped gripping arms open and close synchronously, realizing the precise gripping and picking of goji berries.

[0066] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0067] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fruit posture-adaptive harvesting method based on binocular vision, characterized in that, Includes the following steps: S1. Acquire a pair of synchronized images containing the target fruit using a binocular camera, perform stereo matching and 3D reconstruction on the synchronized image pair, and generate 3D point cloud data containing the fruit. S2. Identify the fruits in the synchronized image pair and determine the two-dimensional coordinates of the center point of the corresponding fruit in the image. Two-dimensional coordinates of the fruit stem attachment point ; S3. Based on the two-dimensional coordinates of the center point of the fruit. Two-dimensional coordinates of the fruit stem attachment point Based on the aforementioned 3D point cloud data, the 3D coordinates of the fruit's center point in the binocular camera coordinate system are determined. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system ; S4. Based on the three-dimensional coordinates of the center point of the fruit in the binocular camera coordinate system. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Determine the fruit's growth posture vector ; S5. Vectorize the fruit's growth posture. Transform from camera coordinate system to robot arm base coordinate system and based on the robotic arm's coordinate system Reverse direction vector As an end effector, it is positioned in the direction of approach to the target at the picking point; S6. Define the preset approach axis direction of the end effector tool coordinate system as a unit vector. Calculation makes Rotate to Alignment required rotation matrix and the rotation matrix This is converted into the end effector angle used to control the robotic arm's attitude angle. S7. Calculate the three-dimensional coordinates of the fruit stem attachment point in the binocular camera coordinate system. Transformation to three-dimensional coordinates in the robot arm's base coordinate system Based on the three-dimensional coordinates of the fruit stem attachment point in the robotic arm's base coordinate system Calculate the target position of the end effector tool center point ; S8. Based on the obtained target position The end effector's attitude angle is used to control the robotic arm to move the end effector to the target pose. S9. Control the end effector to harvest the fruit.

2. The fruit posture adaptation and harvesting method based on binocular vision according to claim 1, characterized in that, The acquisition of synchronized image pairs containing the target fruit specifically involves: The robotic arm moves the binocular camera to the attitude observation position to determine whether there is fruit. If there is, it acquires a synchronized image pair containing the target fruit; if not, it adjusts the attitude observation position and repeats this step. The stereo matching and 3D reconstruction of the synchronized image pairs includes the following steps: Use the OpenCV library to perform Gaussian filtering to denoise the synchronized image pairs; Image distortion correction is performed on synchronized image pairs based on pre-calibrated intrinsic and extrinsic parameters; Run the SGBM stereo matching algorithm, set the matching window size and disparity search range, and generate a pixel-level disparity map; The 3D depth information of each pixel is calculated using a parallax-depth conversion model; the 3D depth information is then used to reconstruct 3D point cloud data containing the fruit.

3. The fruit posture adaptation and harvesting method based on binocular vision according to claim 2, characterized in that, The identification of fruits in synchronized image pairs includes the following steps: Based on the pre-trained YOLOv8 instance segmentation model, target detection is performed on the synchronous image pairs after image distortion correction to select the fruits to be picked. The growth posture vector .

4. The fruit posture adaptation and harvesting method based on binocular vision according to claim 1, characterized in that, The vector of fruit growth posture Transform from camera coordinate system to robot arm base coordinate system Specifically: in, and These are the rotation matrix and translation vector from the camera coordinate system to the tool coordinate system, obtained from hand-eye calibration, respectively. and Let the rotation matrix and translation vector be the distance from the tool coordinate system to the robot arm base coordinate system; The end effector is in the target approach direction at the picking point. .

5. The fruit posture adaptation and harvesting method based on binocular vision according to claim 1, characterized in that, The calculation makes Rotate to Alignment required rotation matrix Includes the following steps: Calculate the axis of rotation ; Calculate the rotation angle ; Constructing a rotation axis antisymmetric matrix ; Calculate the rotation matrix using Rodriguez's formula. ,in It is a 3-order identity matrix; The rotation matrix Specifically, this is converted into the end effector angle for controlling the robotic arm's attitude angle: in, Rotation matrix No. Line 1 Column elements, , and These are the rotation angles around the X-axis, Y-axis, and Z-axis of the tool coordinate system, respectively.

6. The fruit posture adaptation and harvesting method based on binocular vision according to claim 1, characterized in that, The target position of the center point of the end effector tool , To safely approach the distance.

7. The fruit posture adaptation and harvesting method based on binocular vision according to claim 1, characterized in that, The process of controlling the end effector to harvest fruit specifically includes the following steps: The end effector performs the picking action; Determine whether the end effector has picked the fruit. If yes, the process ends; otherwise, control the end effector to pick the fruit again.

8. The fruit posture adaptation and harvesting method based on binocular vision according to claim 1, characterized in that, The fruit posture adaptation harvesting method based on binocular vision also includes the following steps: The robotic arm is controlled to move the end effector, which has been holding the fruit, to the predetermined collection point; The end effector is controlled to perform a release action, releasing the fruit into the collection container to complete a single harvesting operation.

9. A fruit posture-adaptive harvesting system based on binocular vision, characterized in that, To implement the fruit posture adaptation and harvesting method based on binocular vision as described in any one of claims 1 to 8, the method includes: The image acquisition and processing module is used to acquire a pair of synchronized images containing the target fruit, perform stereo matching and three-dimensional reconstruction on the synchronized image pair, and generate three-dimensional point cloud data containing the fruit. The target recognition and pose feature extraction module is used to receive and identify fruits in synchronized image pairs, and determine the two-dimensional coordinates of the center point of the corresponding fruit in the image. Two-dimensional coordinates of the fruit stem attachment point Used to determine the two-dimensional coordinates of the center point of the fruit. Two-dimensional coordinates of the fruit stem attachment point Based on the aforementioned 3D point cloud data, the 3D coordinates of the fruit's center point in the binocular camera coordinate system are determined. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Used to determine the three-dimensional coordinates of the fruit's center point in the stereo camera coordinate system. The three-dimensional coordinates of the fruit stem attachment point in the stereo camera coordinate system Determine the fruit's growth posture vector ; The coordinate transformation and attitude correction calculation module converts the fruit's growth attitude vector... Transform from camera coordinate system to robot arm base coordinate system and based on the robotic arm's coordinate system Reverse direction vector The target approach direction of the end effector at the picking point; the preset approach axis direction used to define the tool coordinate system of the end effector is a unit vector. Calculation makes Rotate to Alignment required rotation matrix and the rotation matrix Converted into the end effector angle for controlling the robotic arm; used to determine the three-dimensional coordinates of the fruit stalk attachment point in the binocular camera coordinate system. Transform to three-dimensional coordinates in the robot arm's base coordinate system Based on the three-dimensional coordinates of the fruit stem attachment point in the robotic arm's base coordinate system Calculate the target position of the end effector tool center point ; The robotic arm motion control module is used to control the motion based on the obtained target position. The end effector's attitude angle is used to control the robotic arm to move the end effector to the target pose. An end effector control module is used to control the end effector to complete fruit picking.

10. A fruit posture-adaptive harvesting device based on binocular vision, characterized in that, The fruit posture adaptation and picking system based on binocular vision as described in claim 9 includes a robotic arm, a binocular camera, an end effector, and a host computer. The robotic arm, the binocular camera, and the end effector are all communicatively connected to the host computer. The end effector and the binocular camera are both located at the end of the robotic arm. The binocular camera is located above the end effector and faces the picking direction. The host computer is used to execute the image acquisition and processing module, the target recognition and posture feature extraction module, the coordinate transformation and posture correction calculation module, the robotic arm motion control module, and the end effector control module.

Images