Lightweight multi-modal tea garden picking robot control system

Abstract

The application discloses a kind of lightweight multi-modal tea garden picking robot control systems, and the application relates to robot control system technical field.The lightweight multi-modal tea garden picking robot control system, including parameter loading module, tea leaf identification judging unit, tea leaf movement prediction module, picking direction planning module, robot motion control module;Tea leaf identification judging unit comprehensively uses image acquisition, pretreatment and computer vision algorithm, accurately identifies tea leaf and judges tender bud maturity.Tea leaf movement prediction module, when wind starts, uses relevant algorithm to predict the stable position of tender bud, to ensure that the cutting basis is reliable.Picking direction planning module under the condition of wind, through sensor monitoring branch sway, calculates relevant parameters and plans cutting path and direction, improves cutting accuracy.Robot motion control module according to the planning result, combined with kinematic model generates joint trajectory and speed curve, can also adjust the angle of scissors, realize smooth and accurate picking and collection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control systems, in particular to a lightweight multi-modal tea garden picking robot control system. BACKGROUND

[0002] The tea garden picking robot is a modern agricultural equipment integrating machine vision, intelligent algorithm and precision mechanical arm, which can autonomously navigate and locate in a complex tea garden environment, realize non-destructive picking through bionic fingers or flexible end effectors, greatly improve the picking efficiency and standardization level, reduce the labor cost, and help the intelligent upgrading of the tea industry.

[0003] Publication No. CN119526383A discloses a tea leaf tender bud picking robot intelligent control system and method, which acquires scene image information through a visual analysis module, accurately identifies the tea leaf tender bud area to be picked based on a preset neural network model, divides the tea leaf tender bud area through a task decomposition module, actually allocates each task in a large number of picking tasks to the corresponding mechanical arm, and formulates the shortest safe working route of the mechanical arm; through the motion control module, the picking range of each mechanical arm of the robot can completely cover the area where the tea leaf tender bud is picked, effectively reducing the missed picking rate of the tea leaf tender bud.

[0004] As shown in the prior art above, the current tea leaf picking robot can realize the identification and picking of tea leaf tender buds, but there are still some problems in actual application, for example, due to exposure to outdoor environment, the tea tree branches at the end will sway in windy weather, and the system for identifying and picking tea leaf tender buds in the prior art is for processing tea leaf tender buds in a static state, how to accurately position and accurately pick in a dynamic state and reduce the error rate is still a problem to be solved. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a lightweight multi-modal tea garden picking robot control system, which solves the above-mentioned problems of the prior art.

[0006] To achieve the above purpose, the present application is implemented by the following technical scheme: a lightweight multi-modal tea garden picking robot control system, comprising:

[0007] Parameter loading module: loading the preset parameters of the tea garden environment, tea tree variety, tea leaf appearance characteristics, robot motion and picking strategy from the storage device;

[0008] Tea leaf recognition and judgment unit: using the robot camera to collect tea garden images in real time and pre-processing, using computer vision algorithm to extract leaf shape and color features, determining whether it is the tea leaf to be picked, and then judging the maturity according to the preset corresponding relationship;

[0009] Tea leaf motion prediction module: Real-time positioning of mature tea leaves is achieved through image processing and target tracking algorithms. When the wind blows, the motion prediction algorithm is used to analyze the motion state of the tea buds and predict the stable position of the buds at the moment of shearing.

[0010] Harvesting Direction Planning Module: When the wind picks up, the robot's sensors monitor the swaying of the tea buds, calculate the amplitude, frequency and direction of the swaying, and combine the position and movement of the tea leaves to predict and plan the harvesting path and direction; considering the changes in the field of vision caused by the robot's swaying during the harvesting process and the harvesting errors caused by the robot's own offset, static calibration points are determined to correct the field of vision and update the planned route.

[0011] Robot motion control module: Based on the planned path and direction, it performs motion planning in conjunction with the robot's kinematic model, generates joint motion trajectories and velocity curves, adjusts the shearing angle of the scissors to perform the shearing operation, and controls the collecting device to collect the picked tea leaves.

[0012] Preferably, in the tea identification and judgment unit, the shape features of the tea leaves are described by contour detection and shape descriptor methods, and the color features of the tea leaves are obtained by extracting the numerical information of the color by converting the image to the HSV color space.

[0013] Formula for judging fruit ripeness: Let the collected fruit color parameters be H, S, and V, respectively, representing hue, saturation, and brightness; if H... min ≤ H ≤ H max And S min ≤ S ≤ S max And V min ≤ V ≤ V max If the fruit reaches the required maturity level for harvesting, then it is determined that the fruit has reached the required maturity level.

[0014] Preferably, the tea identification and judgment unit further prioritizes tea leaves based on their maturity, specifically including: harvesting priority for fruits of different maturity levels: let maturity be A, and divided into n levels, namely A1, A2, ..., A n The corresponding picking priorities are P1, P2, ..., P n And P1>P2>...>P n The higher the priority value, the higher the picking priority. Maturity level A is set as the sum of hue, saturation, and brightness values, i.e., A = H + S + V. Only maturity levels that meet the picking requirements are graded, i.e., H. min +S min +V min≤ A ≤ H max +S max +V max .

[0015] Preferably, before locating the tea buds, the tea motion prediction module transforms the coordinates of the acquired image to the robot coordinates, including: setting the origin of the image coordinate system at the upper left corner of the image, with the x-axis to the right and the y-axis downward; setting the origin of the robot coordinate system at the center of the robot base, with the x-axis forward, the y-axis to the left, and the z-axis upward; and obtaining the transformation matrix T through camera calibration. img−robot Used to convert image coordinates (x, y) to robot coordinates (X, Y, Z), T img−robot It is a 4×3 transformation matrix. The transformation formula from image coordinates to robot coordinates is:

[0016] .

[0017] Preferably, the stable position is set as the position where the tea buds linger for the longest time at several time points. The stable position coordinates are defined as (X0, Y0, Z0) by averaging the coordinates of the tea buds at several time points. This is obtained by statistically averaging the positions of the tea buds over a period of time. Specifically, the position coordinates of the tea buds at N time points (X0, Y0, Z0) are collected. i Y i Z i If i = 1, 2, ..., N, then the stable positions are:

[0018] .

[0019] Preferably, the maximum deviation of the tea bud from its stable position along the XYZ axes is defined by amplitudes a, b, and c, respectively. For the X-axis direction, N time points are identified. The maximum value of is used as an estimate of a. Similarly, b and c are obtained on the Y and Z axes.

[0020] The shaking frequencies of the tea buds in the X, Y, and Z directions are set to f0, respectively. X f Y f Z This was obtained through spectral analysis of sensor data;

[0021] Let the angular frequencies of the tea buds' swaying in the XYZ axis directions be ω. X ω Y ω Z The data was obtained by performing spectral analysis on the swaying data of the branch in three directions collected by the sensor, and ω X =2πf X ωY =2πf Y ω Z =2πf Z .

[0022] Preferably, the initial phases of the tea buds in the XYZ axes at time t=0 are set as φ, ψ, and θ, respectively, and are determined based on the position and velocity information of the tea buds at the initial time; the position X(0) and velocity v of the tea buds on the X-axis at time t=0 are known. X (0), Substitute into the displacement equation X(0)=X0+asin(φ) and the velocity equation v X (0)=aω X cos(φ), solving the simultaneous equations yields the initial phase φ:

[0023] ;

[0024] Similarly, the initial phase ψ in the Y-axis direction and the initial phase θ in the Z-axis direction can be obtained.

[0025] Preferably, the trajectory of the tea bud in three-dimensional space is represented by the displacement equation as follows:

[0026] ;

[0027] Differentiating the displacement equation yields the velocity equation:

[0028] Velocity in the X-axis direction: ;

[0029] Velocity in the Y-axis direction: ;

[0030] Velocity in the Z-axis direction: ;

[0031] Combined speed: ;

[0032] Differentiating the velocity equation yields the acceleration equation:

[0033] Acceleration in the X-axis direction: ;

[0034] Acceleration in the Y-axis direction: ;

[0035] Acceleration in the Z-axis direction: ;

[0036] Resultant acceleration: .

[0037] Preferably, in the picking direction planning module, the path of the robot picking is controlled by first picking according to priority, and then prioritizing picking from the direction where the tea buds swing less, and the picking direction is kept in accordance with the direction of the tea buds swinging.

[0038] During the harvesting process, to address the issues of field-of-view offset and harvesting end coordinate offset caused by robot movement, wind resistance, or robotic arm motion, the following correction calculations are performed:

[0039] (1) Calibration point identification and field of view offset calculation:

[0040] Calibration point selection: The robot's camera identifies a stationary object in the captured image as the calibration point B(x). s y s );

[0041] Field of view offset detection: When robot movement causes the field of view to deviate, the real-time displacement Δx of the calibration point in the image coordinate system is calculated. s Δy s Determine the horizon offset: ΔB = ;

[0042] The offset direction angle is: θ s =arctan ;

[0043] (2) Coordinate offset model of the robotic arm picking end:

[0044] Cause of offset: The robot's center offset or the movement of the robotic arm caused the picking end C(x) to deviate. p y p , z p Coordinate offset;

[0045] Offset calculation: Let the offset of the robot base center be Δx d Δy d Δz d The end effector displacement caused by the movement of the robotic arm is Δx e Δy e Δz e The total offset is: ΔC = (Δx) d +Δx e Δy d +Δy e Δz d +Δz e );

[0046] (3) Comprehensive offset compensation and harvesting end replanning:

[0047] Target picking point correction: Let the original planned picking point be O(x′, y′, z′). After considering the field of view offset ΔB and the robotic arm offset ΔC, the corrected picking point is: O′ = O + ΔB robot +ΔC;

[0048] Where ΔB robot The field of view offset is the transformation amount in the robot coordinate system, expressed through the camera calibration matrix T. img→robot The conversion yields: ΔB robot =T img→robot ·(Δx s Δy s ,0) T .

[0049] Preferably, the scissor tilt angle is set to β, and adjusted according to the tilt direction of the tea buds. The angle between the tilt direction of the tea buds and the positive X-axis of the robot coordinate system is set to γ. Therefore, the scissor tilt angle... Make sure the scissors are perpendicular to the base of the tea buds.

[0050] This invention provides a lightweight, multimodal tea garden harvesting robot control system. Compared with existing technologies, it has the following advantages:

[0051] 1. This lightweight, multimodal tea garden harvesting robot control system features a parameter loading module that presets multiple parameters, providing a comprehensive foundation for system operation. The tea leaf identification and judgment unit comprehensively utilizes image acquisition, preprocessing, and computer vision algorithms to accurately identify tea leaves and determine the maturity of tender buds. The tea leaf motion prediction module uses relevant algorithms to predict the stable position of tender buds in windy conditions, ensuring reliable cutting data. The harvesting direction planning module monitors branch swaying through sensors in windy conditions, calculates relevant parameters, and plans the cutting path and direction, improving cutting accuracy. The robot motion control module, based on the planning results and combined with a kinematic model, generates joint trajectories and velocity curves, and can also adjust the shear angle to achieve smooth and accurate harvesting and collection. Overall, this system improves harvesting accuracy, stability, and adaptability, effectively coping with complex tea garden environments.

[0052] 2. This lightweight, multimodal tea garden harvesting robot control system employs contour detection and shape descriptors to accurately depict the shape of tea leaves, converting images to HSV space to extract color numerical information, thus improving recognition accuracy. Its proposed fruit maturity judgment formula, by setting ranges for H, S, and V, can scientifically determine whether the fruit meets harvesting requirements. Simultaneously, it prioritizes harvesting based on maturity, setting maturity as the sum of hue, saturation, and brightness values, and determining the harvesting order according to the priority values. This series of innovative methods makes tea harvesting more targeted and efficient, effectively improving both harvesting quality and efficiency.

[0053] 3. This lightweight, multimodal tea garden harvesting robot control system first transforms image coordinates to robot coordinates by obtaining a transformation matrix through camera calibration, accurately establishing the connection between the two and providing a precise spatial positioning foundation for subsequent operations. Second, it innovatively sets a stable position, defined by the average of the positions where the buds linger most at multiple time points, which more realistically reflects the actual position of the buds and reduces errors. Third, it meticulously sets the motion parameters of the buds in the XYZ axes, including maximum deviation distance, swaying frequency, angular frequency, and initial phase, obtaining these parameters through scientific methods such as spectrum analysis, comprehensively and accurately describing the motion state of the buds. Finally, it constructs the motion trajectory, velocity, and acceleration equations of the buds in three-dimensional space, enabling real-time and dynamic monitoring of the bud movement. This provides a strong basis for the robot to accurately predict the stable position of the buds at the moment of shearing, greatly improving the accuracy and success rate of harvesting and effectively coping with the complex environment of tea gardens.

[0054] 4. The lightweight, multimodal tea garden harvesting robot control system in this embodiment, on the one hand, prioritizes harvesting during path planning, starting from the direction of the bud's smallest swing amplitude, and the harvesting direction corresponds to the bud's swing, improving the rationality and accuracy of harvesting and reducing harvesting errors. On the other hand, it determines the angle γ of the bud's tilt direction through image analysis and further adjusts the shear tilt angle to ensure that the shears are perpendicular to the bud's root and branch, improving the cutting effect and ensuring harvesting quality and efficiency. Attached Figure Description

[0055] Fig. 1 This is a schematic diagram of the system modules and process of the present invention;

[0056] Fig. 2 This is a schematic diagram of the inclined shearing direction of the scissors of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] See Figs. 1-2 The present invention provides the following four technical solutions:

[0059] First implementation: A lightweight multimodal tea garden harvesting robot control system, which is mounted on a lightweight multimodal tea garden harvesting robot to control its operation, including:

[0060] Parameter loading module: Loads preset tea garden environment parameters, appearance characteristic parameters of buds of different tea varieties (maturity or suitable picking standards corresponding to shape and color), robot motion parameters (such as maximum speed, acceleration, etc.) and picking strategy parameters (such as picking priority of buds of different maturity or standards, etc.) from the storage device.

[0061] Tea leaf identification and judgment unit: The robot uses a camera installed on it to collect real-time images of the tea garden environment, and obtains image data containing tea trees and tender buds. The collected images are preprocessed, and computer vision algorithms are used to extract the shape and color features of the leaves to determine whether they are the tea leaves to be picked. Then, the leaves in the image are identified according to the pre-recorded appearance features of the tender buds to determine whether they are tea leaves. Finally, the extracted color features of the tender buds are analyzed by the correspondence between color features and maturity or suitable picking to determine whether the tender buds meet the picking requirements.

[0062] Tea Motion Prediction Module: Utilizing image processing and target tracking algorithms, the module locates the identified suitable buds for picking in real time, determining the buds' position information in the image coordinate system. When wind is detected causing the branches to sway, the module uses motion prediction algorithms to analyze the impact of the branch swaying on the buds' position, predicting the stable position of the buds at the moment of cutting, thus providing a basis for the cutting operation.

[0063] Harvesting Direction Planning Module: In windy conditions, sensors installed on the robot monitor the swaying of branches in real time; by collecting and analyzing sensor data, the amplitude, frequency and direction of swaying are calculated; based on the calculation results of branch swaying, the direction with smaller branch swaying amplitude is found; combined with the position and motion prediction information of the buds, the robot's cutting path and direction are planned to improve the accuracy of cutting.

[0064] Robot motion control module: Based on the planned cutting path and direction, and combined with the robot's kinematic model, motion planning is performed; the motion trajectory and velocity curve of each joint of the robot are generated to ensure that the robot can move smoothly and accurately to the target position; when the robot reaches the vicinity of the bud root, the tilt angle of the shears is adjusted according to the condition of the bud root branches identified in the image, so that the shears are perpendicular to the bud root branches; then the shears are controlled to close, completing the cutting of the root branches and picking the bud; the picked buds are collected through the collection device on the robot (such as a collection basket, conveyor belt, etc.).

[0065] The parameter loading module presets multiple parameters, providing a comprehensive foundation for system operation. The tea leaf identification and judgment unit comprehensively utilizes image acquisition, preprocessing, and computer vision algorithms to accurately identify tea leaves and determine the maturity of tender buds. The tea leaf motion prediction module uses relevant algorithms to predict the stable position of tender buds in windy conditions, ensuring reliable cutting data. The picking direction planning module monitors branch swaying through sensors in windy conditions, calculates relevant parameters, and plans the cutting path and direction, improving cutting accuracy. The robot motion control module, based on the planning results and combined with a kinematic model, generates joint trajectories and velocity curves, and can also adjust the shear angle to achieve smooth and accurate picking and collection. Overall, this improves the accuracy, stability, and adaptability of picking, effectively coping with complex tea garden environments.

[0066] The second implementation method differs from the first implementation method in that: in the tea identification and judgment unit, the shape features of the tea leaves are described by contour detection and shape descriptor methods, and the color features of the tea leaves are obtained by extracting the numerical information of the color by converting the image to the HSV color space.

[0067] Formula for judging fruit ripeness: Let the collected fruit color parameters be (H, S, V), where H, S, and V are hue (range 0 - 360), saturation (range 0 - 1), and brightness (range 0 - 1), respectively; if H... min ≤ H ≤ H max And S min ≤ S ≤ S max And V min ≤ V ≤ V max If the fruit reaches the required maturity level for harvesting, then it is determined that the fruit has reached the required maturity level.

[0068] The tea identification and judgment unit also prioritizes tea leaves based on their maturity, specifically including: Harvesting priority for fruits at different maturity levels: Let maturity be A, and divided into n levels, namely A1, A2, ..., A... n The corresponding picking priorities are P1, P2, ..., P n And P1>P2>...>P n The higher the priority value, the higher the picking priority. Maturity level A is set as the sum of hue, saturation, and brightness values, i.e., A = H + S + V. Only maturity levels that meet the picking requirements are graded, i.e., H. min +S min +V min ≤ A≤ H max +S max +V max .

[0069] In terms of feature description, contour detection and shape descriptors are used to accurately depict the shape of tea leaves, and the image is converted to HSV space to extract color numerical information, improving recognition accuracy. The proposed formula for judging fruit maturity, by setting the ranges of H, S, and V, can scientifically determine whether the fruit meets the picking requirements. Simultaneously, picking priorities are divided according to maturity, with maturity set as the sum of hue, saturation, and brightness values, and the picking order is determined according to the priority values. This series of innovative methods makes tea picking more targeted and efficient, effectively improving both picking quality and efficiency.

[0070] The third implementation differs from the first in that: before locating the tea buds, the tea motion prediction module transforms the coordinates of the acquired image to the robot coordinate system. This includes: setting the origin of the image coordinate system at the upper left corner of the image, with the x-axis to the right and the y-axis downward; and setting the origin of the robot coordinate system at the center of the robot base, with the x-axis forward, the y-axis to the left, and the z-axis upward; and obtaining the transformation matrix T through camera calibration. img−robot Used to convert image coordinates (x, y) to robot coordinates (X, Y, Z), T img−robot It is a 4×3 transformation matrix. The transformation formula from image coordinates to robot coordinates is:

[0071] .

[0072] The stable position is defined as the position where the tea buds linger for the longest time at several time points. The stable position coordinates are defined as (X0, Y0, Z0) by averaging the coordinates of the tea buds at several time points. Specifically, the stable position coordinates are collected at N time points (X0, Y0, Z0). i Y i Z i If i = 1, 2, ..., N, then the stable positions are:

[0073] .

[0074] The maximum deviation of the tea bud from its stable position along the XYZ axes is defined by amplitudes a, b, and c, respectively. For the X-axis, the maximum deviation from the stable position along the X, Y, and Z axes is determined at N time points. The maximum value of is used as an estimate of a. Similarly, b and c are obtained on the Y and Z axes.

[0075] The shaking frequencies of the tea buds in the X, Y, and Z directions are set to f0, respectively. X f Y f Z(Unit: Hertz, Hz), obtained through spectral analysis of sensor data;

[0076] Let the angular frequencies of the tea buds' swaying in the XYZ axis directions be ω. X ω Y ω Z The data was obtained by performing spectral analysis on the swaying data of the branch in three directions collected by the sensor, and ω X =2πf X ω Y =2πf Y ω Z =2πf Z .

[0077] Let the initial phases of the tea buds along the X, Y, and Z axes at time t=0 be φ, ψ, and θ, respectively, and determine them based on the initial position and velocity information of the tea buds; the position X(0) and velocity v of the tea buds along the X-axis at time t=0 are known. X (0), Substitute into the displacement equation X(0)=X0+asin(φ) and the velocity equation v X (0)=aω X cos(φ), solving the simultaneous equations yields the initial phase φ:

[0078] ;

[0079] Similarly, the initial phase ψ in the Y-axis direction and the initial phase θ in the Z-axis direction can be obtained.

[0080] The trajectory of a tea bud in three-dimensional space can be represented by a displacement equation as follows:

[0081] ;

[0082] Differentiating the displacement equation yields the velocity equation:

[0083] Velocity in the X-axis direction: ;

[0084] Velocity in the Y-axis direction: ;

[0085] Velocity in the Z-axis direction: ;

[0086] Combined speed: ;

[0087] Differentiating the velocity equation yields the acceleration equation:

[0088] Acceleration in the X-axis direction: ;

[0089] Acceleration in the Y-axis direction: ;

[0090] Acceleration in the Z-axis direction: ;

[0091] Resultant acceleration: .

[0092] First, the image coordinates are transformed to robot coordinates using camera calibration to obtain the transformation matrix, accurately establishing the connection between the two and providing a precise spatial positioning foundation for subsequent operations. Second, an innovative stable position is defined using the average of the positions where the buds linger most at multiple time points, more realistically reflecting the actual position of the buds and reducing errors. Third, detailed motion parameters of the buds in the XYZ axes are defined, including maximum deviation distance, swaying frequency, angular frequency, and initial phase, obtained through scientific methods such as spectrum analysis, comprehensively and accurately describing the motion state of the buds. Finally, the motion trajectory, velocity, and acceleration equations of the buds in three-dimensional space are constructed, enabling real-time and dynamic monitoring of the bud movement. This provides a strong basis for the robot to accurately predict the stable position of the buds at the moment of shearing, greatly improving the accuracy and success rate of harvesting and effectively coping with the complex environment of tea gardens.

[0093] The fourth implementation method differs from the first implementation method in that the picking direction planning module controls the robot's picking path by first picking according to priority, then prioritizing picking from the direction where the tea buds swing less, and the picking direction corresponds to the direction of the tea buds swinging.

[0094] During the harvesting process, to address the issues of field-of-view offset and harvesting end coordinate offset caused by robot movement, wind resistance, or robotic arm motion, the following correction calculations are performed:

[0095] (1) Calibration point identification and field of view offset calculation:

[0096] Calibration point selection: The robot's camera identifies a stationary object (such as a fixed pole or tree stump) in the captured image as the calibration point B(x). s y s );

[0097] Field of view offset detection: When robot movement causes the field of view to deviate, the real-time displacement Δx of the calibration point in the image coordinate system is calculated. s Δy s Determine the horizon offset: ΔB = ;

[0098] The offset direction angle is: θ s =arctan ;

[0099] (2) Coordinate offset model of the robotic arm picking end:

[0100] Cause of offset: The robot's center offset or the movement of the robotic arm caused the picking end C(x) to deviate. p y p , z p Coordinate offset;

[0101] Offset calculation: Let the offset of the robot base center be Δx d Δy d Δz d The end effector displacement caused by the movement of the robotic arm is Δx e Δy e Δz e The total offset is: ΔC = (Δx) d +Δx e Δy d +Δy e Δz d +Δz e );

[0102] (3) Comprehensive offset compensation and harvesting end replanning:

[0103] Target picking point correction: Let the original planned picking point be O(x′, y′, z′). After considering the field of view offset ΔB and the robotic arm offset ΔC, the corrected picking point is: O′ = O + ΔB robot +ΔC;

[0104] Where ΔB robot The field of view offset is the transformation amount in the robot coordinate system, expressed through the camera calibration matrix T. img→robot The conversion yields: ΔB robot =T img→robot ·(Δx s Δy s ,0) T .

[0105] Set the scissor tilt angle to β (unit: degrees, °), and adjust it according to the tilt direction of the tea buds. Let γ be the angle between the tilt direction of the tea buds and the positive X-axis of the robot coordinate system (obtained through image analysis). Then the scissor tilt angle is... Make sure the scissors are perpendicular to the base of the tea buds.

[0106] In this embodiment, on the one hand, when planning the path, picking is done first according to priority, and then starting from the direction where the buds swing with the smallest amplitude. The picking direction corresponds to the bud swing, which improves the rationality and accuracy of picking and reduces picking errors. On the other hand, the angle γ of the bud tilt direction is determined by image analysis, and the tilt angle of the shears is further adjusted to ensure that the shears are perpendicular to the bud root and branch, improving the cutting effect and ensuring the quality and efficiency of picking.

[0107] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0109] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight, multimodal tea garden harvesting robot control system, characterized in that, include: Parameter loading module: Loads preset parameters from storage devices, including tea garden environment, tea tree variety and tea appearance characteristics, robot movement and picking strategy; Tea leaf identification and judgment unit: The robot camera collects and preprocesses images of the tea garden in real time, uses computer vision algorithms to extract leaf shape and color features, determines whether the tea leaves are the ones to be picked, and then judges the maturity based on the preset correspondence. Tea leaf motion prediction module: Real-time positioning of mature tea leaves is achieved through image processing and target tracking algorithms. When the wind blows, the motion prediction algorithm is used to analyze the motion state of the tea buds and predict the stable position of the buds at the moment of shearing. Before locating the tea buds, the tea movement prediction module converts the coordinates of the acquired images to robot coordinates. This includes: setting the origin of the image coordinate system at the upper left corner of the image, with the x-axis to the right and the y-axis downwards; and setting the origin of the robot coordinate system at the center of the robot base, with the x-axis forward, the y-axis to the left, and the z-axis upwards. The stable position is set as the position with the longest dwell time at several time points. The stable position coordinates are defined as (X0, Y0, Z0) by averaging the coordinates of the tea buds over a period of time. The maximum deviation of the tea buds from the stable position in the X, Y, and Z axis directions is set to amplitudes a, b, and c, respectively. For the X-axis direction, the maximum deviation of the tea buds from the stable position at N time points is determined. The maximum value is used as an estimate of a, and similarly b and c are obtained on the Y and Z axes; The shaking frequencies of the tea buds in the X, Y, and Z directions are set to f0, respectively. X f Y f Z This was obtained through spectral analysis of sensor data; Let the angular frequencies of the tea buds' swaying in the XYZ axis directions be ω. X ω Y ω Z The data was obtained by performing spectral analysis on the swaying data of the branches in three directions collected by the sensor. The initial phases of the tea buds in the XYZ axis directions at time t=0 are set as φ, ψ, and θ, respectively, and are determined based on the position and velocity information of the tea buds at the initial time. Harvesting Direction Planning Module: When the wind blows, the robot's sensors monitor the swaying of the tea buds, calculate the amplitude, frequency, and direction of the swaying, and combine the position and movement of the tea leaves to predict and plan the harvesting path and direction; considering the changes in the field of vision caused by the robot's swaying during the harvesting process and the harvesting errors caused by the robot's own deviation, a static calibration point is determined to correct the field of vision and update the planned route; in the harvesting direction planning module, the robot's harvesting path is controlled according to priority, and then priority is given to picking from the direction where the swaying amplitude of the tea buds is small, and the picking direction is kept to correspond to the swaying direction of the tea buds; Robot motion control module: Based on the planned path and direction, it performs motion planning in conjunction with the robot's kinematic model, generates joint motion trajectories and velocity curves, adjusts the shearing angle of the scissors to perform the shearing operation, and controls the collecting device to collect the picked tea leaves.

2. The lightweight multimodal tea garden harvesting robot control system according to claim 1, characterized in that: In the tea identification and judgment unit, the shape features of tea leaves are described by contour detection and shape descriptor methods, and the color features of tea leaves are obtained by extracting the numerical information of color by converting the image to the HSV color space. Formula for judging fruit ripeness: Let the collected fruit color parameters be H, S, and V, respectively, representing hue, saturation, and brightness; if H... min ≤H≤H max And S min ≤S≤S max And V min ≤V≤V max If the fruit reaches the required maturity level for harvesting, then it is determined that the fruit has reached the required maturity level.

3. The lightweight multimodal tea garden harvesting robot control system according to claim 2, characterized in that: The tea identification and judgment unit also prioritizes tea leaves based on their maturity, specifically including: harvesting priority for fruits of different maturity levels: let maturity be A, and divided into n levels, namely A1, A2, ..., A n The corresponding picking priorities are P1, P2, ..., P n And P1>P2>...>P n The higher the priority value, the higher the picking priority. Maturity level A is set as the sum of hue, saturation, and brightness values, i.e., A = H + S + V. Only maturity levels that meet the picking requirements are graded, i.e., H... min +S min +V min ≤A≤H max +S max +V max .

4. The lightweight multimodal tea garden harvesting robot control system according to claim 1, characterized in that: The transformation matrix T is obtained through camera calibration. img−robot Used to convert image coordinates (x, y) to robot coordinates (X, Y, Z), T img−robot It is a 4×3 transformation matrix. The transformation formula from image coordinates to robot coordinates is: 。 5. The lightweight multimodal tea garden harvesting robot control system according to claim 4, characterized in that: Specifically, this involves collecting the location coordinates (X, X) of tea buds at N time points. i Y i Z i If i = 1, 2, ..., N, then the stable positions are: 。 6. The lightweight multimodal tea garden harvesting robot control system according to claim 5, characterized in that: oh X =2πf X 、oh Y =2πf Y 、oh Z =2πf Z 。 7. The lightweight multimodal tea garden harvesting robot control system according to claim 6, characterized in that: Given the position X(0) and velocity v of the tea bud on the X-axis at time t=0. X (0), Substitute into the displacement equation X(0)=X0+asin(φ) and the velocity equation v X (0)=aω X cos(φ), solving the simultaneous equations yields the initial phase φ: ； Similarly, the initial phase ψ in the Y-axis direction and the initial phase θ in the Z-axis direction can be obtained.

8. The lightweight multimodal tea garden harvesting robot control system according to claim 7, characterized in that: The trajectory of a tea bud in three-dimensional space can be represented by the displacement equation as follows: ； Differentiating the displacement equation yields the velocity equation: Velocity in the X-axis direction: ; Velocity in the Y-axis direction: ; Velocity in the Z-axis direction: ; Combined speed: ; Differentiating the velocity equation yields the acceleration equation: Acceleration in the X-axis direction: ; Acceleration in the Y-axis direction: ; Acceleration in the Z-axis direction: ; Resultant acceleration: .

9. The lightweight multimodal tea garden harvesting robot control system according to claim 4, characterized in that: During the harvesting process, to address the issues of field-of-view offset and harvesting end coordinate offset caused by robot movement, wind resistance, or robotic arm motion, the following correction calculations are performed: (1) Calibration point identification and field of view offset calculation: Calibration point selection: The robot's camera identifies a stationary object in the captured image as the calibration point B(x). s y s ); Field of view offset detection: When robot movement causes the field of view to deviate, the real-time displacement Δx of the calibration point in the image coordinate system is calculated. s Δy s Determine the horizon offset: ΔB = ; The offset direction angle is: θ s =arctan ; (2) Coordinate offset model of the robotic arm picking end: Cause of offset: The robot's center offset or the movement of the robotic arm caused the picking end C(x) to deviate. p y p , z p Coordinate offset; Offset calculation: Let the offset of the robot base center be Δx d Δy d Δz d The end effector displacement caused by the movement of the robotic arm is Δx e Δy e Δz e The total offset is: ΔC = (Δx) d +Δx e Δy d +Δy e Δz d +Δz e ); (3) Comprehensive offset compensation and harvesting end replanning: Target picking point correction: Let the original planned picking point be O(x′, y′, z′). After considering the field of view offset ΔB and the robotic arm offset ΔC, the corrected picking point is: O′ = O + ΔB robot +ΔC; Where ΔB robot The field of view offset is the transformation amount in the robot coordinate system, expressed through the camera calibration matrix T. img→robot The conversion yields: ΔB robot =T img→robot ·(Δx s Δy s ,0) T .

10. The lightweight multimodal tea garden harvesting robot control system according to claim 1, characterized in that: Set the scissor tilt angle to β, and adjust it according to the tilt direction of the tea buds. Let γ be the angle between the tilt direction of the tea buds and the positive X-axis of the robot coordinate system. Then the scissor tilt angle is... Make sure the scissors are perpendicular to the base of the tea buds.

Images