Picking mechanical arm motion planning method and system

Abstract

The invention provides a picking mechanical arm motion planning method and system, and relates to the technical field of agricultural picking, and the method comprises the steps: carrying out the collection of mature fruit and vegetable information in a to-be-picked region, generating first picking information M1, carrying out the air collection of visual image data in the to-be-picked region through an unmanned aerial vehicle, and generating second picking information M2; performing comparative analysis on the mature fruit and vegetable information, generating information data of the mature fruits and vegetables shielded by branches and leaves, collecting surrounding environment information of the mature fruits and vegetables shielded by the branches and leaves, and generating third picking information M3; fusing collected data in the first picking information M1, the second picking information M2 and the third picking information M3, analyzing and processing to generate a picking evaluation model P, generating an initial planning path H1 based on a value domain set of the picking evaluation P, and generating a branch and leaf shielding evaluation index ZDZs, and judging picking motion planning of the fruit and vegetable picking mechanical arm according to a comparison threshold value U of the branch and leaf shielding evaluation index ZDZs, and sequentially generating a planning path H2 and a planning path H3.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural harvesting, and in particular to a harvesting robot arm motion planning method and system. Background Art

[0002] The motion planning of the fruit and vegetable picking robot involves inverse kinematics, trajectory generation, collision detection and avoidance, path planning, and force / torque control methods. All of them are based on the use of visual cameras to obtain the position information of mature fruits and vegetables before performing subsequent picking steps. Inverse kinematics is used to solve joint angles, generate smooth trajectories, detect and avoid collisions, plan effective paths, and achieve flexible control, ensuring that the robot moves efficiently, safely, and accurately during picking tasks and adapts to the shapes and hardness of different fruit and vegetable plants.

[0003] For example, the Chinese patent with announcement number CN115139315A discloses a method for planning the grabbing motion of a picking robot arm. By dividing the working space of the robot arm and defining the pre-picking plane, the entire picking planning process is divided into two parts: offline planning and online planning. The offline planning part is trained based on the reinforcement learning algorithm PPO algorithm. First, the position of the fruit is initially identified based on a wide angle of view, and the position of the fruit pre-picking point is obtained according to the projection of the fruit position to the pre-picking plane. The trajectory to each pre-picking point is output through reinforcement learning. Then we model the offline planning part and the online planning part, model them into a traveling salesman problem, and plan the optimal picking order. Finally, the robot arm arrives at the pre-picking point in turn according to the planned picking order to identify the fruit stems from a close angle to complete the picking. The robot arm adopts offline planning and planned picking order to significantly improve the success rate and picking efficiency of the picking robot.

[0004] The shortcomings are as follows: the Chinese patent with announcement number CN115139315A divides the working space of the robot arm to realize offline planning and online planning of the picking path, but the surrounding environment of mature fruits and vegetables is often blocked by branches and leaves. It is difficult to identify the blocked mature fruits and vegetables by relying solely on the visual recognition camera on the robot arm. In the picking path planning for mature fruits and vegetables blocked by branches and leaves, there are cases where the robot arm is scratched or stuck by branches and leaves. How to collect visual images of picked fruits and vegetables, analyze the position information of mature fruits and vegetables blocked by branches and leaves, and respectively construct a picking evaluation model and a branch and leaf occlusion evaluation index, and plan the pre-picking path between the robot arm and the target fruit based on the value range set of the picking evaluation model, determine the information of the blocked mature fruits and vegetables, and generate the initial planning path H1; then, based on the comparison of the branch and leaf occlusion evaluation index with the set threshold, it is judged whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion when moving to plan the paths H2 and H3 for picking fruits and vegetables;

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a picking robot arm motion planning method and system to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The picking robot arm motion planning method includes the following specific steps:

[0009] S1. Obtain historical picking data and a botanical database of the area to be picked, and use them to identify key information on the growth cycle and maturity characteristics of different fruits and vegetables; collect information on mature fruits and vegetables on the overall moving path of the picking robot arm in the area to be picked, and generate first picking information M 1 ,

[0010] S2, using a drone equipped with a visual recognition camera to collect visual image data of the area to be picked, and adjusting the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables hidden by the branches and leaves, and generate a large range of second picking information M for picking mature fruits and vegetables 2 ; Get the first picking information M 1 With the second picking information M 2 Then, the information of mature fruits and vegetables is compared and analyzed to generate the information data of mature fruits and vegetables that are blocked by branches and leaves;

[0011] S3, collecting the surrounding environment information of the ripe fruits and vegetables blocked by branches and leaves and generating the third picking information M 3 ; Fusion of the first picking information M 1 , the second picking information M 2 and the third picking information M 3 The collected data in is analyzed and processed to generate a picking evaluation model P, and a value range set of the picking evaluation model P is determined. Based on the value range set of the picking evaluation model P, a pre-picking path between the robot arm and the target fruit is planned to generate an initial planning path H1;

[0012] S4. Obtaining the first picking information M 1 , the second picking information M 2 and the third picking information M 3After that, the branch and leaf occlusion evaluation index ZDZs is generated, and the comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during the picking motion planning, and the planning path H2 and the planning path H3 are generated in turn;

[0013] S5. Continuously obtain data on environmental changes of the picking robot arm during the picking process, set an emergency response mechanism for the picking robot arm to escape from a stuck state, and determine whether new obstructions appear or the existing obstruction status changes to determine whether to trigger the emergency response mechanism.

[0014] A picking robot arm motion planning system, the system is used to execute the motion planning method, comprising:

[0015] The first picking information establishment module: obtains historical picking data and botanical database of the area to be picked, and uses them to identify key information on the growth cycle and maturity characteristics of different fruits and vegetables; collects information on mature fruits and vegetables on the overall moving path of the picking robot arm in the area to be picked, and generates the first picking information M 1 ,

[0016] The second picking information establishment module: The drone equipped with a visual recognition camera collects visual image data of the picking area from the air, and adjusts the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables hidden by the branches and leaves, and generates a large range of second picking information M for picking mature fruits and vegetables. 2 ; Get the first picking information M 1 With the second picking information M 2 Then, the information of mature fruits and vegetables is compared and analyzed to generate the information data of mature fruits and vegetables that are blocked by branches and leaves;

[0017] Picking evaluation model building module: collect the surrounding environment information of mature fruits and vegetables blocked by branches and leaves and generate the third picking information M 3 ; Fusion of the first picking information M 1 , the second picking information M 2 and the third picking information M 3 The collected data in is analyzed and processed to generate a picking evaluation model P, and a value range set of the picking evaluation model P is determined. Based on the value range set of the picking evaluation model P, a pre-picking path between the robot arm and the target fruit is planned to generate an initial planning path H1;

[0018] Branch and leaf occlusion evaluation index establishment module: obtain the first picking information M 1 , the second picking information M 2 and the third picking information M 3After that, the branch and leaf occlusion evaluation index ZDZs is generated, and the comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during the picking motion planning, and the planning path H2 and the planning path H3 are generated in turn;

[0019] Emergency response mechanism feedback module: continuously obtains environmental change data of the picking robot arm during the picking process, sets an emergency response mechanism for the picking robot arm to escape from a stuck state, and determines whether new occlusions appear or the existing occlusion status changes to determine whether to trigger the emergency response mechanism.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] By collecting information about mature fruits and vegetables in the area to be picked, the first picking information M is generated. 1 , use a drone equipped with a visual recognition camera to collect visual image data from the area to be picked, and generate the second picking information M 2 ; Get the first picking information M 1 With the second picking information M 2 Then, the mature fruits and vegetables information is compared and analyzed to generate the mature fruits and vegetables information data blocked by branches and leaves, and the surrounding environment information of the mature fruits and vegetables blocked by branches and leaves is collected to generate the third picking information M 3 ; Fusion of the first picking information M 1 , the second picking information M 2 and the third picking information M 3 The collected data in is analyzed and processed to generate a picking evaluation model P. The pre-picking path between the robot arm and the target fruit is planned based on the value range set of the picking evaluation P to generate an initial planning path H1 and a branch and leaf occlusion evaluation index ZDZs. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to determine whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during picking motion planning, and the planning paths H2 and H3 are generated in turn. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall method flow of the present invention;

[0023] Figure 2 is a system block diagram of the present invention;

[0024] Figure 3 It is a schematic diagram of the planning of H1, H2 and H3 of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] See also Figure 1 , Figure 2 and Figure 3 , the present invention provides a technical solution:

[0028] Embodiment 1:

[0029] A picking robot arm motion planning method comprises the following steps:

[0030] S1. Obtain historical picking data and a botanical database of the area to be picked, and use them to identify key information on the growth cycle and maturity characteristics of different fruits and vegetables; collect information on mature fruits and vegetables on the overall moving path of the picking robot arm in the area to be picked, and generate first picking information M 1 ,

[0031] S2, using a drone equipped with a visual recognition camera to collect visual image data of the area to be picked, and adjusting the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables hidden by the branches and leaves, and generate a large range of second picking information M for picking mature fruits and vegetables 2 ; Get the first picking information M 1 With the second picking information M 2 Then, the information of mature fruits and vegetables is compared and analyzed to generate the information data of mature fruits and vegetables that are blocked by branches and leaves;

[0032] S3, collecting the surrounding environment information of the ripe fruits and vegetables blocked by branches and leaves and generating the third picking information M 3 ; Fusion of the first picking information M 1 , the second picking information M 2 and the third picking information M 3The collected data in is analyzed and processed to generate a picking evaluation model P, and a value range set of the picking evaluation model P is determined. Based on the value range set of the picking evaluation model P, a pre-picking path between the robot arm and the target fruit is planned to generate an initial planning path H1;

[0033] S4. Obtaining the first picking information M 1 , the second picking information M 2 and the third picking information M 3 After that, the branch and leaf occlusion evaluation index ZDZs is generated, and the comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during the picking motion planning, and the planning path H2 and the planning path H3 are generated in turn;

[0034] S5. Continuously obtain data on environmental changes of the picking robot arm during the picking process, set an emergency response mechanism for the picking robot arm to escape from a stuck state, and determine whether new obstructions appear or the existing obstruction status changes to determine whether to trigger the emergency response mechanism.

[0035] Embodiment 2:

[0036] Further explanation based on the first embodiment, the acquisition of historical picking data and botanical database of the area to be picked and used to identify key information of growth cycles and maturity characteristics of different fruits and vegetables specifically includes the following contents:

[0037] The historical harvesting data includes the date, time, yield, and variety of fruit and vegetable harvesting in the area to be harvested in the past;

[0038] Usage: By analyzing this data, we can predict the best harvesting period for different fruits and vegetables, as well as the expected yield and other information.

[0039] The botanical database contains the growth cycles and maturity characteristics of different fruits and vegetables, where the maturity characteristics are specifically color, size, shape, and the best time to pick them;

[0040] Purpose: Provides decision support on which fruits and vegetables to plant and when, and helps identify ripe fruits and vegetables.

[0041] The picking robot arm collects information about ripe fruits and vegetables on the overall moving path of the area to be picked, and generates first picking information M 1 , specifically including the following:

[0042] First picking information M 1 The data obtained by the external visual sensor includes the following collected data:

[0043] Fruit and vegetable location and maturity data: Capture the precise location information of fruits and vegetables through external visual sensors 1 and maturity information CSd. Maturity information CSd is obtained by collecting color C1, size C2, shape C3 data of fruits and vegetables in the image and calculating the position information Wz 1 GPS coordinates including latitude, longitude and altitude;

[0044] Fruit and vegetable type and characteristic data: According to the type characteristics LXTz of different fruits and vegetables, record their specific growth cycle and maturity characteristics;

[0045] The current position and status data of the robot arm: including the coordinates Zb, direction Fx, and the current task status ZT data of the robot arm; the robot arm is equipped with a variety of sensors that can monitor its coordinates, direction, and current task status in real time, which are obtained through built-in sensors and control systems;

[0046] First picking information M 1 The formula is:

[0047]

[0048] in Respectively represent natural logarithm, square root, sine, and exponential functions; (α, β) are adjustment parameters;

[0049] W 1 =(GPS lat ,GPS lon ,GPS alt )

[0050] Wz 1 The elements of are combined into a scalar value, specifically using the average of longitude, latitude, and altitude to represent Wz 1 The scalar value of is, and the calibration formula is as follows:

[0051]

[0052] Among them, Wz 1 Indicates the precise location information of fruits and vegetables, including GPS longitude lat , latitude GPS lon , and altitude GPS alt ;

[0053] CSd=k 1 C1+k 2 C2+k 3 C3

[0054] Among them, CSd represents the maturity information, which is a function based on the color C1, size C2 and shape C3 of fruits and vegetables. 1 , k2 , and k 3 It is a weight parameter set according to actual conditions, which is used to adjust the contribution of color, size and shape to maturity assessment;

[0055] Set M 1 The range of is [0.1,1]:

[0056] When M 1 When it is close to 0.1, it indicates that the picking efficiency or accuracy is low; this is because the position of fruits and vegetables is unclear. 1 , the maturity assessment CSd is inaccurate or the coordinates / direction / task status of the robot arm is not ideal (Zb, Fx, ZT);

[0057] When M 1 When it is close to 1, it means that the picking efficiency or accuracy is high, which means that all parameters are in an ideal state.

[0058] Embodiment three:

[0059] Further explanation is given on the basis of the first embodiment, the drone equipped with a visual recognition camera collects visual image data of the area to be picked in the air, and adjusts the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables blocked by branches and leaves, and generates a large range of second picking information M for picking mature fruits and vegetables. 2 , obtain the first picking information M 1 With the second picking information M 2 Then, the mature fruit and vegetable information is compared and analyzed to generate the mature fruit and vegetable information data blocked by branches and leaves, including the following contents:

[0060] Monitor the impact of the wind generated by the rotating drone rotors on the surrounding vegetation; adjust the drone's flight altitude and angle, or adjust the rotor speed to generate appropriate wind to expose the fruits and vegetables hidden by branches and leaves;

[0061] While adjusting the wind direction, the visual recognition camera on the drone continues to capture images and identify fruits and vegetables exposed by the wind. The collected image data is sent to the processing center for image analysis and identification of mature fruits and vegetables.

[0062] Adjust the flight altitude and angle of the drone, or adjust the rotor speed. The specific steps are as follows;

[0063] The flight altitude adjustment quantization step sets the adjustment range of the drone’s flight altitude H to H min To H max , this range should be based on the safe flight altitude of the drone and the average height of the target crops. The adjustment of the flight altitude can be expressed by the following formula:

[0064]

[0065] Where H is the flight altitude of the drone;

[0066] S is the occlusion degree score obtained from the visual system, and the score range is set from 1 to 10. The higher the score, the more severe the occlusion.

[0067] Among them, 1 means 0%-10% blocked, 2 means 11%-20% blocked, 3 means 21%-30% blocked, 4 means 31%-40% blocked, 5 means 41%-50% blocked, 6 means 51%-60% blocked, 7 means 61%-70% blocked, 8 means 71%-80% blocked, 9 means 81%-90% blocked, and 10 means 91%-100% blocked;

[0068] Flight angle adjustment quantification;

[0069] Set the flight angle of the drone to θ min to θ max , the adjustment of the flight angle can quantify the direction of the branch and leaf occlusion area as needed:

[0070]

[0071] Where θ is the flight angle of the drone relative to the horizontal plane;

[0072] D is the target direction recognized by the visual system, such as north, south, east, west, or an angle between them, converted into a value from 0 to D max The value of

[0073] D max It is the maximum value of the direction, 360°;

[0074] Quantification of rotor speed adjustment;

[0075] Set the adjustment range of rotor speed R to R min To R max ; The adjustment of rotor speed can be quantified based on wind demand:

[0076]

[0077] Where: R is the rotation speed of the rotor;

[0078] F is the wind demand score based on real-time video analysis, ranging from 0 to F max; The higher the score, the stronger the wind force is needed to remove the obstruction. The wind force requirement score is proportional to the obstruction degree score, that is, the higher the obstruction degree, the higher the required wind force level. When the obstruction score is set to 1, the required initial wind force level is 5 Newtons. For each additional level, the wind force increases by 10%;

[0079] For soft leaves, such as lettuce and spinach, when the shading score is 1, the required initial wind force level is 5 Newtons, and the wind force increases by 10% for each additional level;

[0080] For leaves of medium thickness and hardness, such as carrot leaves and celery leaves, when the shading score is 1, the required initial wind force level is 10 Newtons, and the wind force increases by 15% for each additional level.

[0081] For thicker and harder leaves, such as cabbage leaves and radish leaves, when the shading score is 1, the required initial wind force level is 20 Newtons, and the wind force increases by 12% for each additional level.

[0082] F max is the maximum value of the wind demand score, and the adaptive improvement is made according to the leaf type of the picked fruits and vegetables. max The wind force represented is capable of blowing the corresponding blade surface until its lateral horizontal displacement reaches 80% of the original blade surface width;

[0083] The second picking information M 2 Specifically, the following data are collected:

[0084] Wide-area fruit and vegetable distribution data: The wide-area distribution map of the area to be picked provided by the drone, including the group distribution QTFb of fruits and vegetables and the row spacing HJj;

[0085] The drone uses the wind generated by its rotors to reveal information about fruits and vegetables that are hidden by branches and leaves. The specific data includes:

[0086] Blocked location information: the latitude, longitude and altitude of the blocked fruits and vegetables in the farmland;

[0087] Visible part features KJBf: Features of the visible area after the fruits and vegetables are blocked. The visible area features include D1 color, D2 size, and D3 shape;

[0088] The occlusion degree ZDCs indicates the degree to which fruits and vegetables are obscured, such as complete or partial occlusion. According to the degree of occlusion, it is divided into 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 levels. The larger the value, the larger the obscured area.

[0089] 1 means 0%-10% blocked, 2 means 11%-20% blocked, 3 means 21%-30% blocked, 4 means 31%-40% blocked, 5 means 41%-50% blocked, 6 means 51%-60% blocked, 7 means 61%-70% blocked, 8 means 71%-80% blocked, 9 means 81%-90% blocked, 10 means 91%-100% blocked;

[0090] The above level settings are critical to determining whether branches and leaves need to be moved for picking;

[0091] Fruit and vegetable group information GSQt: data on the situation where multiple fruits and vegetables in the area are blocked;

[0092] Branch and leaf movement data YDSj: the movement of branches and leaves caused by the wind force of the drone rotor, including the movement amplitude YDFd and frequency f of the branches and leaves;

[0093] Combine external visual sensors and drone data, use data fusion algorithms such as Kalman filters to optimize the accuracy of fruit and vegetable location and maturity information, and build a multi-level decision tree model to analyze and determine picking priorities;

[0094] Second picking information M 2 The formula is:

[0095]

[0096] Among them, (e x ,tanh(x),Γ(x),x y ,ln(x)) represent exponential, hyperbolic tangent, gamma function, power function and natural logarithm respectively;

[0097] QTFb represents the wide-area distribution of fruit and vegetable groups;

[0098] HJj represents the row spacing in the farmland;

[0099] The location information of the obscured fruits and vegetables is Wz 2 The formula is:

[0100] W 2 =(GPS lat2 ,GPS lon2 ,GPS alt2 )

[0101] The characteristic KJBf formula of the visible part of fruits and vegetables is:

[0102] KJBf=k 4 ·D1+k 5 ·D2+k 6 D3

[0103] The parameters are explained as follows: color D1, size D2, shape D3, k 4 , k 5 , and k 6 It is a weight parameter set according to actual conditions, which is used to adjust the contribution of color, size and shape to the visible features of fruits and vegetables;

[0104] ZDCs represents the degree to which fruits and vegetables are shaded, for example, ZDCs = 5 means 50% shaded;

[0105] The YDFd calculation formula is:

[0106]

[0107] It represents the movement of branches and leaves caused by the wind force of the UAV rotor, including the movement amplitude YDFd and frequency f;

[0108] Set M 2 The range of is [0,10]:

[0109] When M 2 When it is close to 0, it indicates that the recognition and positioning capabilities in complex environments are poor, because the fruits and vegetables are severely blocked by ZDCs or the movement of branches and leaves is large YDSj;

[0110] When M 2 When it is close to 10, it indicates strong ability, and the distribution of fruit and vegetable groups QTFb and row spacing HJj are conducive to identification and positioning;

[0111] Embodiment 4:

[0112] Further explanation based on the first embodiment, the surrounding environment information of the ripe fruits and vegetables blocked by branches and leaves is collected and the third picking information M is generated. 3 , specifically including the following:

[0113] The third picking information M 3 Including integrated meteorological data QXSj, light condition data GZTj, vegetation density ZBMd and historical harvesting data LSCz:

[0114] The integrated meteorological data QXSj specifically collects the temperature Wd, humidity Sd, wind speed Fs, and light condition GZTj data around the mature fruits and vegetables blocked by branches and leaves. The integrated meteorological data QXSj is obtained through a meteorological station or a mobile meteorological sensor;

[0115] The light condition data GZTj include sunshine duration t1 and sun position tw;

[0116] The third picking information M 3 The formula is:

[0117]

[0118] in, They represent the standard normal distribution function, the cube root, and the logarithm of a with base b respectively;

[0119] Set M 3 The value range of is [0.5,5];

[0120] When M 3 When it is close to 0.5, it means that the current climate and environmental conditions are not conducive to picking, such as unfavorable climate data, insufficient light conditions or too high vegetation density;

[0121] When M 3 When it is close to 5, it means that the climate and environmental conditions are very conducive to picking, such as ideal meteorological conditions, sufficient sunlight and moderate vegetation density, and a good match between historical data and current conditions;

[0122] Interactive content supplement:

[0123] Under different climate and environmental conditions, such as different (QXSj, GZTj, ZBMd, LSCz), M 3 The value of will change, thus affecting the formulation of picking strategies; for example, under poor climatic conditions, QXSj is not ideal, and it is necessary to adjust the picking time or use mechanical equipment that is more adaptable to bad weather; on the other hand, under high light conditions GZTj and moderate vegetation density ZBMd), the picking efficiency can be greatly improved;

[0124] M 3 Changes in values ​​can also provide guidance on how to better utilize historical picking data LSCz; if historical data show that picking works better under specific climate conditions, then those successful strategies can be repeated under similar conditions;

[0125] To M 3 Real-time monitoring and analysis can help decision makers respond more quickly to environmental changes and optimize harvesting plans, such as harvesting earlier when adverse weather conditions are expected or expanding the harvesting range under ideal conditions;

[0126] In addition, by analyzing M 3 Historical trends and patterns can be used to predict future picking conditions, allowing for planning and preparation in advance to address challenges as they arise;

[0127] The fusion first picking information M 1 , the second picking information M 2 and the third picking information M 3 The collected data in is analyzed and processed to generate a picking evaluation model P, and the value range set of the picking evaluation model P is determined, which specifically includes the following contents:

[0128] Set the picking evaluation model P as:

[0129]

[0130] in:

[0131] W 1 : Represents the precise location information of fruits and vegetables;

[0132] CSd: represents maturity information, which is calculated from the color C1, size C2, and shape C3 of fruits and vegetables;

[0133] The specific calculation steps of CSd are as follows: make judgment based on the historical picking data of the area to be picked and the botanical database;

[0134] Color C1: represents the color characteristics of fruits and vegetables. Different types of fruits and vegetables will show different colors at different stages of maturity. For example, when a tomato changes from green to red, it indicates maturity.

[0135] Size C2: represents the physical size of fruits and vegetables. Fruits and vegetables will gradually grow larger during the growth process, and reaching a certain size usually means maturity.

[0136] Shape C3: represents the shape characteristics of fruits and vegetables. Some fruits and vegetables will change in shape when they mature, such as bananas, which will change from straight to curved;

[0137] Construction of function f:

[0138] Data collection: First, we need to collect a lot of data on the color, size, and shape of different types of fruits and vegetables at different levels of maturity;

[0139] Feature quantification: converting qualitative features such as color, size, and shape into quantifiable values. For example, color can be represented by RGB values, size can be represented by volume or length, and shape can be quantified by proportion or specific shape indicators;

[0140] Algorithm design: Use machine learning or other statistical methods to train a model based on the collected data. This model will learn the relationship between color, size, and shape and maturity, and be able to predict the maturity corresponding to a given color, size, and shape.

[0141] Maturity calculation

[0142] Input feature values: Extract the values ​​of color C1, size C2, and shape C3 of fruits and vegetables from the collected images.

[0143] Apply function f: Take these values ​​as input and apply them to function f to get the predicted value of maturity CSd.

[0144] Maturity judgment: Based on the calculated (CSd) value, determine whether the fruits and vegetables have reached the maturity for picking.

[0145] LXTz: represents fruit and vegetable types and characteristics;

[0146] Zb, Fx, ZT: represent the coordinates, direction and current task status of the robot arm respectively;

[0147] M 1 , M 2 , M 3 : Represents the first, second and third picking information respectively;

[0148] f(i) is an advanced integral function, representing a comprehensive analysis of environmental factors;

[0149] YDFd and GSQt represent the movement amplitude of branches and leaves and the information of fruit and vegetable population, respectively;

[0150] QTFb,HJj,Wz 2 ,KJBf, and ZDCs represent the distribution of fruit and vegetable groups, row spacing, occlusion position information, characteristics of the visible part, and occlusion degree, respectively;

[0151] The advanced integral function f(i) is defined as:

[0152]

[0153] This function is an integral of a Gaussian function, which is used to simulate the impact of environmental factors on the picking path;

[0154] The value range of formula P is (0, +∞), and different values ​​represent different picking efficiency and path optimization degree; lower P value indicates higher picking efficiency and more optimized path planning, while higher P value means that the picking strategy or path planning needs to be improved;

[0155] The pre-picking path between the robot arm and the target fruit is planned based on the value range set of the picking evaluation P to generate an initial planning path H1, which specifically includes the following contents:

[0156] Set the interval values ​​A and B of the picking evaluation model P, and A is less than B, and divide the value interval of P into high efficiency interval (0, A), medium efficiency interval (A, B) and low efficiency interval (B, +∞);

[0157] Among them, the high efficiency interval (0, A) represents very high picking efficiency and optimized path. The path planning principle is to respond quickly to significant changes in environmental factors f(i) and generate the initial planning path H1;

[0158] The medium efficiency interval (A, B) indicates that the picking efficiency is at a medium level. The path planning principle is to set up a feedback mechanism to analyze whether the efficiency decreases due to the fruit and vegetable type characteristics LXTz or the branch and leaf movement amplitude YDFd, and make corresponding adjustments to generate the initial planning path H1;

[0159] The low efficiency interval (B, +∞) indicates that the picking efficiency is at a low level. The path planning principle is to set up a deep analysis process, focusing on analyzing the occlusion degree ZDCs and occlusion position information Wz 2 If the measurement results are within this range for several consecutive times, the picking strategy or path planning is fully redesigned to generate the initial planning path H1. In this embodiment, the number of times is set to 3;

[0160] Specifically, high efficiency interval (0, A): Since this interval represents very high picking efficiency and optimized path, A is a relatively small value; in practical applications, this means that the movement of the robot arm is very precise and efficient, without unnecessary movement. A is set to a small positive number, such as 10, which means that when the value of P is less than 10, we believe that the picking path is highly optimized;

[0161] Adaptability to environmental factors: In the high efficiency range, if a significant change in the environmental factor f(i) value is detected, such as a sudden change in light or temperature fluctuation, the system can quickly adjust to keep the P value within this range;

[0162] Specifically: In the high efficiency range, the system must be able to respond quickly to significant changes in environmental factors f(i); taking light changes and temperature fluctuations as examples:

[0163] Adaptability to light changes: Set a light intensity threshold, such as 500-700 loads; if the sensor detects that the light intensity drops below or exceeds this range, the system will adjust the speed and path of the robot to adapt to the light change; for example, if the light intensity suddenly increases to 800 loads, the robot needs to slow down to prevent overexposure from affecting its visual recognition system;

[0164] Medium efficiency interval (A, B): This interval indicates that the picking efficiency is at a medium level; B is set to be greater than A, and it is possible to distinguish between medium efficiency and low efficiency; in this interval, the robot arm needs to perform some path optimization and parameter adjustments, but these adjustments should not be too complicated; considering these factors, B is set to a medium value, such as 50; this means that when the value of P is between 10 and 50, the picking efficiency of the robot arm is at a medium level;

[0165] Feedback adjustment mechanism: In the medium efficiency range, if the P value continues to fluctuate near the upper limit of the range, the system should activate the feedback mechanism to analyze whether the efficiency decrease is caused by the fruit and vegetable type characteristics LXTz or the branch and leaf movement amplitude YDFd, and make corresponding adjustments;

[0166] Specifically, in the medium efficiency range, the continuous fluctuation of P value indicates the need for adjustment; the following is an example of the implementation of this mechanism:

[0167] LXTz adjustment: Assuming that a fruit or vegetable becomes softer as it matures, this will affect the gripping efficiency of the robot arm. If the P value continues to fluctuate between 40-50, the system will recognize that the gripping pressure needs to be adjusted. For example, if the robot arm is handling soft fruits and vegetables, the pressure needs to be reduced to avoid damaging the fruit.

[0168] YDFd adjustment: If the movement of branches and leaves increases due to increased wind speed, the path planning of the robot arm needs to be adjusted; for example, if the wind speed increases from 5km / h to 15km / h, causing the P value to rise to 45, the system needs to increase the redundancy of the path planning to adapt to this change;

[0169] Low efficiency range: When the value of P is greater than 50, the picking efficiency is at a low level; this means that the movement path of the robot arm needs to be significantly adjusted, or environmental factors have a great impact on the picking efficiency; in this range, picking path planning faces great challenges and requires in-depth analysis and redesign;

[0170] Deep analysis and redesign: In the low efficiency range, the system should trigger the deep analysis process, focusing on analyzing the occlusion degree ZDCs and occlusion position information Wz 2 If the measurement results are within this range for several consecutive times, the picking strategy or path planning should be completely redesigned;

[0171] Specifically, in the low efficiency range, the system needs to be deeply analyzed for redesign; the following is a specific implementation plan:

[0172] Analyze the impact of ZDCs: If the P value is higher than 50 for several consecutive pickings, it indicates that there is a serious occlusion problem. For example, if the P value is continuously higher than 60 when picking fruits and vegetables with dense leaves, it indicates that the occlusion level is too high, resulting in the inability of the robot's visual system to effectively identify the fruits and vegetables. Solutions include improving the visual system or adjusting the robot's motion path to better handle high occlusion environments.

[0173] Analysis of Wz 2 Impact: If the occlusion position information Wz is detected 2There is a significant impact on efficiency, such as the P value of several consecutive measurements exceeding 60, and the occlusion position changes frequently, indicating that the path planning needs to be redesigned; for example, the robot arm needs to adopt more flexible path planning to adapt to the frequently changing occlusion situation, or increase the ability to recognize occlusions to adjust the path more accurately;

[0174] Embodiment five:

[0175] Further explanation based on Example 4:

[0176] In order to verify and demonstrate the effectiveness of the invention, this embodiment is designed to test three efficiency intervals: high efficiency interval (0, 10), medium efficiency interval (10, 50), and low efficiency interval (50, +∞); these tests will be conducted for different environmental factors and types of fruits and vegetables;

[0177] Experimental preparation

[0178] Equipment and Materials: Use an advanced harvesting robot with multiple sensors and an adjustable robotic arm; prepare multiple types of fruits and vegetables, including soft and hard fruits and vegetables, and different densities and positions of obstructions such as leaves and branches;

[0179] Environment settings: Create multiple control environments to simulate different lighting and wind speed conditions; for example, set up an indoor environment where the lighting can be adjusted from 200 loads to 1000 loads; the wind speed can vary from 0 to 20km / h;

[0180] Data collection system: Prepare an advanced data recording system to record in real time the values ​​of the picking evaluation model P, environmental factors f(i), such as light intensity, temperature and wind speed, as well as the type and characteristics of fruits and vegetables LXTz, and the amplitude of branch and leaf movement YDFd;

[0181] Implementation process

[0182] High efficiency interval test: Set the light and temperature within the normal range, such as 500-700 loads of light; start the picking operation and record the P value; during the test, suddenly change the light to 800 loads, observe and record the reaction of the robot arm and the change of P value;

[0183] Medium efficiency range test: adjust the light and temperature settings to keep the P value between 10 and 50; introduce fruits and vegetables of different hardness, and simulate changes in wind speed, such as increasing from 5km / h to 15km / h; record the adjustment of the robot arm and the change in P value;

[0184] Low efficiency interval test: Increase the density and complexity of the occlusions to make the P value exceed 50; record the occlusion degree ZDCs and obtain the following experimental data table

[0185] The following table shows the specific data of the experiment:

[0186] Test subjects Light intensity load (unit) Wind speed (km / h) Fruit and vegetable types Initial P value Adjusted P value Lighting changes 500 5 ordinary 9 9 Lighting changes 800 0 ordinary 9 10 Wind speed changes 600 6 ordinary 45 48 Wind speed changes 520 16 Soft 45 49 Occlusion 570 13 Dense leaves 55 55 Occlusion 500 6 Dense leaves 55 60 Blocking position 560 7 Variation Occlusion 65 65 Blocking position 680 12 Variation Occlusion 65 70 Control Group 500 5 ordinary 9 9

[0187] Table 1

[0188] Data analysis, by comparing the experimental data in the above table, we can clearly see the effectiveness and beneficial effects of the invention:

[0189] Adaptability to light changes: When the light intensity gradually increases from 500 loads to 800 loads, for the picking of ordinary fruits and vegetables, the initial P value is 9, indicating that the robotic arm is in a high efficiency range; when the light intensity suddenly increases, the P value increases from 9 to 10. Although it increases slightly, it is still in the high efficiency range, indicating that the robotic arm can effectively adapt to light changes and maintain a high picking efficiency. This shows that the light change adaptation mechanism in the invention can ensure that the robotic arm can still maintain high picking efficiency when the ambient light changes significantly;

[0190] Adjustment mechanism for wind speed changes: When the wind speed increased from 5km / h to 15km / h, the P value increased from 45 to 48 when processing soft fruits and vegetables; although there was an increase, it was still within the medium efficiency range; this shows that even under the influence of wind speed changes, the system is able to adapt to environmental changes and maintain a medium level by adjusting path planning and grasping strategies.

[0191] Embodiment six:

[0192] Further explanation based on the first embodiment, the first picking information M is obtained. 1 , the second picking information M 2 and the third picking information M 3 After that, the branch and leaf occlusion evaluation index ZDZs is generated, and the comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion during the picking motion planning, and generate the planning path H2 and the planning path H3 in turn, which specifically include the following contents:

[0193] Get the first picking information M 1 , the second picking information M 2 and the third picking information M 3 Generate a branch and leaf occlusion evaluation index ZDZs, and set a comparison threshold U of the branch and leaf occlusion evaluation index ZDZs. The set threshold U of the branch and leaf occlusion evaluation index ZDZs is used to determine whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables when it moves;

[0194] The calculation formula of the branch and leaf shading evaluation index ZDZs is as follows:

[0195]

[0196] Among them, φ(M 1 ,M 2 ,M 3 ,t) is an integral function based on the picking information, which represents the temporal dynamic change of occlusion during the picking process, and its form is:

[0197] φ(M 1 ,M 2 ,M 3 ,t)=λ 1 ·M 1 (t)+λ 2 ·sin(M 2 (t))+λ 3 ·cos(M 3 (t))

[0198] ω i is the normalized weight, F i (H2, H3) is a complex information filtering function based on path planning, which represents the occlusion situation under different path planning; this function can be specifically expressed as:

[0199]

[0200] Among them, M 1 , M 2 , M 3 Represent the first, second and third picking information respectively; t represents the time variable; T is the total duration of the picking process; A is the set threshold; λ 1 ,λ 2 ,λ 3 is the adjustment coefficient; (H2, H3) represents two different path plans; N is the number of path plans; ò is a small amount to prevent the denominator from being zero;

[0201] The value range of ZDZs is set to (0, U). When ZDZs is closer to U, the movement of the fruit and vegetable picking robot arm in the occlusion environment is less affected, and the optimized planning path H2 is generated. The path planning ignores the occlusion of branches and leaves and takes the shortest path trajectory of mature fruits and vegetables, that is, the shortest path is selected to directly pass through the branch and leaf area.

[0202] If the branch and leaf occlusion evaluation index ZDZs is closer to 0, it is judged that the movement of the fruit and vegetable picking robot arm in the occlusion environment is greatly affected, and it is more inclined to choose a path that avoids the occlusion, and generate the optimized planning path H3. Then, the path planning focuses on the minimum contact points of the branch and leaf occlusion. For example, there are several staggered branch and leaf occlusions on the picking path of the fruit that can be picked. Based on the picking evaluation model, the path with the least contact with the branch and leaf occlusion is designed to reduce the probability of the fruit and vegetable picking robot arm being stuck by the branches and leaves when turning;

[0203] Further explanation: the (0,U) interval is divided into ten equal parts, and the partitions from small to large are Q1, Q2...Q10. Only when the ZDZs value is in the range of Q9 and Q10, the path planning ignores the occlusion of branches and leaves and performs the shortest path trajectory of mature fruits and vegetables, that is, the shortest path is selected to directly pass through the branch and leaf area, and the corresponding planning path H2 is generated. The remaining range intervals generate the planning path H3.

[0204] Characters explained:

[0205] e is the base of natural logarithms;

[0206] dt) is the integral about time, which represents the cumulative effect of occlusion over time;

[0207] φ is the integral function based on the picking information;

[0208] is a summation function used to weight the information filtering results of different path planning; ω i is the normalized weight.

[0209] Embodiment seven:

[0210] Further explanation based on Example 6:

[0211] This embodiment aims to verify the application effect of the constructed branch and leaf occlusion evaluation index ZDZs on the fruit and vegetable picking robot arm in agricultural automated picking; the experiment is divided into two parts: the first part is to collect the first, second and third picking information M 1 , M 2 , M 3 and calculate ZDZs; the second part is to use ZDZs to guide the path planning of the robot arm, implement the picking task, and record the picking efficiency and success rate;

[0212] Before the experiment began, a fruit and vegetable picking robot equipped with an external visual sensor and drone monitoring was prepared; the experimental area selected a branch and leaf occlusion environment of different densities and heights to simulate real agricultural picking conditions; the first, second, and third picking information were obtained through the real-time monitoring system, sensor data, and historical picking records;

[0213] To calculate ZDZs, first use the formula φ(M 1 ,M 2 ,M 3,t) Integrate the occlusion information during the picking process, and then calculate the occlusion evaluation index through the ZDZs formula; set the threshold A according to the ZDZs value to determine the path planning strategy of the picking robot; when the ZDZs value is high, the robot will choose the shortest path through the branch area; when the ZDZs value is low, the robot will try to avoid the occlusion of branches and leaves;

[0214] In the experiment, the robot arm performed the picking task according to the ZDZs value, while monitoring the frequency of unexpected situations (such as being stuck by branches and leaves). Every time an unexpected situation occurred, the robot arm would initiate an emergency response mechanism, including reverse movement, occlusion analysis, and path replanning. The experiment recorded the picking efficiency and success rate of the robot arm under different ZDZs values.

[0215] The test form is as follows:

[0216]

[0217] Table 2

[0218] Tabular data analysis

[0219] The experimental data include the picking efficiency, picking success rate and emergency response times under different ZDZs values; through the analysis of these data, we can clearly see the beneficial effects of the invention;

[0220] The relationship between ZDZs value and picking efficiency and success rate:

[0221] Improved efficiency and success rate: When the ZDZs value increases, such as in Experiments 5, 6, and 7, the picking efficiency refers to the number of fruits and vegetables picked per unit time, and the picking success rate refers to the ratio of successfully picked fruits and vegetables to the total fruits and vegetables. The picking efficiency and picking success rate are significantly improved, which indicates that under high ZDZs values, the robot arm can more effectively traverse the branch and leaf area, reducing the time of detours and searching for fruits that can be picked;

[0222] Challenges of low ZDZs values: On the contrary, when the ZDZs value is low, such as in Experiments 1, 2, and 9, the picking efficiency and success rate are reduced. This is because the low ZDZs value means that the occlusion of branches and leaves has a greater impact on the movement of the robot arm, resulting in more occlusions to be avoided in path planning, which increases the difficulty and time of picking;

[0223] The relationship between emergency response times and ZDZs value:

[0224] Reduced emergency responses: In the case of higher ZDZs values, such as in Tests 5, 6, and 7, the number of emergency responses was significantly reduced, even to zero, which means that the robot arm encountered fewer obstacles during the picking process, resulting in fewer instances of getting stuck by branches and leaves;

[0225] High-risk low ZDZs values: In the case of low ZDZs values, such as experiments 1, 2, and 9, the number of emergency responses increased, reflecting that the robot arm encountered obstacles and got stuck by branches more frequently during the picking process, which further verified the complexity and risk of path planning under low ZDZs values;

[0226] Verification of the invention by experimental data:

[0227] Optimization of path planning: Through experimental data, it can be seen that the ZDZs value effectively guides the path planning of the fruit and vegetable picking robot arm. When the ZDZs value is high, the robot arm can directly pass through the branch area and choose the shortest path, thereby improving the picking efficiency and success rate. This reflects the innovation of the invention content in optimizing the path planning;

[0228] Dynamic responsiveness: The reduction in emergency response times demonstrates the robotic arm’s ability to respond efficiently in real-time dynamic environments.

[0229] Embodiment eight:

[0230] Further explanation based on the first embodiment, the continuous acquisition of environmental change data of the picking robot arm during the picking process and judging whether new occlusion occurs or the existing occlusion state changes to determine whether to trigger the emergency response mechanism specifically includes the following contents:

[0231] During execution, external visual sensors and drones are continuously used to monitor environmental changes, such as the appearance of new occlusions or changes in the status of existing occlusions;

[0232] If the robot encounters an accident during execution, such as being stuck by branches and leaves, the system will trigger an emergency response mechanism, which includes:

[0233] Reverse movement: The robot arm tries to move back slightly to get out of the stuck state;

[0234] Occlusion analysis: Re-evaluate occluded areas to determine if additional mechanical actions, such as rotation or lateral movement, are required to avoid obstacles;

[0235] Path replanning: Dynamically replan the path based on new occlusion data to find a new optimal path.

[0236] The above formulas are all dimensionless and numerical calculations. The formula is a formula obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0237] Specific, ongoing environmental monitoring:

[0238] Sensor deployment: The robotic arm is equipped with various sensors, such as visual sensors, proximity sensors, and tactile sensors, for real-time monitoring of the surrounding environment;

[0239] Data collection: Sensors continuously collect environmental data, including the location, size, and movement status of obstacles;

[0240] Occlusion judgment: By analyzing the collected data, the system can determine whether new occlusions have appeared or whether the status of existing occlusions has changed.

[0241] Emergency response mechanism:

[0242] Reverse Movement When the robot detects that it may be stuck or is approaching a dangerous area, it will slightly back up to escape the potential risk.

[0243] This action prevents the robot arm from getting damaged or damaging surrounding objects;

[0244] The occlusion analysis system re-evaluates the occluded area to determine if additional mechanical actions, such as rotation or lateral movement, are required to avoid the obstacle.

[0245] The analysis is based on the latest sensor data and pre-set algorithms;

[0246] Path Replanning Based on the new occlusion data, the system dynamically replans the path of the robot arm to find a new optimal path to the target location; this includes adjusting the movement trajectory or changing the picking order.

[0247] When the robotic arm itself fails, such as joint jamming or sensor failure, the emergency response should also include automatic shutdown and alarm.

[0248] Characteristics of different crops: Adjust the strategies of occlusion analysis and path planning according to the differences of crops, such as size, shape, and vulnerability.

[0249] Embodiment nine:

[0250] A picking robot arm motion planning system, the system is used to execute the motion planning method, comprising:

[0251] The first picking information establishment module: obtains historical picking data and botanical database of the area to be picked, and uses them to identify key information on the growth cycle and maturity characteristics of different fruits and vegetables; collects information on mature fruits and vegetables on the overall moving path of the picking robot arm in the area to be picked, and generates the first picking information M 1 ,

[0252] The second picking information establishment module: The drone equipped with a visual recognition camera collects visual image data of the picking area from the air, and adjusts the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables hidden by the branches and leaves, and generates a large range of second picking information M for picking mature fruits and vegetables. 2 ; Get the first picking information M 1 With the second picking information M 2 Then, the information of mature fruits and vegetables is compared and analyzed to generate the information data of mature fruits and vegetables that are blocked by branches and leaves;

[0253] Picking evaluation model building module: collect the surrounding environment information of mature fruits and vegetables blocked by branches and leaves and generate the third picking information M 3 ; Fusion of the first picking information M 1 , the second picking information M 2 and the third picking information M 3 The collected data in is analyzed and processed to generate a picking evaluation model P, and a value range set of the picking evaluation model P is determined. Based on the value range set of the picking evaluation model P, a pre-picking path between the robot arm and the target fruit is planned to generate an initial planning path H1;

[0254] Branch and leaf occlusion evaluation index establishment module: obtain the first picking information M 1 , the second picking information M 2 and the third picking information M 3 After that, the branch and leaf occlusion evaluation index ZDZs is generated, and the comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during the picking motion planning, and the planning path H2 and the planning path H3 are generated in turn;

[0255] Emergency response mechanism feedback module: continuously obtains environmental change data of the picking robot arm during the picking process, sets an emergency response mechanism for the picking robot arm to escape from a stuck state, and determines whether new occlusions appear or the existing occlusion status changes to determine whether to trigger the emergency response mechanism.

[0256] The specific values ​​of δ, ε, ∈, σ, etc. in the formula are generally determined by those skilled in the art according to actual conditions. The essence of the formula of the present application is to perform comprehensive analysis by weighted summation. Those skilled in the art collect multiple groups of sample data and set corresponding preset proportional coefficients for each group of sample data; substitute the preset proportional coefficients and the collected sample data into the formula, any four formulas constitute a set of four-variable linear equations, screen the calculated coefficients and take the average, and obtain the values ​​of δ, ε, ∈, σ, etc.;

[0257] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by computer software, electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution.

[0258] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A picking robot arm motion planning method, characterized in that: The specific steps include: S1, obtaining historical picking data and botanical database of the area to be picked, and using them to identify key information of growth cycles and maturity characteristics of different fruits and vegetables; collecting information of mature fruits and vegetables on the overall moving path of the picking robot arm in the area to be picked, and generating first picking information M1, S2, using a drone equipped with a visual recognition camera to collect visual image data of the area to be picked in the air, and adjusting the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables blocked by branches and leaves, and generating second picking information M2 for picking mature fruits and vegetables in a large range; after obtaining the first picking information M1 and the second picking information M2, performing a comparison analysis of the mature fruits and vegetables information, and generating information data of mature fruits and vegetables blocked by branches and leaves; S3, collecting the surrounding environment information of the mature fruits and vegetables blocked by branches and leaves and generating the third picking information M3; integrating the collected data in the first picking information M1, the second picking information M2 and the third picking information M3 and generating a picking evaluation model P after analyzing and processing, and determining the value range set of the picking evaluation model P, and planning the pre-picking path between the robot arm and the target fruit based on the value range set of the picking evaluation P, and generating an initial planning path H1; S4. After obtaining the first picking information M1, the second picking information M2 and the third picking information M3, a branch and leaf occlusion evaluation index ZDZs is generated, and a comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to determine whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to perform fruit and vegetable picking during the picking motion planning, and a planning path H2 and a planning path H3 are generated in turn; S5. Continuously obtain data on environmental changes of the picking robot arm during the picking process, set an emergency response mechanism for the picking robot arm to escape from a stuck state, and determine whether new obstructions appear or the existing obstruction status changes to determine whether to trigger the emergency response mechanism.

2. The picking robot arm motion planning method according to claim 1, characterized in that: The acquisition of historical harvesting data and a botanical database of the area to be harvested is used to identify key information on the growth cycle and maturity characteristics of different fruits and vegetables, specifically including the following: The historical harvesting data includes the date, time, yield, and variety of fruit and vegetable harvesting in the area to be harvested in the past; The botanical database contains the growth cycles and maturity characteristics of different fruits and vegetables, where the maturity characteristics are specifically color, size, shape, and the best time to pick them; The picking robot arm collects information on mature fruits and vegetables on the overall moving path of the area to be picked, and generates first picking information M1, which specifically includes the following contents: The first picking information M1 is data obtained by an external visual sensor, and specifically includes the following collected data: The location and maturity data of fruits and vegetables are captured by external visual sensors. The maturity information is calibrated with CSd. The maturity information CSd is obtained by collecting the color C1, size C2, and shape C3 data of fruits and vegetables in the image and then calculating the function. The location information Wz1 includes the GPS coordinates of latitude, longitude, and altitude. Fruit and vegetable type and characteristic data, based on the type characteristics LXTz of different fruits and vegetables, record their specific growth cycle and maturity characteristics; The current position and status data of the robot arm, including the coordinates Zb and direction Fx of the robot arm, and the status ZT data of the task currently being executed; The formula for the first picking information M1 is: in Respectively represent natural logarithm, square root, sine, and exponential functions; (α, β) are adjustment parameters; Wz1=(GPS lat ,GPS lon ,GPSalt) Merge the elements of Wz1 into a scalar value. Specifically, use the average value of longitude, latitude, and altitude to represent the scalar value of Wz1. The calibration formula is as follows: Among them, Wz1 represents the precise location information of fruits and vegetables, including longitude GPS lat , latitude GPS lon , and altitudeGPSalt; CSd=k1·C1+k2·C2+k3·C3 CSd represents maturity information, which is a function based on the color C1, size C2 and shape C3 of fruits and vegetables. k1, k2, and k3 are weight parameters set according to actual conditions, which are used to adjust the contribution of color, size and shape to maturity assessment.

3. The picking robot arm motion planning method according to claim 1, characterized in that: The method uses a drone equipped with a visual recognition camera to collect visual image data of the area to be picked in the air, and adjusts the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables blocked by branches and leaves, and generates second picking information M2 for picking mature fruits and vegetables in a large range. After obtaining the first picking information M1 and the second picking information M2, a comparison analysis of the mature fruits and vegetables information is performed to generate information data of mature fruits and vegetables blocked by branches and leaves, which specifically includes the following contents: Monitor the impact of the wind generated by the rotating drone rotors on the surrounding vegetation; adjust the drone's flight altitude and angle, or adjust the rotor speed to generate appropriate wind to expose the fruits and vegetables hidden by branches and leaves; While adjusting the wind direction, the visual recognition camera on the drone continues to capture images and identify fruits and vegetables exposed by the wind. The collected image data is sent to the processing center for image analysis and identification of mature fruits and vegetables. The second picking information M2 specifically includes the following collected data: The wide-area fruit and vegetable distribution data represents the wide-area distribution map of the area to be picked provided by the drone, including the group distribution QTFb and row spacing HJj of fruits and vegetables; The drone reveals the information data of fruits and vegetables blocked by branches and leaves through the wind generated by its rotor. The specific data includes the blocking position information, the characteristics of the visible part, the blocking degree, the fruit and vegetable group information, and the branch and leaf movement data. The data are calibrated in turn to form the blocking position information Wz2, the characteristics of the visible part KJBf, the blocking degree ZDCs, the fruit and vegetable group information GSQt, and the branch and leaf movement data YDSj: The blocking position information Wz2 indicates the latitude, longitude and altitude of the blocked fruits and vegetables in the farmland; The feature KJBf of the visible part represents the feature of the visible area after the fruits and vegetables are blocked. The features of the visible area include D1 color, D2 size, and D3 shape; The occlusion degree ZDCs indicates the degree to which fruits and vegetables are obscured. The fruit and vegetable group information GSQt represents the data of multiple fruits and vegetables in the area being blocked; The branch and leaf movement data YDSj represents the movement of branches and leaves caused by the wind force of the drone rotor, including the movement amplitude YDFd and frequency f of the branches and leaves; The formula for the second picking information M2 is: Among them, (e x ,tanh(x),Γ(x),x y ,ln(x)) represent exponential, hyperbolic tangent, gamma function, power function and natural logarithm respectively; QTFb represents the wide-area distribution of fruit and vegetable groups; HJj represents the row spacing in the farmland; The formula for the position information Wz2 of the obscured fruits and vegetables is: Wz2=(GPS lat2 ,GPS lon2 ,GPS alt2 ) Combine the elements of Wz2 into a scalar value. Specifically, use the average value of longitude, latitude, and altitude to represent the scalar value of Wz2. The calibration formula is as follows: Among them, Wz2 represents the precise location information of fruits and vegetables, including longitude GPS lat2 , latitude GPS lon2 , and altitude GPS alt2 The characteristic KJBf formula of the visible part of fruits and vegetables is: KJBf=k4·D1+k5·D2+k6·D3 Among them, color D1, size D2, shape D3, k4, k5, and k6 are weight parameters set according to actual conditions.

4. The picking robot arm motion planning method according to claim 1, characterized in that: The collecting of the surrounding environment information of the ripe fruits and vegetables blocked by the branches and leaves and generating the third picking information M3 specifically includes the following contents: The third picking information M3 includes integrated meteorological data QXSj, light condition data GZTj, vegetation density ZBMd and historical picking data LSCz: The integrated meteorological data QXSj specifically collects the temperature Wd, humidity Sd, wind speed Fs, and light condition GZTj data around the mature fruits and vegetables that are blocked by branches and leaves; The light condition data GZTj include sunshine duration t1 and sun position tw; The third picking information M3 formula is: in, They represent the standard normal distribution function, the cube root, and the logarithm of a with base b respectively; The step of fusing the collected data in the first picking information M1, the second picking information M2 and the third picking information M3 and analyzing and processing the data to generate the picking evaluation model P, and determining the value range set of the picking evaluation model P, specifically includes the following contents: Set the picking evaluation model P as: in: Wz1: represents the precise location information of fruits and vegetables; CSd: represents maturity information, which is calculated from the color C1, size C2, and shape C3 of fruits and vegetables; LXTz: represents fruit and vegetable types and characteristics; Zb, Fx, ZT: represent the coordinates, direction and current task status of the robot arm respectively; M1, M2, M3: represent the first, second and third picking information respectively; f(i) is an advanced integral function, representing a comprehensive analysis of environmental factors; YDFd and GSQt represent the movement amplitude of branches and leaves and the information of fruit and vegetable population, respectively; QTFb, HJj, Wz2, KJBf, and ZDCs represent the distribution of fruit and vegetable groups, row spacing, occlusion position information, characteristics of the visible part, and occlusion degree, respectively; The advanced integral function f(i) is defined as:

5. The picking robot arm motion planning method according to claim 4, characterized in that: The value range of the formula P is (0, +∞), and different values ​​represent different picking efficiencies and path optimization degrees; A lower P value indicates higher picking efficiency and more optimized path planning, while a higher P value means that the picking strategy or path planning needs to be improved; The pre-picking path between the robot arm and the target fruit is planned based on the value range set of the picking evaluation P to generate an initial planning path H1, which specifically includes the following contents: Set the interval values ​​A and B of the picking evaluation model P, and A is less than B, and divide the value interval of P into high efficiency interval (0, A), medium efficiency interval (A, B) and low efficiency interval (B, +∞); Among them, the high efficiency interval (0, A) represents very high picking efficiency and optimized path. The path planning principle is to respond quickly to significant changes in environmental factors f(i) and generate the initial planning path H1; The medium efficiency interval (A, B) indicates that the picking efficiency is at a medium level. The path planning principle is to set up a feedback mechanism to analyze whether the efficiency decreases due to the fruit and vegetable type characteristics LXTz or the branch and leaf movement amplitude YDFd, and make corresponding adjustments to generate the initial planning path H1; The low efficiency interval (B, +∞) indicates that the picking efficiency is at a low level. The path planning principle is to set up a deep analysis process and focus on analyzing the influence of the occlusion degree ZDCs and the occlusion position information Wz2. If the measurement results are in this interval for several consecutive times, the picking strategy or path planning is comprehensively redesigned to generate the initial planning path H1.

6. The picking robot arm motion planning method according to claim 1, characterized in that: After obtaining the first picking information M1, the second picking information M2 and the third picking information M3, a branch and leaf occlusion evaluation index ZDZs is generated, and a comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during the picking motion planning, and the planned path H2 and the planned path H3 are generated in turn, which specifically include the following contents: The first picking information M1, the second picking information M2 and the third picking information M3 are obtained and a branch and leaf occlusion evaluation index ZDZs is generated, and a comparison threshold A of the branch and leaf occlusion evaluation index ZDZs is set. The threshold A of the branch and leaf occlusion evaluation index ZDZs is used to determine whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables when moving; The calculation formula of the branch and leaf shading evaluation index ZDZs is as follows: Among them, φ(M1,M2,M3,t) is an integral function based on picking information, which represents the temporal dynamic change of occlusion during the picking process, and its form is: φ(M1,M2,M3,t)=λ1·M1(t)+λ2·sin(M2(t))+λ3·cos(M3(t)) ω i is the normalized weight, F i (H2, H3) is a complex information filtering function based on path planning, which represents the occlusion situation under different path planning; this function can be specifically expressed as: Among them, M1, M2, M3 represent the first, second, and third picking information respectively; t represents the time variable; T is the total duration of the picking process; A is the set threshold; λ1, λ2, λ3 are adjustment coefficients; (H2, H3) represent two different path planning; N is the number of path planning; ò is a small amount to prevent the denominator from being zero; The value range of ZDZs is set to (0, A). When ZDZs is closer to A, the movement of the fruit and vegetable picking robot arm in the occlusion environment is less affected, and the optimized planning path H2 is generated. The path planning ignores the occlusion of branches and leaves and takes the shortest path trajectory of mature fruits and vegetables, that is, the shortest path is selected to directly pass through the branch and leaf area; If the branch and leaf occlusion evaluation index ZDZs is closer to 0, it is judged that the movement of the fruit and vegetable picking robot arm in the occlusion environment is greatly affected, and it is more inclined to choose a path that avoids the occlusion, and generate the optimized planning path H3. In this way, the path planning focuses on the minimum touch points of branch and leaf occlusion, reducing the probability of the fruit and vegetable picking robot arm being stuck by branches and leaves when turning; Characters explained: e is the base of natural logarithms; dt) is the integral about time, which represents the cumulative effect of occlusion over time; φ is the integral function based on the picking information; is a summation function used to weight the information filtering results of different path planning; ω i is the normalized weight.

7. The picking robot arm motion planning method according to claim 1, characterized in that: The continuously acquiring the environmental change data of the picking robot arm during the picking process and judging whether there is a new occlusion or the existing occlusion state changes to determine whether to trigger the emergency response mechanism specifically includes the following contents: Emergency response mechanisms include: Reverse movement: The robot arm tries to move back slightly to get out of the stuck state; Occlusion analysis: Re-evaluate occluded areas to determine if additional mechanical actions, such as rotation or lateral movement, are required to avoid obstacles; Path replanning: Dynamically replan the path based on new occlusion data to find a new optimal path.

8. A picking robot arm motion planning system, characterized in that: The system is used to execute the motion planning method according to any one of claims 1 to 7, comprising: The first picking information establishment module: obtains historical picking data and botanical database of the area to be picked, and uses them to identify key information on the growth cycle and maturity characteristics of different fruits and vegetables; collects information on mature fruits and vegetables on the overall moving path of the picking robot arm in the area to be picked, and generates the first picking information M1. Second picking information establishment module: Use a drone equipped with a visual recognition camera to collect visual image data of the picking area from the air, and adjust the wind direction generated by the rotation of the drone's rotor to reveal the fruits and vegetables blocked by branches and leaves, and generate second picking information M2 for picking mature fruits and vegetables in a large range; After obtaining the first picking information M1 and the second picking information M2, compare and analyze the mature fruits and vegetables information to generate information data of mature fruits and vegetables blocked by branches and leaves; Picking evaluation model construction module: collect the surrounding environment information of mature fruits and vegetables blocked by branches and leaves and generate the third picking information M3; generate the picking evaluation model P after integrating the collected data in the first picking information M1, the second picking information M2 and the third picking information M3 and analyzing and processing them, and determine the value range set of the picking evaluation model P, plan the pre-picking path between the robot arm and the target fruit based on the value range set of the picking evaluation P, and generate the initial planning path H1; Branch and leaf occlusion evaluation index establishment module: after obtaining the first picking information M1, the second picking information M2 and the third picking information M3, the branch and leaf occlusion evaluation index ZDZs is generated, and the comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is set. The comparison threshold U of the branch and leaf occlusion evaluation index ZDZs is used to judge whether the fruit and vegetable picking robot arm can directly ignore the branch and leaf occlusion to pick fruits and vegetables during the picking motion planning, and the planning path H2 and the planning path H3 are generated in turn; Emergency response mechanism feedback module: continuously obtains environmental change data of the picking robot arm during the picking process, sets an emergency response mechanism for the picking robot arm to escape from a stuck state, and determines whether new occlusions appear or the existing occlusion status changes to determine whether to trigger the emergency response mechanism.

Images