Automatic roxburgh rose picking method and device based on RGB image detection

Abstract

The invention discloses an automatic roxburgh rose picking method and device based on RGB image detection, and the method comprises the steps: 1), judging the growth state of roxburgh roses, and marking a target frame of the roxburgh roses as a target image of the roxburgh roses to be picked if the growth state of the roxburgh roses is mature; 2) calculating a central point image coordinate of the mth target image of the roxburgh rose to be picked; 3) converting the central point image coordinate into a central point geographic coordinate based on a conversion relation between the image coordinate and the geographic coordinate, generating a roxburgh rose picking signal, and completing picking; the device comprises a first driving motor, a first crawler belt, a second driving motor, a second crawler belt, a frame, a picking frame, an industrial control host, a six-axis robot, transmission wheels, a camera, a laser radar, a pneumatic soft claw and pneumatic scissors. The method avoids the subjective influence of the maturity observed by naked eyes, is high in recognition precision, is high in discrimination speed, and can meet the requirements of real-time picking.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to an automatic roxburghii picking method and device based on RGB image detection. Background Art

[0002] The current traditional methods of picking roxburghii, such as hand picking and pole bucket picking, are prone to cause different degrees of damage to the roxburghii, leading to rot during storage and sales. In addition, traditional roxburghii picking is a labor-intensive and time-consuming process. Therefore, it is of great significance to develop an automatic picking device that can operate reliably in natural scenes to replace manual picking.

[0003] When picking, over-ripe pears are not easy to preserve, causing the pears to rot and deteriorate when transported to the sales area, causing great losses to fruit farmers. Unripe pears have a poor taste and are not easy to sell.

[0004] Therefore, when the automatic picking device is picking, it is also necessary to accurately identify the maturity of the sea buckthorn and pick the sea buckthorn that meets the requirements. Summary of the invention

[0005] The object of the present invention is to provide a method for automatically picking roxburghii based on RGB image detection, comprising the following steps:

[0006] 1) Obtain the map of the roxburghii orchard and store it in the industrial control host;

[0007] The industrial control host (6) divides the roxburghii orchard into multiple picking areas D = {D1, D2, ..., D n}, and set the initial position point D of the automatic picking device of roxburghii in each picking area i0 and the end point D if , generate the initial position point D i0 To the end point D if The driving trajectory d i,0-f ;D n is the nth picking area in the roxburghii orchard; i = 1, 2, ..., n; n is the number of picking areas;

[0008] 2) The industrial control host drives the automatic picking device of the roxburghii to the initial position D of the i-th picking area. i0 =(x i0 ,y i0 ); x i0 ,y i0 is the initial position point D i0 The horizontal and vertical coordinates of; the initial value of i is 1;

[0009] Arrived at the initial position D i0 =(x i0 ,y i0) after which the industrial control host retrieves the driving trajectory d i,0-f , according to the driving trajectory d i,0-f Driving the automatic prickly pear picking device to move;

[0010] 3) During the driving process, the camera of the automatic pear picking device periodically captures the RGB image P of the a-th side of the automatic pear picking device a , and transmit it to the industrial control host;

[0011] The industrial control host converts the a-th side RGB image P a Input into the roxburghii fruit tree recognition model to identify whether there is a roxburghii fruit tree at the a-th side. If yes, control the roxburghii automatic picking device to stop moving forward and enter step 4); if not, jump to step 13); the initial value of g is 1;

[0012] The a-th side RGB image P a As shown below:

[0013]

[0014] In the formula, (R MN ,G MN ,B MN ) is the RGB image P at the ath side a The R, G, and B components of the middle pixel (M, N);

[0015] The construction steps of the roxburghii fruit tree identification model are as follows:

[0016] 3.1) Obtain no more than 20 images containing roxburghii fruit trees, and use LabelImg software to mark the target frames of the roxburghii fruit trees as a small sample training set;

[0017] 3.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training data set with target boxes of roxburghii fruit trees;

[0018] 3.3) Using keras as the deep learning framework and YOLOv5 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii fruit tree recognition model;

[0019] 4) Start the laser radar;

[0020] The laser radar detects the straight-line distance H between the automatic picking device for roxburghii and the roxburghii fruit tree at the a-th side, and transmits it to the industrial control host;

[0021] 5) The industrial control host generates a moving signal V1 based on the straight-line distance H = {V 11 ,V 12}, driving the automatic roxburghii picking device to turn so that the automatic roxburghii picking device faces the roxburghii fruit tree at the a-th side;

[0022] Then the industrial control host drives the automatic prickly pear picking device to move forward, so that the straight-line distance between the automatic prickly pear picking device and the prickly pear tree at the a-th side satisfies ΔH1≤H≤ΔH2; ΔH1 is the redundant distance for the six-axis robot to move; ΔH2 is the maximum movement distance of the six-axis robot; V 11 is the turning angle of the automatic picking device for roxburghii, V 12 It is the straight-line travel distance of the automatic prickly pear picking device after turning;

[0023] 6) The camera takes the j-th position image P of the roxburghii fruit tree j , and transmitted to the industrial control host (6); the initial value of j is 1;

[0024]

[0025] In the formula, (R FE ,G FE ,B FE ) is the j-th orientation image P of the roxburghii fruit tree j The R, G, and B values ​​of the middle pixel (F, E);

[0026] The industrial control host inputs the j-th azimuth image of the roxburghii fruit tree into the roxburghii recognition model, and recognizes the roxburghii target frame in the j-th azimuth image of the roxburghii fruit tree, which is recorded as Q={q1,q2,...,q W}; W is the number of roxburghii target boxes in the j-th orientation image of the roxburghii fruit tree; q W represents the Wth roxburghii target frame in the jth orientation image of the roxburghii fruit tree;

[0027] Among them, the w-th roxburghii target box P w As shown below:

[0028]

[0029] In the formula, The target frame P for the prickly pear w Middle pixel (F w ,E w )'s R, G, B values; w = 1, 2, ..., W;

[0030] The steps for building the roxburghii recognition model are as follows:

[0031] 6.1) Obtain no more than 20 images containing roxburghii, and use LabelImg software to mark the roxburghii target frame as a small sample training set;

[0032] 6.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training dataset with the target box of the roxburghii tree;

[0033] 6.3) Using keras as the deep learning framework and YOLOv3 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii recognition model;

[0034] 7) The industrial control host sets the target frame P of the roxburghii w Input into the sea buckthorn maturity recognition model to determine the growth state of the sea buckthorn in the sea buckthorn target frame. If the growth state of the sea buckthorn is mature, the sea buckthorn target frame is marked as the sea buckthorn target image to be picked; the initial value of w is 1;

[0035] The growth state of the roxburghii includes unripe, ripe and overripe;

[0036] The step of determining the growth state of the roxburghii thorn in the roxburghii thorn target frame comprises:

[0037] 7.1) The industrial control host sets the target frame P of the roxburghii w The RGB components of each point in the image are matched with the pre-stored RGB component threshold ranges of unripe state (90-190, 120-255, 0-80), ripe state RGB component threshold ranges (220-255, 185-255, 0-80), and overripe state RGB component threshold ranges (200-255, 140-185, 0-80). If the RGB component falls into any RGB component threshold range, and R≥G falls into the ripe state RGB component threshold range, and G≥R falls into the unripe state RGB component threshold range, then the point is regarded as a valid point;

[0038] 7.2) Calculate the average value of the RGB component of the unripe state based on the threshold range of the RGB component of the unripe state Based on the threshold range of the RGB component of the mature state, calculate the average RGB value of the unripe state Based on the threshold range of the overripe state RGB component, calculate the average value of the underripe state RGB

[0039] Among them, the RGB average values ​​under different growth conditions of roxburghii are as follows:

[0040]

[0041] In the formula, R t,max , R t,min is the upper and lower limits of the R component threshold corresponding to the growth state t; G t,max , G t,min is the upper and lower limits of the G component threshold corresponding to the growth state t; B t,max , B t,minare the upper and lower limits of the B component threshold corresponding to growth state t; t = 1, 2, 3, representing immature, mature, and overmature, respectively;

[0042] 7.3) respectively calculating the variance S1 between all valid points and the average value of the RGB component threshold of the unripe state, the variance S2 between all valid points and the average value of the RGB component threshold of the ripe state, and the variance S3 between all valid points and the average value of the RGB component threshold of the overripe state;

[0043] Among them, the variance S t As shown below:

[0044]

[0045] Where U uR , U uG , U uB are the R, G, and B components of the u-th valid point;

[0046] 7.4) Compare variance S1, variance S2, and variance S3. If variance S1 is the smallest, then the current target frame P of the roxburghii roxburghii w The growth state of the Chinese prickly ash is unripe. If the variance S2 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly ash is mature. If the variance S3 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly pear is over-mature;

[0047] Marking a target frame of a thorn pear in a mature growth state as a target image of a thorn pear to be picked;

[0048] The threshold ranges of the RGB components for the unripe, ripe, and overripe states are obtained by the following steps:

[0049] s1) 1) obtaining a plurality of roxburghii roxburghii, extracting juice from each roxburghii roxburghii, and analyzing the roxburghii roxburghii juice by high performance liquid chromatography to determine the Vc content;

[0050] s2) dividing the growth status of the roxburghii into unripe, mature and overripe according to the range of Vc content, and labeling each roxburghii with a status label;

[0051] s3) taking RGB images of roxburghii roxburghii in different growth states, and obtaining RGB components of each RGB image of roxburghii roxburghii;

[0052] s4) performing statistics on the RGB components of the RGB images of the roxburghii roxburghii with the same growth state, and taking the overlapping range of the RGB components as the RGB component threshold range of the current growth state;

[0053] 8) Calculate the image coordinates of the center point of the mth target image of the prickly ash to be picked, the steps are as follows:

[0054] 8.1) extracting the outline of the roxburghii fruit in the target image of the roxburghii to be picked based on the threshold range of the three components of RGB in the maturity state;

[0055] 8.2) Binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image;

[0056] 8.3) Consider the foreground binary image of the roxburghii as a circular shape and construct the roxburghii contour equation, namely:

[0057] X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(9)

[0058] Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined;

[0059] 8.4) Taking the minimum root mean square error γ as the goal, fit and solve the coefficients α0, α1, α2, α3, and α4;

[0060] The root mean square error γ is given by:

[0061]

[0062] In the formula, (X c ,Y c ) is the cth pixel on the foreground binary image of the roxburghii; C is the number of pixels;

[0063] 8.5) Calculate the center point image coordinates (X0, Y0), that is:

[0064]

[0065] 9) based on the conversion relationship between the image coordinates and the geographic coordinates, converting the center point image coordinates into the center point geographic coordinates; the center point geographic coordinates include the distance between the automatic picking device for roxburghii roxburghii and the center point of the roxburghii roxburghii to be picked in the three-dimensional direction;

[0066] Based on the geographic coordinates of the center point, the industrial control host generates a prickly pear picking signal and drives the six-axis robot of the prickly pear automatic picking device to move, so that the pneumatic soft claws mounted on the six-axis robot clamp the prickly pear, and then the pneumatic scissors located above the pneumatic soft claws cut the prickly pear stalks;

[0067] After cutting off the roxburghii stalk, the six-axis robot is driven to rotate so that the pneumatic soft claw is located above the picking frame, and the vertical distance between the pneumatic soft claw and the picking frame is less than hmax; then, the pneumatic soft claw is released to allow the roxburghii to fall into the picking frame;

[0068] 10) Let m=m+1, and return to step 8), until all the mature roxburghii thorns at the j-th position of the roxburghii thorn tree are picked;

[0069] 11) Determine whether the automatic pear picking device has circled the current pear tree. If so, proceed to step 12). If not, use the industrial control host to drive the automatic pear picking device to the j+1th orientation image of the pear tree, set j=j+1, and return to step 6);

[0070] 12) Determine whether g=2 is true. If not, drive the six-axis robot to rotate so that the camera is aimed at the g+1th side position, update g=g+1, and return to step 4). If yes, control the automatic picking device of roxburghii to move along the road of the i-th picking area, and return to step 3);

[0071] 13) Determine whether the automatic prickly roxburghii picking device has reached the end position point D of the i-th picking area if =(x if ,y if ), if not, return to step 3), if yes, control the automatic prickly roxburghii picking device to move, so that the automatic prickly roxburghii picking device reaches the initial position point D of the i+1th picking area (i+1)0 =(x (i+1)0 ,y (i+1)0 ), let i=i+1, and return to step 3), until the automatic prickly roxburghii picking device traverses all picking areas of the prickly roxburghii orchard, and the picking task is completed; x if ,y if The end point D if The horizontal and vertical coordinates of (i+1)0 ,y (i+1)0 is the initial position point D (i+1)0 The horizontal and vertical coordinates of .

[0072] Further, the steps of dividing the roxburghii orchard into multiple picking areas are:

[0073] a1) dividing the roxburghii orchard into a plurality of areas with regular shapes according to the roxburghii orchard map;

[0074] a2) each regularly shaped area is divided into a plurality of mutually parallel picking areas, each picking area comprising a roxburghii orchard road and two rows of roxburghii fruit trees planted along the roxburghii orchard road.

[0075] Furthermore, the initial position point D of the first picking area 10 It is the location point closest to the gate of the sea buckthorn orchard and has sea buckthorn trees on both sides;

[0076] The initial position point D of the kth picking area k0 Select the end point D of the k-1th picking area ifThe nearest location point with two side locations of a sea buckthorn tree; k=2, ..., n; n is the number of picking areas.

[0077] Furthermore, the conversion relationship between the center point image coordinates and the geographic coordinates is determined by the following steps:

[0078] b1) selecting a roxburghii from the roxburghii fruit trees, and using a camera of an automatic roxburghii picking device to capture an RGB image of the selected roxburghii;

[0079] b2) extracting the outline of the roxburghii fruit in the selected roxburghii RGB image;

[0080] b3) binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image;

[0081] b4) The foreground binary image of the roxburghii is regarded as a circular shape, and the contour equation of the roxburghii is constructed, namely:

[0082] X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(12)

[0083] Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined;

[0084] b5) Taking the minimum root mean square error γ as the goal, the coefficients α0, α1, α2, α3, and α4 to be determined are fitted and solved;

[0085] The root mean square error γ is given by:

[0086]

[0087] b6) Calculate the center point image coordinates (X0, Y0), that is:

[0088]

[0089] b7) using a laser radar to obtain a three-dimensional distance between the automatic prickly thorn picking device and the selected prickly thorn as a geographic coordinate;

[0090] b8) repeating steps b1) to b7) to obtain multiple sets of training data, each set of training data including the center point image coordinates and the corresponding geographic coordinates;

[0091] b9) Based on the training data, the relationship between the geographic coordinates and the center point image coordinates is fitted to obtain the conversion relationship between the center point image coordinates and the geographic coordinates.

[0092] The automatic prickly pear picking device based on the method comprises a first drive motor, a first crawler, a second drive motor, a second crawler, a frame, a picking frame, an industrial control host, a six-axis robot, a transmission wheel, a camera, a laser radar, a pneumatic soft claw, and a pneumatic scissors.

[0093] The interior of the vehicle frame is used for placing the first drive motor, the second drive motor, and the industrial control host.

[0094] The industrial control host is used to control the operation of the first drive motor, the second drive motor, the six-axis robot, the camera, the laser radar, the pneumatic soft claws, and the pneumatic scissors.

[0095] The first crawler track and the second crawler track are symmetrically installed at both ends of the frame.

[0096] A plurality of transmission wheels are installed inside the first crawler belt and the second crawler belt.

[0097] The first drive motor drives the transmission wheel in the first crawler belt to rotate, thereby driving the first crawler belt to move.

[0098] The second drive motor drives the transmission wheel in the second crawler belt to rotate, thereby driving the second crawler belt to move.

[0099] A picking frame for placing roxburghii roxburghii is fixed on the upper surface of the shell.

[0100] The six-axis robot is connected to the upper surface of the vehicle frame.

[0101] The camera is mounted on a six-axis robot and is used to capture RGB images and transmit them to an industrial control host.

[0102] The laser radar is mounted on the six-axis robot and is used to detect the distance between the six-axis robot and the sea buckthorn tree.

[0103] Furthermore, it also includes a power battery.

[0104] The power battery provides power for the first drive motor, the second drive motor, the industrial control host, the six-axis robot, the camera, and the lidar.

[0105] Furthermore, it also includes a heat dissipation module mounted on the vehicle frame.

[0106] The heat dissipation module is used to dissipate heat for the first drive motor, the second drive motor, and the industrial control host.

[0107] Furthermore, it also includes a gas storage tank.

[0108] The industrial control host controls the valve switch of the gas storage tank, thereby starting or closing the pneumatic soft claws and pneumatic scissors.

[0109] Furthermore, the vehicle frame includes a main frame, an upper guard plate, and a lower guard plate.

[0110] The upper guard plate and the lower guard plate cover the top surface and the bottom surface of the main frame to form a box body.

[0111] Furthermore, a mass sensor is included.

[0112] The quality sensor is mounted on the bottom of the picking frame and is used to monitor the quality of the picking frame and transmit the quality to the industrial control host.

[0113] The industrial control host stores a quality upper limit. When the quality of the picking frame reaches the quality upper limit, the industrial control host controls the automatic roxburgh pear picking device to travel, so that the automatic roxburgh pear picking device reaches the nearest unloading point and unloads the picked roxburgh pears.

[0114] The technical effect of the present invention is unquestionable. Based on the rules of fruit tree planting, the present invention divides the orchard area and plans the driving path of the automatic picking device in each area. There is no need to drive back and forth on the same road repeatedly, which saves energy and shortens the picking time.

[0115] The present invention constructs a roxburgh tree recognition model and a roxburgh recognition model through the YOLOv3 algorithm, and can realize the accurate positioning of the roxburgh tree and the roxburgh.

[0116] The invention uses an RGB image of the roxburghii as a basis, judges the maturity state of the roxburghii by using an RGB channel threshold, and then controls a pneumatic soft claw and a pneumatic scissors to automatically pick the mature roxburghii.

[0117] The invention avoids the subjective influence of observing the maturity with naked eyes, has high recognition accuracy and fast identification speed, and can meet the needs of real-time picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 is the device structure diagram;

[0119] Figure 2 For the main frame;

[0120] Figure 3 For upper and lower guard plates;

[0121] Figure 4 For driving the motor;

[0122] Figure 5 For picking flow chart. DETAILED DESCRIPTION

[0123] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0124] Embodiment 1:

[0125] See also Figures 1 to 5 , a method for automatic picking of roxburghii based on RGB image detection, comprising the following steps:

[0126] 1) Obtaining a map of the roxburghii orchard and storing it in the industrial control host 6;

[0127] The industrial control host 6 divides the roxburghii orchard into multiple picking areas D = {D1, D2, ..., D n}, and set the initial position point D of the automatic picking device of roxburghii in each picking area i0 and the end point D if , generate the initial position point D i0 To the end point D if The driving trajectory d i,0-f ;D n is the nth picking area in the roxburghii orchard; i = 1, 2, ..., n; n is the number of picking areas;

[0128] 2) The industrial control host drives the automatic picking device of the roxburghii to the initial position D of the i-th picking area. i0 =(x i0 ,y i0 ); x i0 ,y i0 is the initial position point D i0 The horizontal and vertical coordinates of; the initial value of i is 1;

[0129] Arrived at the initial position D i0 =(x i0 ,y i0 ) after which the industrial control host retrieves the driving trajectory d i,0-f , according to the driving trajectory d i,0-f Driving the automatic prickly pear picking device to move;

[0130] 3) During the driving process, the camera 9 of the automatic roxburghii picking device periodically captures the RGB image P of the a-th side of the automatic roxburghii picking device. a , and transmit it to the industrial control host 6;

[0131] The industrial control host 6 receives the RGB image P of the a-th side a Input into the roxburghii fruit tree recognition model to identify whether there is a roxburghii fruit tree at the a-th side. If yes, control the roxburghii automatic picking device to stop moving forward and enter step 4); if not, jump to step 13); the initial value of g is 1;

[0132] The a-th side RGB image P a As shown below:

[0133]

[0134] In the formula, (R MN ,G MN ,B MN ) is the RGB image P at the ath side a The R, G, and B components of the middle pixel (M, N);

[0135] The construction steps of the roxburghii fruit tree identification model are as follows:

[0136] 3.1) Obtain no more than 20 images containing roxburghii fruit trees, and use LabelImg software to mark the target frames of the roxburghii fruit trees as a small sample training set;

[0137] 3.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training data set with target boxes of roxburghii fruit trees;

[0138] 3.3) Using keras as the deep learning framework and YOLOv5 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii fruit tree recognition model;

[0139] 4) Start the laser radar;

[0140] The laser radar detects the straight-line distance H between the automatic picking device for roxburghii and the roxburghii fruit tree at the a-th side, and transmits it to the industrial control host 6;

[0141] 5) The industrial control host 6 generates a moving signal V1 based on the straight-line distance H = {V 11 ,V 12}, driving the automatic roxburghii picking device to turn so that the automatic roxburghii picking device faces the roxburghii fruit tree at the a-th side;

[0142] Then the industrial control host 6 drives the automatic prickly pear picking device to move forward, so that the straight-line distance between the automatic prickly pear picking device and the prickly pear tree at the a-th side satisfies ΔH1≤H≤ΔH2; ΔH1 is the redundant distance for the six-axis robot 7 to move; ΔH2 is the maximum movement distance of the six-axis robot 7; V 11 is the turning angle of the automatic picking device for roxburghii, V 12 It is the straight-line travel distance of the automatic prickly pear picking device after turning;

[0143] 6) Camera 9 captures the j-th position image P of the roxburghii fruit tree j , and transmitted to the industrial control host 6; the initial value of j is 1;

[0144]

[0145] In the formula, (R FE ,GFE ,B FE ) is the j-th orientation image P of the roxburghii fruit tree j The R, G, and B values ​​of the middle pixel (F, E);

[0146] The industrial control host 6 inputs the j-th azimuth image of the roxburghii fruit tree into the roxburghii recognition model, and recognizes the roxburghii target frame in the j-th azimuth image of the roxburghii fruit tree, which is recorded as Q={q1,q2,...,q W}; W is the number of roxburghii target boxes in the j-th orientation image of the roxburghii fruit tree; q W represents the Wth roxburghii target frame in the jth orientation image of the roxburghii fruit tree;

[0147] Among them, the w-th roxburghii target box P w As shown below:

[0148]

[0149] In the formula, The target frame P for the prickly pear w Middle pixel (F w ,E w )'s R, G, B values; w = 1, 2, ..., W;

[0150] The steps for building the roxburghii recognition model are as follows:

[0151] 6.1) Obtain no more than 20 images containing roxburghii, and use LabelImg software to mark the roxburghii target frame as a small sample training set;

[0152] 6.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training dataset with the target box of the roxburghii tree;

[0153] 6.3) Using keras as the deep learning framework and YOLOv3 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii recognition model;

[0154] 7) The industrial control host 6 sets the target frame P of the roxburghii w Input into the sea buckthorn maturity recognition model to determine the growth state of the sea buckthorn in the sea buckthorn target frame. If the growth state of the sea buckthorn is mature, the sea buckthorn target frame is marked as the sea buckthorn target image to be picked; the initial value of w is 1;

[0155] The growth states of the roxburghii include unripe (about 50 to 100 days after the peak flowering period), mature (about 100 to 120 days after the peak flowering period) and overripe (about 120 days after the peak flowering period);

[0156] The step of determining the growth state of the roxburghii thorn in the roxburghii thorn target frame comprises:

[0157] 7.1) The industrial control host 6 sets the target frame P of the roxburghii w The RGB components of each point in the image are matched with the pre-stored RGB component threshold ranges of unripe state (90-190, 120-255, 0-80), ripe state RGB component threshold ranges (220-255, 185-255, 0-80), and overripe state RGB component threshold ranges (200-255, 140-185, 0-80). If the RGB component falls into any RGB component threshold range, and R≥G falls into the ripe state RGB component threshold range, and G≥R falls into the unripe state RGB component threshold range, then the point is regarded as a valid point;

[0158] 7.2) Calculate the average value of the RGB component of the unripe state based on the threshold range of the RGB component of the unripe state Based on the threshold range of the RGB component of the mature state, calculate the average RGB value of the unripe state Based on the threshold range of the overripe state RGB component, calculate the average value of the underripe state RGB

[0159] Among them, the RGB average values ​​under different growth conditions of roxburghii are as follows:

[0160]

[0161] In the formula, R t,max , R t,min is the upper and lower limits of the R component threshold corresponding to the growth state t; G t,max , G t,min is the upper and lower limits of the G component threshold corresponding to the growth state t; B t,max , B t,min are the upper and lower limits of the B component threshold corresponding to growth state t; t = 1, 2, 3, representing immature, mature, and overmature, respectively;

[0162] 7.3) respectively calculating the variance S1 between all valid points and the average value of the RGB component threshold of the unripe state, the variance S2 between all valid points and the average value of the RGB component threshold of the ripe state, and the variance S3 between all valid points and the average value of the RGB component threshold of the overripe state;

[0163] Among them, the variance S t As shown below:

[0164]

[0165] Where U uR , U uG , U uB are the R, G, and B components of the u-th valid point;

[0166] 7.4) Compare variance S1, variance S2, and variance S3. If variance S1 is the smallest, then the current target frame P of the roxburghii roxburghii w The growth state of the Chinese prickly ash is unripe. If the variance S2 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly ash is mature. If the variance S3 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly pear is over-mature;

[0167] Marking a target frame of a thorn pear in a mature growth state as a target image of a thorn pear to be picked;

[0168] The threshold ranges of the RGB components for the unripe, ripe, and overripe states are obtained by the following steps:

[0169] s1)1) Obtain a plurality of sea roxburghii fruits and extract juice from each sea roxburghii fruit, and analyze the sea roxburghii fruit juice using high performance liquid chromatography to determine the Vc content; when the fruit is ripe, the Vc content reaches a maximum value (841.58-3541.13 mg / 100 g), while the Vc content of unripe sea roxburghii fruits is approximately 1 / 20 of that of ripe fruits.

[0170] s2) dividing the growth status of the roxburghii into unripe, mature and overripe according to the range of Vc content, and labeling each roxburghii with a status label;

[0171] s3) taking RGB images of roxburghii roxburghii in different growth states, and obtaining RGB components of each RGB image of roxburghii roxburghii;

[0172] s4) performing statistics on the RGB components of the RGB images of the roxburghii roxburghii with the same growth state, and taking the overlapping range of the RGB components as the RGB component threshold range of the current growth state;

[0173] 8) Calculate the image coordinates of the center point of the mth target image of the prickly ash to be picked, the steps are as follows:

[0174] 8.1) extracting the outline of the roxburghii fruit in the target image of the roxburghii to be picked based on the threshold range of the three components of RGB in the maturity state;

[0175] 8.2) Binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image;

[0176] 8.3) Consider the foreground binary image of the roxburghii as a circular shape and construct the roxburghii contour equation, namely:

[0177] X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(9)

[0178] Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined;

[0179] 8.4) Taking the minimum root mean square error γ as the goal, fit and solve the coefficients α0, α1, α2, α3, and α4;

[0180] The root mean square error γ is given by:

[0181]

[0182] In the formula, (X c ,Y c ) is the cth pixel on the foreground binary image of the roxburghii; C is the number of pixels;

[0183] 8.5) Calculate the center point image coordinates (X0, Y0), that is:

[0184]

[0185] 9) based on the conversion relationship between the image coordinates and the geographic coordinates, converting the center point image coordinates into the center point geographic coordinates; the center point geographic coordinates include the distance between the automatic picking device for roxburghii roxburghii and the center point of the roxburghii roxburghii to be picked in the three-dimensional direction;

[0186] Based on the geographic coordinates of the center point, the industrial control host 6 generates a prickly pear picking signal and drives the six-axis robot 7 of the prickly pear automatic picking device to move, so that the pneumatic soft claws 10 mounted on the six-axis robot 7 clamp the prickly pear, and then the pneumatic scissors 11 located above the pneumatic soft claws 10 cut the prickly pear stalks;

[0187] After cutting off the roxburghii stalk, the six-axis robot 7 is driven to rotate so that the pneumatic soft claw 10 is located above the picking frame 5, and the vertical distance between the pneumatic soft claw 10 and the picking frame 5 is less than hmax; then, the pneumatic soft claw 10 is released to allow the roxburghii to fall into the picking frame 5;

[0188] 10) Let m=m+1, and return to step 8), until all the mature roxburghii thorns at the j-th position of the roxburghii thorn tree are picked;

[0189] 11) Determine whether the automatic pear picking device has circled the current pear tree. If so, proceed to step 12). If not, use the industrial control host 6 to drive the automatic pear picking device to the j+1th orientation image of the pear tree, set j=j+1, and return to step 6);

[0190] 12) Determine whether g=2 is established. If not, drive the six-axis robot 7 to rotate so that the camera is aimed at the g+1th side position, update g=g+1, and return to step 4). If yes, control the automatic picking device of roxburghii to move along the road of the i-th picking area, and return to step 3);

[0191] 13) Determine whether the automatic prickly roxburghii picking device has reached the end position point D of the i-th picking area if =(x if ,y if ), if not, return to step 3), if yes, control the automatic prickly roxburghii picking device to move, so that the automatic prickly roxburghii picking device reaches the initial position point D of the i+1th picking area (i+1)0 =(x (i+1)0 ,y (i+1)0 ), let i=i+1, and return to step 3), until the automatic prickly roxburghii picking device traverses all picking areas of the prickly roxburghii orchard, and the picking task is completed; x if ,y if The end point D if The horizontal and vertical coordinates of (i+1)0 ,y (i+1)0 is the initial position point D (i+1)0 The horizontal and vertical coordinates of .

[0192] The steps to divide the sea buckthorn orchard into multiple picking areas are:

[0193] a1) dividing the roxburghii orchard into a plurality of areas with regular shapes according to the roxburghii orchard map;

[0194] a2) each regularly shaped area is divided into a plurality of mutually parallel picking areas, each picking area comprising a roxburghii orchard road and two rows of roxburghii fruit trees planted along the roxburghii orchard road.

[0195] The first picking area initial position point D 10 It is the location point closest to the gate of the sea buckthorn orchard and has sea buckthorn trees on both sides;

[0196] The initial position point D of the kth picking area k0 Select the end point D of the k-1th picking area if The nearest location point with two side locations of a sea buckthorn tree; k=2, ..., n; n is the number of picking areas.

[0197] The conversion relationship between the center point image coordinates and geographic coordinates is determined by the following steps:

[0198] b1) selecting a roxburghii from the roxburghii fruit trees, and using a camera of an automatic roxburghii picking device to capture an RGB image of the selected roxburghii;

[0199] b2) extracting the outline of the roxburghii fruit in the selected roxburghii RGB image;

[0200] b3) binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image;

[0201] b4) The foreground binary image of the roxburghii is regarded as a circular shape, and the contour equation of the roxburghii is constructed, namely:

[0202] X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(12)

[0203] Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined;

[0204] b5) Taking the minimum root mean square error γ as the goal, the coefficients α0, α1, α2, α3, and α4 to be determined are fitted and solved;

[0205] The root mean square error γ is given by:

[0206]

[0207] b6) Calculate the center point image coordinates (X0, Y0), that is:

[0208]

[0209] b7) using a laser radar to obtain a three-dimensional distance between the automatic prickly thorn picking device and the selected prickly thorn as a geographic coordinate;

[0210] b8) repeating steps b1) to b7) to obtain multiple sets of training data, each set of training data including the center point image coordinates and the corresponding geographic coordinates;

[0211] b9) Based on the training data, the relationship between the geographic coordinates and the center point image coordinates is fitted to obtain the conversion relationship between the center point image coordinates and the geographic coordinates.

[0212] Embodiment 2:

[0213] See also Figures 1 to 4 The automatic picking device of sea buckthorn based on the method includes a first driving motor 1, a first crawler 2, a second driving motor 3, a second crawler 4, a frame, a picking frame 5, an industrial control host 6, a six-axis robot 7, a transmission wheel 8, a camera 9, a laser radar, a pneumatic soft claw 10, and a pneumatic scissors 11.

[0214] The interior of the frame is used to place the first drive motor 1 , the second drive motor 3 , and the industrial control host 6 .

[0215] The industrial control host 6 is used to control the operation of the first drive motor 1 , the second drive motor 3 , the six-axis robot 7 , the camera 9 , the laser radar, the pneumatic soft claw 10 , and the pneumatic scissors 11 .

[0216] The first crawler 2 and the second crawler 4 are symmetrically mounted on both ends of the frame.

[0217] A plurality of transmission wheels 8 are installed inside the first crawler belt 2 and the second crawler belt 4 .

[0218] The first drive motor 1 drives the transmission wheel 8 in the first crawler belt 2 to rotate, thereby driving the first crawler belt 2 to move.

[0219] The second drive motor 3 drives the transmission wheel 8 in the second crawler belt 4 to rotate, thereby driving the second crawler belt 4 to move.

[0220] A picking frame 5 for placing roxburghii is fixed on the upper surface of the shell.

[0221] The six-axis robot 7 is connected to the upper surface of the frame.

[0222] The camera 9 is mounted on the six-axis robot 7 and is used to capture RGB images and transmit them to the industrial control host 6 .

[0223] The laser radar is mounted on the six-axis robot 7 and is used to detect the distance between the six-axis robot 7 and the roxburghii tree.

[0224] Also includes power batteries.

[0225] The power battery supplies power to the first drive motor 1, the second drive motor 3, the industrial control host 6, the six-axis robot 7, the camera 9, and the laser radar.

[0226] It also includes a heat dissipation module 12 mounted on the vehicle frame.

[0227] The heat dissipation module 12 is used to dissipate heat for the first drive motor 1 , the second drive motor 3 , and the industrial control host 6 .

[0228] Also includes gas tank.

[0229] The industrial control host 6 controls the valve switch of the gas storage tank, thereby starting or closing the pneumatic soft claws 10 and the pneumatic scissors 11.

[0230] The vehicle frame includes a main frame 13 , an upper guard plate 14 , and a lower guard plate 15 .

[0231] The upper guard plate 14 and the lower guard plate 15 cover the top surface and the bottom surface of the main frame 13 to form a box body.

[0232] A mass sensor is also included.

[0233] The quality sensor is mounted on the bottom of the picking frame 5 to monitor the quality of the picking frame 5 and transmit the quality to the industrial control host 6 .

[0234] The industrial control host 6 stores a quality upper limit. When the quality of the picking frame 5 reaches the quality upper limit, the industrial control host 6 controls the automatic roxburgh pear picking device to travel so that the automatic roxburgh pear picking device reaches the nearest unloading point and unloads the picked roxburgh pears.

[0235] Embodiment 3:

[0236] See also Figures 1 to 5 , a method for automatic picking of roxburghii based on RGB image detection, comprising the following steps:

[0237] 1) Obtaining a map of the roxburghii orchard and storing it in the industrial control host 6;

[0238] The industrial control host 6 divides the roxburghii orchard into multiple picking areas D = {D1, D2, ..., D n}, and set the initial position point D of the automatic picking device of roxburghii in each picking area i0 and the end point D if , generate the initial position point D i0 To the end point D if The driving trajectory d i,0-f ;D n is the nth picking area in the roxburghii orchard; i = 1, 2, ..., n; n is the number of picking areas;

[0239] 2) The industrial control host drives the automatic picking device of the roxburghii to the initial position D of the i-th picking area. i0 =(x i0 ,y i0 ); x i0 ,y i0 is the initial position point D i0 The horizontal and vertical coordinates of; the initial value of i is 1;

[0240] Arrived at the initial position D i0 =(x i0 ,y i0 ) after which the industrial control host retrieves the driving trajectory d i,0-f , according to the driving trajectory d i,0-f Driving the automatic prickly pear picking device to move;

[0241] 3) During the driving process, the camera 9 of the automatic roxburghii picking device periodically captures the RGB image P of the a-th side of the automatic roxburghii picking device. a , and transmit it to the industrial control host 6;

[0242] The industrial control host 6 receives the RGB image P of the a-th side aInput into the roxburghii fruit tree recognition model to identify whether there is a roxburghii fruit tree at the a-th side. If yes, control the roxburghii automatic picking device to stop moving forward and enter step 4); if not, jump to step 13); the initial value of g is 1;

[0243] The a-th side RGB image P a As shown below:

[0244]

[0245] In the formula, (R MN ,G MN ,B MN ) is the RGB image P at the ath side a The R, G, and B components of the middle pixel (M, N);

[0246] The construction steps of the roxburghii fruit tree identification model are as follows:

[0247] 3.1) Obtain no more than 20 images containing roxburghii fruit trees, and use LabelImg software to mark the target frames of the roxburghii fruit trees as a small sample training set;

[0248] 3.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training data set with target boxes of roxburghii fruit trees;

[0249] 3.3) Using keras as the deep learning framework and YOLOv5 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii fruit tree recognition model;

[0250] 4) Start the laser radar;

[0251] The laser radar detects the straight-line distance H between the automatic picking device for roxburghii and the roxburghii fruit tree at the a-th side, and transmits it to the industrial control host 6;

[0252] 5) The industrial control host 6 generates a moving signal V1 based on the straight-line distance H = {V 11 ,V 12}, driving the automatic roxburghii picking device to turn so that the automatic roxburghii picking device faces the roxburghii fruit tree at the a-th side;

[0253] Then the industrial control host 6 drives the automatic prickly pear picking device to move forward, so that the straight-line distance between the automatic prickly pear picking device and the prickly pear tree at the a-th side satisfies ΔH1≤H≤ΔH2; ΔH1 is the redundant distance for the six-axis robot 7 to move; ΔH2 is the maximum movement distance of the six-axis robot 7; V 11 is the turning angle of the automatic picking device for roxburghii, V 12 It is the straight-line travel distance of the automatic prickly pear picking device after turning;

[0254] 6) Camera 9 captures the j-th position image P of the roxburghii fruit tree j , and transmitted to the industrial control host 6; the initial value of j is 1;

[0255]

[0256] In the formula, (R FE ,G FE ,B FE ) is the j-th orientation image P of the roxburghii fruit tree j The R, G, and B values ​​of the middle pixel (F, E);

[0257] The industrial control host 6 inputs the j-th azimuth image of the roxburghii fruit tree into the roxburghii recognition model, and recognizes the roxburghii target frame in the j-th azimuth image of the roxburghii fruit tree, which is recorded as Q={q1,q2,...,q W}; W is the number of roxburghii target boxes in the j-th orientation image of the roxburghii fruit tree; q W represents the Wth roxburghii target frame in the jth orientation image of the roxburghii fruit tree;

[0258] Among them, the w-th roxburghii target box P w As shown below:

[0259]

[0260] In the formula, The target frame P for the prickly pear w Middle pixel (F w ,E w )'s R, G, B values; w = 1, 2, ..., W;

[0261] The steps for building the roxburghii recognition model are as follows:

[0262] 6.1) Obtain no more than 20 images containing roxburghii, and use LabelImg software to mark the roxburghii target frame as a small sample training set;

[0263] 6.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training dataset with the target box of the roxburghii tree;

[0264] 6.3) Using keras as the deep learning framework and YOLOv3 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii recognition model;

[0265] 7) The industrial control host 6 sets the target frame P of the roxburghii wInput into the sea buckthorn maturity recognition model to determine the growth state of the sea buckthorn in the sea buckthorn target frame. If the growth state of the sea buckthorn is mature, the sea buckthorn target frame is marked as the sea buckthorn target image to be picked; the initial value of w is 1;

[0266] The growth state of the roxburghii includes unripe, ripe and overripe;

[0267] The step of determining the growth state of the roxburghii thorn in the roxburghii thorn target frame comprises:

[0268] 7.1) The industrial control host 6 sets the target frame P of the roxburghii w The RGB components of each point in the image are matched with the pre-stored RGB component threshold ranges of unripe state (90-190, 120-255, 0-80), ripe state RGB component threshold ranges (220-255, 185-255, 0-80), and overripe state RGB component threshold ranges (200-255, 140-185, 0-80). If the RGB component falls into any RGB component threshold range, and R≥G falls into the ripe state RGB component threshold range, and G≥R falls into the unripe state RGB component threshold range, then the point is regarded as a valid point;

[0269] 7.2) Calculate the average value of the RGB component of the unripe state based on the threshold range of the RGB component of the unripe state Based on the threshold range of the RGB component of the mature state, calculate the average RGB value of the unripe state Based on the threshold range of the overripe state RGB component, calculate the average value of the underripe state RGB

[0270] Among them, the RGB average values ​​under different growth conditions of roxburghii are as follows:

[0271]

[0272] In the formula, R t,max , R t,min is the upper and lower limits of the R component threshold corresponding to the growth state t; G t,max , G t,min is the upper and lower limits of the G component threshold corresponding to the growth state t; B t,max , B t,min are the upper and lower limits of the B component threshold corresponding to growth state t; t = 1, 2, 3, representing immature, mature, and overmature, respectively;

[0273] 7.3) respectively calculating the variance S1 between all valid points and the average value of the RGB component threshold of the unripe state, the variance S2 between all valid points and the average value of the RGB component threshold of the ripe state, and the variance S3 between all valid points and the average value of the RGB component threshold of the overripe state;

[0274] Among them, the variance St As shown below:

[0275]

[0276] Where U uR , U uG , U uB are the R, G, and B components of the u-th valid point;

[0277] 7.4) Compare variance S1, variance S2, and variance S3. If variance S1 is the smallest, then the current target frame P of the roxburghii roxburghii w The growth state of the Chinese prickly ash is unripe. If the variance S2 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly ash is mature. If the variance S3 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly pear is over-mature;

[0278] Marking a target frame of a thorn pear in a mature growth state as a target image of a thorn pear to be picked;

[0279] The threshold ranges of the RGB components for the unripe, ripe, and overripe states are obtained by the following steps:

[0280] s1) 1) obtaining a plurality of roxburghii roxburghii, extracting juice from each roxburghii roxburghii, and analyzing the roxburghii roxburghii juice by high performance liquid chromatography to determine the Vc content;

[0281] s2) dividing the growth status of the roxburghii into unripe, mature and overripe according to the range of Vc content, and labeling each roxburghii with a status label;

[0282] s3) taking RGB images of roxburghii roxburghii in different growth states, and obtaining RGB components of each RGB image of roxburghii roxburghii;

[0283] s4) performing statistics on the RGB components of the RGB images of the roxburghii roxburghii with the same growth state, and taking the overlapping range of the RGB components as the RGB component threshold range of the current growth state;

[0284] 8) Calculate the image coordinates of the center point of the mth target image of the prickly ash to be picked, the steps are as follows:

[0285] 8.1) extracting the outline of the roxburghii fruit in the target image of the roxburghii to be picked based on the threshold range of the three components of RGB in the maturity state;

[0286] 8.2) Binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image;

[0287] 8.3) Consider the foreground binary image of the roxburghii as a circular shape and construct the roxburghii contour equation, namely:

[0288] X 2+α0XY+α1Y 2 +α2X+α3Y+α4=0(9)

[0289] Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined;

[0290] 8.4) Taking the minimum root mean square error γ as the goal, fit and solve the coefficients α0, α1, α2, α3, and α4;

[0291] The root mean square error γ is given by:

[0292]

[0293] 8.5) Calculate the center point image coordinates (X0, Y0), that is:

[0294]

[0295] 9) based on the conversion relationship between the image coordinates and the geographic coordinates, converting the center point image coordinates into the center point geographic coordinates; the center point geographic coordinates include the distance between the automatic picking device for roxburghii roxburghii and the center point of the roxburghii roxburghii to be picked in the three-dimensional direction;

[0296] Based on the geographic coordinates of the center point, the industrial control host 6 generates a prickly pear picking signal and drives the six-axis robot 7 of the prickly pear automatic picking device to move, so that the pneumatic soft claws 10 mounted on the six-axis robot 7 clamp the prickly pear, and then the pneumatic scissors 11 located above the pneumatic soft claws 10 cut the prickly pear stalks;

[0297] After cutting off the roxburghii stalk, the six-axis robot 7 is driven to rotate so that the pneumatic soft claw 10 is located above the picking frame 5, and the vertical distance between the pneumatic soft claw 10 and the picking frame 5 is less than hmax; then, the pneumatic soft claw 10 is released to allow the roxburghii to fall into the picking frame 5;

[0298] 10) Let m=m+1, and return to step 8), until all the mature roxburghii thorns at the j-th position of the roxburghii thorn tree are picked;

[0299] 11) Determine whether the automatic pear picking device has circled the current pear tree. If so, proceed to step 12). If not, use the industrial control host 6 to drive the automatic pear picking device to the j+1th orientation image of the pear tree, set j=j+1, and return to step 6);

[0300] 12) Determine whether g=2 is established. If not, drive the six-axis robot 7 to rotate so that the camera is aimed at the g+1th side position, update g=g+1, and return to step 4). If yes, control the automatic picking device of roxburghii to move along the road of the i-th picking area, and return to step 3);

[0301] 13) Determine whether the automatic prickly roxburghii picking device has reached the end position point D of the i-th picking area if =(x if ,y if ), if not, return to step 3), if yes, control the automatic prickly roxburghii picking device to move, so that the automatic prickly roxburghii picking device reaches the initial position point D of the i+1th picking area (i+1)0 =(x (i+1)0 ,y (i+1)0 ), let i=i+1, and return to step 3), until the automatic prickly roxburghii picking device traverses all picking areas of the prickly roxburghii orchard, and the picking task is completed; x if ,y if The end point D if The horizontal and vertical coordinates of (i+1)0 ,y (i+1)0 is the initial position point D (i+1)0 The horizontal and vertical coordinates of .

[0302] Embodiment 4:

[0303] A method for automatic picking of roxburghii based on RGB image detection, the technical content is the same as that of Example 3, and further, the step of dividing the roxburghii orchard into multiple picking areas is as follows:

[0304] a1) dividing the roxburghii orchard into a plurality of areas with regular shapes according to the roxburghii orchard map;

[0305] a2) each regularly shaped area is divided into a plurality of mutually parallel picking areas, each picking area comprising a roxburghii orchard road and two rows of roxburghii fruit trees planted along the roxburghii orchard road.

[0306] Embodiment 5:

[0307] A method for automatic picking of roxburghii based on RGB image detection, the technical content is the same as any one of embodiments 3-4, further, the initial position point D of the first picking area 10 It is the location point closest to the gate of the sea buckthorn orchard and has sea buckthorn trees on both sides;

[0308] The initial position point D of the kth picking area k0 Select the end point D of the k-1th picking area if The nearest location point with two side locations of a sea buckthorn tree; k=2, ..., n; n is the number of picking areas.

[0309] Embodiment 6:

[0310] A method for automatic picking of roxburghii based on RGB image detection, the technical content of which is the same as any one of embodiments 3-5, and further, the conversion relationship between the center point image coordinates and the geographic coordinates is determined by the following steps:

[0311] b1) selecting a roxburghii from the roxburghii fruit trees, and using a camera of an automatic roxburghii picking device to capture an RGB image of the selected roxburghii;

[0312] b2) extracting the outline of the roxburghii fruit in the selected roxburghii RGB image;

[0313] b3) binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image;

[0314] b4) The foreground binary image of the roxburghii is regarded as a circular shape, and the contour equation of the roxburghii is constructed, namely:

[0315] X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(12)

[0316] Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined;

[0317] b5) Taking the minimum root mean square error γ as the goal, the coefficients α0, α1, α2, α3, and α4 to be determined are fitted and solved;

[0318] The root mean square error γ is given by:

[0319]

[0320] b6) Calculate the center point image coordinates (X0, Y0), that is:

[0321]

[0322] b7) using a laser radar to obtain a three-dimensional distance between the automatic prickly thorn picking device and the selected prickly thorn as a geographic coordinate;

[0323] b8) repeating steps b1) to b7) to obtain multiple sets of training data, each set of training data including the center point image coordinates and the corresponding geographic coordinates;

[0324] b9) Based on the training data, the relationship between the geographic coordinates and the center point image coordinates is fitted to obtain the conversion relationship between the center point image coordinates and the geographic coordinates.

[0325] Embodiment 7:

[0326] A method for automatic picking of roxburghii based on RGB image detection, the technical content of which is the same as any one of embodiments 3-6, and further, after collecting the image and before processing it, the image is normalized.

[0327] Embodiment 8:

[0328] A method for automatic picking of roxburghii based on RGB image detection, the technical content of which is the same as any one of embodiments 3-7, and further, when extracting the contour of roxburghii fruit in the target image of roxburghii to be picked, edge extraction processing is also performed using the Canny algorithm.

[0329] Embodiment 9:

[0330] The automatic prickly pear picking device based on the method comprises a first drive motor 1, a first crawler 2, a second drive motor 3, a second crawler 4, a frame, a picking frame 5, an industrial control host 6, a six-axis robot 7, a transmission wheel 8, a camera 9, a laser radar, a pneumatic soft claw 10, and a pneumatic scissors 11.

[0331] The interior of the frame is used to place the first drive motor 1 , the second drive motor 3 , and the industrial control host 6 .

[0332] The industrial control host 6 is used to control the operation of the first drive motor 1 , the second drive motor 3 , the six-axis robot 7 , the camera 9 , the laser radar, the pneumatic soft claw 10 , and the pneumatic scissors 11 .

[0333] The first crawler 2 and the second crawler 4 are symmetrically mounted on both ends of the frame.

[0334] A plurality of transmission wheels 8 are installed inside the first crawler belt 2 and the second crawler belt 4 .

[0335] The first drive motor 1 drives the transmission wheel 8 in the first crawler belt 2 to rotate, thereby driving the first crawler belt 2 to move.

[0336] The second drive motor 3 drives the transmission wheel 8 in the second crawler belt 4 to rotate, thereby driving the second crawler belt 4 to move.

[0337] A picking frame 5 for placing roxburghii is fixed on the upper surface of the shell.

[0338] The six-axis robot 7 is connected to the upper surface of the frame.

[0339] The camera 9 is mounted on the six-axis robot 7 and is used to capture RGB images and transmit them to the industrial control host 6 .

[0340] The laser radar is mounted on the six-axis robot 7 and is used to detect the distance between the six-axis robot 7 and the roxburghii tree.

[0341] Embodiment 10:

[0342] The automatic sea buckthorn picking device based on the method has the same technical content as Example 9, and further includes a power battery.

[0343] The power battery supplies power to the first drive motor 1, the second drive motor 3, the industrial control host 6, the six-axis robot 7, the camera 9, and the laser radar.

[0344] Embodiment 11:

[0345] The automatic sea buckthorn picking device based on the method has the same technical content as any one of Examples 9-10, and further includes a heat dissipation module 12 mounted on the frame.

[0346] The heat dissipation module 12 is used to dissipate heat for the first drive motor 1 , the second drive motor 3 , and the industrial control host 6 .

[0347] Embodiment 12:

[0348] The automatic sea buckthorn picking device based on the method has the same technical content as any one of Examples 9-11, and further includes a gas storage tank.

[0349] The industrial control host 6 controls the valve switch of the gas storage tank, thereby starting or closing the pneumatic soft claws 10 and the pneumatic scissors 11.

[0350] Embodiment 13:

[0351] The automatic sea buckthorn picking device based on the method has the same technical content as any one of Examples 9-12. Furthermore, the frame includes a main frame 13, an upper guard plate 14, and a lower guard plate 15.

[0352] The upper guard plate 14 and the lower guard plate 15 cover the top surface and the bottom surface of the main frame 13 to form a box body.

[0353] Embodiment 14:

[0354] The automatic sea buckthorn picking device based on the method has the same technical content as any one of Examples 9-13, and further includes a quality sensor.

[0355] The quality sensor is mounted on the bottom of the picking frame 5 to monitor the quality of the picking frame 5 and transmit the quality to the industrial control host 6 .

[0356] The industrial control host 6 stores a quality upper limit. When the quality of the picking frame 5 reaches the quality upper limit, the industrial control host 6 controls the automatic roxburgh pear picking device to travel so that the automatic roxburgh pear picking device reaches the nearest unloading point and unloads the picked roxburgh pears.

[0357] Embodiment 15:

[0358] The automatic sea buckthorn picking device based on the method has the same technical content as any one of embodiments 9-14. Further, the method of unloading the picked sea buckthorn includes: the staff directly replaces the empty picking frame 5, or the staff unloads the sea buckthorn in the picking frame 5 and then puts the picking frame 5 back.

[0359] Embodiment 16:

[0360] The automatic sea buckthorn picking device based on the method has the same technical content as any one of Examples 9-15. Furthermore, a buffer pad (such as a sponge) is placed in the picking frame 5 to prevent the sea buckthorn from being damaged.

Claims

1. A method for automatic picking of roxburghii based on RGB image detection, characterized in that: The following steps are involved: 1) Obtaining a map of the roxburghii orchard and storing it in the industrial control host (6). The industrial control host (6) divides the roxburghii orchard into multiple picking areas D = {D1, D2, ..., D n }, and set the initial position point D of the automatic picking device of roxburghii in each picking area i0 and the end point D if , generate the initial position point D i0 To the end point D if The driving trajectory d i,0-f ;D n is the nth picking area in the sea buckthorn orchard; i = 1, 2, ..., n; n is the number of picking areas. 2) The industrial control host drives the automatic picking device of the roxburghii to the initial position D of the i-th picking area. i0 =(x i0 ,y i0 ); x i0 ,y i0 is the initial position point D i0 The horizontal and vertical coordinates of; the initial value of i is 1; Arrived at the initial position D i0 =(x i0 ,y i0 ) after which the industrial control host retrieves the driving trajectory d i,0-f , according to the driving trajectory d i,0-f Driving the automatic prickly roxburghii picking device to move; 3) During the driving process, the camera (9) of the automatic pear picking device periodically captures the RGB image P of the a-th side of the automatic pear picking device a , and transmit it to the industrial control host (6); The industrial control host (6) converts the a-th side RGB image P a Input into the roxburghii fruit tree recognition model to identify whether there is a roxburghii fruit tree at the a-th side. If yes, control the roxburghii automatic picking device to stop moving forward and enter step 4); if not, jump to step 13); the initial value of g is 1; The a-th side RGB image P a As shown below: In the formula, (R MN ,G MN ,B MN ) is the RGB image P at the ath side a The R, G, and B components of the middle pixel (M, N); The construction steps of the roxburghii fruit tree identification model are as follows: 3.1) Obtain no more than 20 images containing roxburghii fruit trees, and use LabelImg software to mark the target frames of the roxburghii fruit trees as a small sample training set; 3.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training data set with target boxes of roxburghii fruit trees; 3.3) Using keras as the deep learning framework and YOLOv5 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii fruit tree recognition model; 4) Start the laser radar; The laser radar detects the straight-line distance H between the automatic roxburghii picking device and the roxburghii fruit tree at the a-th side, and transmits the distance to the industrial control host (6); 5) The industrial control host (6) generates a moving signal V1 based on the straight-line distance H = {V 11 ,V 12 }, driving the automatic roxburghii picking device to turn so that the automatic roxburghii picking device faces the roxburghii fruit tree at the a-th side; Then, the industrial control host (6) drives the automatic prickly thorn picking device to move forward, so that the straight-line distance between the automatic prickly thorn picking device and the prickly thorn fruit tree at the a-th side satisfies ΔH1≤H≤ΔH2; ΔH1 is the redundant distance for the six-axis robot (7) to move; ΔH2 is the maximum movement distance of the six-axis robot (7); V 11 is the turning angle of the automatic picking device for roxburghii, V 12 It is the straight-line travel distance of the automatic prickly pear picking device after turning; 6) Camera (9) captures the j-th position image P of the roxburghii fruit tree j , and transmitted to the industrial control host (6); the initial value of j is 1; In the formula, (R FE ,G FE ,B FE ) is the j-th orientation image P of the roxburghii fruit tree j The R, G, and B values ​​of the middle pixel (F, E); The industrial control host (6) inputs the j-th azimuth image of the roxburghii fruit tree into the roxburghii recognition model, and recognizes the roxburghii target frame in the j-th azimuth image of the roxburghii fruit tree, which is recorded as Q={q1,q2,...,q W }; W is the number of roxburghii target boxes in the j-th orientation image of the roxburghii fruit tree; q W represents the Wth roxburghii target frame in the jth orientation image of the roxburghii fruit tree; Among them, the w-th roxburghii target box P w As shown below: In the formula, The target frame P for the prickly pear w Middle pixel (F w ,E w )'s R, G, B values; w = 1, 2, ..., W; The steps for building the roxburghii recognition model are as follows: 6.1) Obtain no more than 20 images containing roxburghii, and use LabelImg software to mark the roxburghii target frame as a small sample training set; 6.2) Expand the small sample training set by horizontal rotation, mirroring and / or adding noise to construct a training dataset with the target box of the roxburghii tree; 6.3) Using keras as the deep learning framework and YOLOv3 algorithm as the iterative algorithm, the neural network model was trained using the training data set to obtain the roxburghii recognition model; 7) The industrial control host (6) sets the target frame P of the roxburghii w Input into the prickly pear maturity recognition model to determine the growth state of the prickly pear in the prickly pear target frame. If the prickly pear growth state is mature, the prickly pear target frame is marked as the prickly pear target image to be picked; the initial value of w is 1; The growth state of the roxburghii includes unripe, ripe and overripe; The step of determining the growth state of the roxburghii thorn in the roxburghii thorn target frame comprises: 7.1) The industrial control host (6) sets the target frame P of the roxburghii w The RGB components of each point in the image are matched with the pre-stored RGB component threshold ranges of unripe state (90-190, 120-255, 0-80), ripe state RGB component threshold ranges (220-255, 185-255, 0-80), and overripe state RGB component threshold ranges (200-255, 140-185, 0-80). If the RGB component falls into any RGB component threshold range, and R≥G falls into the ripe state RGB component threshold range, and G≥R falls into the unripe state RGB component threshold range, then the point is regarded as a valid point; 7.2) Calculate the average value of the RGB component of the unripe state based on the threshold range of the RGB component of the unripe state Based on the threshold range of the RGB component of the mature state, calculate the average RGB value of the unripe state Based on the threshold range of the overripe state RGB component, calculate the average value of the underripe state RGB Among them, the RGB average values ​​under different growth conditions of roxburghii are as follows: In the formula, R t,max , R t,min is the upper and lower limits of the R component threshold corresponding to the growth state t; G t,max , G t,min is the upper and lower limits of the G component threshold corresponding to the growth state t; B t,max , B t,min are the upper and lower limits of the B component threshold corresponding to growth state t; t = 1, 2, 3, representing immature, mature, and overmature, respectively; 7.3) respectively calculating the variance S1 between all valid points and the average value of the RGB component threshold of the unripe state, the variance S2 between all valid points and the average value of the RGB component threshold of the ripe state, and the variance S3 between all valid points and the average value of the RGB component threshold of the overripe state; Among them, the variance S t As shown below: Where U uR , U uG , U uB are the R, G, and B components of the u-th valid point; 7.4) Compare variance S1, variance S2, and variance S3. If variance S1 is the smallest, then the current target frame P of the roxburghii roxburghii w The growth state of the Chinese prickly ash is unripe. If the variance S2 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly ash is mature. If the variance S3 is the smallest, then the current prickly ash target frame P w The growth state of the Chinese prickly pear is over-mature; Marking a target frame of a thorn pear in a mature growth state as a target image of a thorn pear to be picked; The threshold ranges of the RGB components for the unripe, ripe, and overripe states are obtained by the following steps: s1) 1) obtaining a plurality of roxburghii roxburghii, extracting juice from each roxburghii roxburghii, and analyzing the roxburghii roxburghii juice by high performance liquid chromatography to determine the Vc content; s2) dividing the growth status of the roxburghii into unripe, mature and overripe according to the range of Vc content, and labeling each roxburghii with a status label; s3) taking RGB images of roxburghii roxburghii in different growth states, and obtaining RGB components of each RGB image of roxburghii roxburghii; s4) performing statistics on the RGB components of the RGB images of the roxburghii roxburghii with the same growth state, and taking the overlapping range of the RGB components as the RGB component threshold range of the current growth state; 8) Calculate the image coordinates of the center point of the mth target image of the prickly ash to be picked, the steps are as follows: 8.1) extracting the outline of the roxburghii fruit in the target image of the roxburghii to be picked based on the threshold range of the three components of RGB in the maturity state; 8.2) Binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image; 8.3) Consider the foreground binary image of the roxburghii as a circular shape and construct the roxburghii contour equation, namely: X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(9) Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined; 8.4) Taking the minimum root mean square error γ as the goal, fit and solve the coefficients α0, α1, α2, α3, and α4; The root mean square error γ is given by: In the formula, (X c ,Y c ) is the cth pixel on the foreground binary image of the roxburghii; C is the number of pixels; 8.5) Calculate the center point image coordinates (X0, Y0), that is: 9) based on the conversion relationship between the image coordinates and the geographic coordinates, converting the center point image coordinates into the center point geographic coordinates; the center point geographic coordinates include the distance between the automatic picking device for roxburghii roxburghii and the center point of the roxburghii roxburghii to be picked in the three-dimensional direction; Based on the geographical coordinates of the center point, the industrial control host (6) generates a roxburgh pear picking signal and drives a six-axis robot (7) of the roxburgh pear automatic picking device to move, so that a pneumatic soft claw (10) mounted on the six-axis robot (7) clamps the roxburgh pear, and then a pneumatic scissors (11) located above the pneumatic soft claw (10) cuts off the roxburgh pear stalk; After the roxburghii stalk is cut off, the six-axis robot (7) is driven to rotate so that the pneumatic soft claw (10) is located above the picking frame (5), and the vertical distance between the pneumatic soft claw (10) and the picking frame (5) is less than hmax; then, the pneumatic soft claw (10) is released to allow the roxburghii stalk to fall into the picking frame (5); 10) Let m=m+1, and return to step 8), until all the mature roxburghii thorns at the j-th position of the roxburghii thorn tree are picked; 11) determining whether the automatic pear picking device has circled the current pear tree, if so, proceeding to step 12), if not, using the industrial control host (6) to drive the automatic pear picking device to the j+1th orientation image of the pear tree, setting j=j+1, and returning to step 6); 12) Determine whether g=2 is established. If not, drive the six-axis robot (7) to rotate so that the camera is aimed at the g+1th side position, update g=g+1, and return to step 4). If yes, control the automatic picking device of roxburghii to move along the road of the i-th picking area, and return to step 3); 13) Determine whether the automatic prickly roxburghii picking device has reached the end position point D of the i-th picking area if =(x if ,y if ), if not, return to step 3), if yes, control the automatic prickly roxburghii picking device to move, so that the automatic prickly roxburghii picking device reaches the initial position point D of the i+1th picking area (i+1)0 =(x (i+1)0 ,y (i+1)0 ), let i=i+1, and return to step 3), until the automatic prickly roxburghii picking device traverses all picking areas of the prickly roxburghii orchard, and the picking task is completed; x if ,y if The end point D if The horizontal and vertical coordinates of (i+1)0 ,y (i+1)0 is the initial position point D (i+1)0 The horizontal and vertical coordinates of .

2. A method for automatically picking roxburghii based on RGB image detection according to claim 1, characterized in that: The steps to divide the sea buckthorn orchard into multiple picking areas are: 1) According to the map of the roxburghii orchard, the roxburghii orchard is divided into a plurality of areas with regular shapes; 2) Each regularly shaped area is divided into a plurality of mutually parallel picking areas, each picking area comprising a sea buckthorn orchard road and two rows of sea buckthorn fruit trees planted along the sea buckthorn orchard road.

3. A method for automatically picking roxburghii based on RGB image detection according to claim 2, characterized in that, The first picking area initial position point D 10 It is the location point closest to the gate of the sea buckthorn orchard and has sea buckthorn trees on both sides; The initial position point D of the kth picking area k0 Select the end point D of the k-1th picking area if The nearest location point with two side locations of a sea buckthorn tree; k=2, ..., n; n is the number of picking areas.

4. A method for automatically picking roxburghii based on RGB image detection according to claim 1, characterized in that: The conversion relationship between the center point image coordinates and geographic coordinates is determined by the following steps: 1) Selecting a roxburghii from a roxburghii fruit tree, and using a camera of a roxburghii automatic picking device to capture an RGB image of the selected roxburghii; 2) extracting the outline of the roxburghii fruit in the selected roxburghii RGB image; 3) Binarizing the outline of the roxburghii fruit, and then extracting the roxburghii foreground binary image; 4) The foreground binary image of the roxburghii is regarded as a circular shape, and the contour equation of the roxburghii is constructed, namely: X 2 +α0XY+α1Y 2 +α2X+α3Y+α4=0(12) Where (X, Y) is the pixel point on the binary image of the foreground of the roxburghii tree; α0, α1, α2, α3, and α4 are the coefficients to be determined; 5) Taking the minimum root mean square error γ as the goal, the coefficients α0, α1, α2, α3, and α4 to be determined are fitted and solved; The root mean square error γ is given by: 6) Calculate the center point image coordinates (X0, Y0), that is: 7) using laser radar to obtain the three-dimensional distance between the automatic picking device for roxburghii and the selected roxburghii as the geographic coordinates; 8) Repeat steps 1) to 7) to obtain multiple sets of training data, each set of training data includes the center point image coordinates and the corresponding geographic coordinates; 9) Based on the training data, the relationship between the geographic coordinates and the center point image coordinates is fitted to obtain the conversion relationship between the center point image coordinates and the geographic coordinates.

5. An automatic roxburghii picking device based on the method according to any one of claims 1 to 4, characterized in that: It comprises a first drive motor (1), a first crawler (2), a second drive motor (3), a second crawler (4), a frame, a picking frame (5), an industrial control host (6), a six-axis robot (7), a transmission wheel (8), a camera (9), a laser radar, a pneumatic soft claw (10), and a pneumatic scissors (11); The interior of the frame is used to place a first drive motor (1), a second drive motor (3), and an industrial control host (6); The industrial control host (6) is used to control the operation of the first drive motor (1), the second drive motor (3), the six-axis robot (7), the camera (9), the laser radar, the pneumatic soft claw (10), and the pneumatic scissors (11); A first crawler (2) and a second crawler (4) are symmetrically mounted at both ends of the frame; A plurality of transmission wheels (8) are installed inside the first crawler belt (2) and the second crawler belt (4); The first drive motor (1) drives the transmission wheel (8) in the first crawler (2) to rotate, thereby driving the first crawler (2) to move; The second drive motor (3) drives the transmission wheel (8) in the second crawler belt (4) to rotate, thereby driving the second crawler belt (4) to move; A picking frame (5) for placing roxburghii roxburghii is fixed on the upper surface of the shell; The six-axis robot (7) is connected to the upper surface of the frame; The camera (9) is mounted on the six-axis robot (7) and is used to capture RGB images and transmit them to the industrial control host (6); The laser radar is mounted on the six-axis robot (7) and is used to detect the distance between the six-axis robot (7) and the roxburghii tree.

6. The automatic picking device for roxburghii according to claim 5, characterized in that: Also includes power batteries; The power battery supplies power to the first drive motor (1), the second drive motor (3), the industrial control host (6), the six-axis robot (7), the camera (9), and the laser radar.

7. The automatic picking device for roxburghii according to claim 5, characterized in that: Also included is a heat dissipation module (12) mounted on the vehicle frame; The heat dissipation module (12) is used to dissipate heat for the first drive motor (1), the second drive motor (3), and the industrial control host (6).

8. The automatic roxburghii picking device according to claim 5, characterized in that: Also includes gas tanks; The industrial control host (6) controls the valve switch of the gas storage tank, thereby starting or closing the pneumatic soft claw (10) and the pneumatic scissors (11).

9. The automatic roxburghii picking device according to claim 4, characterized in that: The vehicle frame comprises a main frame (13), an upper guard plate (14), and a lower guard plate (15); The upper guard plate (14) and the lower guard plate (15) cover the top surface and the bottom surface of the main frame (13), thereby forming a box body.

10. The automatic picking device for roxburghii according to claim 5, characterized in that: Also included are mass sensors; The quality sensor is mounted on the bottom of the picking frame (5) and is used to monitor the quality of the picking frame (5) and transmit the quality to the industrial control host (6); The industrial control host (6) stores a quality upper limit. When the quality of the picking frame (5) reaches the quality upper limit, the industrial control host (6) controls the automatic roxburgh pear picking device to travel so that the automatic roxburgh pear picking device reaches the nearest unloading point and unloads the picked roxburgh pears.

Images