Nut green peel fruit recognition positioning and automatic picking control method based on image recognition and picking device

Abstract

The invention provides a nut green peel fruit recognition positioning and automatic picking control method based on image recognition and a picking device.The nut green peel fruit recognition positioning and automatic picking control method and the picking device.The nut green peel fruit recognition positioning and automatic picking control device belong to the technical field of nut recognition picking. And then image fusion is carried out, so that the fruit shielded by the leaves can be identified. According to the invention, the nut tree is shot from different angles, fruits shielded by leaves can be identified, and reflection identification of different angles can be realized for nut green-peel fruits (the arc-shaped contours of the fruits can be identified more easily from different angles) through shooting from different angles. According to the method, the positions of the nut green apples can be identified more accurately, the fruits shielded by the leaves can be displayed through multi-image fusion, and picking omission is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of nut identification and picking, and in particular to a method and a picking device for identifying, locating and automatically picking green-skinned nuts based on image recognition. Background Art

[0002] Macadamia nuts are generally harvested manually or by mechanical vibration. Manual harvesting is when the fruit is ripe, using fruit hooks to peel off the involucres one by one. When using mechanical harvesting, the fruit trees are sprayed with ethephon two weeks before harvesting to accelerate ripening, and then the branches are vibrated mechanically to make the ripe fruits fall. The use of ethephon will cause the leaves to fall off early, which will affect the growth of the fruit trees and the yield of the following year. At present, most of the methods used in China are mechanical vibration, or manipulators are used for assisted harvesting. However, the manipulator picking does not have the function of automatically locating and identifying the green-skinned fruits of nuts, so the picking efficiency is low, the speed is slow, and the effect is not good.

[0003] Since the color of the green-skinned fruit of the nut is very similar to the color of the leaves, it is difficult to identify it when using image recognition. Since the color of the fruit and the leaves are similar, the difficulty of identification is increased, and the position of the green-skinned fruit of the nut cannot be accurately identified. At the same time, the position of the green-skinned fruit of the nut is on the branch and is easily blocked by the leaves, which is easy to be missed, making it very difficult to use a manipulator to pick it, and the efficiency is low. Therefore, it is necessary to design a method and a picking device for identifying, locating and automatically picking green-skinned fruit of the nut based on image recognition. Summary of the invention

[0004] The purpose of the present invention is to provide a method and a picking device for identifying and locating green-skinned nuts and fruits based on image recognition, and to solve the technical problems of difficult identification and positioning of green-skinned nuts and fruits and low picking efficiency. By moving a camera to shoot the nut tree at different angles, and then identifying the images taken at each angle, and then performing image fusion, the fruits blocked by leaves can also be identified.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for identifying and locating green-skinned nuts based on image recognition, the method comprising the following steps:

[0007] Step 1: Install a camera mobile shooting device on the nut and green fruit picking cart;

[0008] Step 2: Drive the nut green-skin fruit picking cart to the front of the nut tree, and the camera shoots the nut tree at the initial position of the mobile shooting device to obtain an initial image;

[0009] Step 3: Perform object contour recognition on the initial image to obtain the position points of the nut green-skinned fruit at a number of initial recognized points, and mark a number of points as recognition points;

[0010] Step 4: Construct a two-dimensional coordinate plane, paste the initial image on the two-dimensional coordinate plane, obtain the coordinate position of the nut green peel fruit and the coordinates of the identification points, and set the center point;

[0011] Step 5: The camera moves on the mobile shooting device, takes pictures after each fixed distance of movement to obtain images in different directions, and then summarizes the direction images of all points to obtain a direction image set;

[0012] Step 6: Fit each image in the directional image set to the two-dimensional coordinate plane according to the identification point to obtain several layers of image sets, and then identify the fruit position coordinates of the green-skinned nuts in each layer of the image;

[0013] Step 7: The fruit position coordinates of all images are aggregated into a coordinate plane, and the position coordinates of each fruit remain unchanged, so as to realize the recognition and positioning of green-skinned nuts.

[0014] Further, in step 1, the camera mobile shooting device includes a camera, a sliding chassis and a sliding frame, the camera is set on the sliding chassis, the sliding chassis is clamped in the sliding frame and can be movably set, the sliding frame includes a vertical slide rail and a horizontal slide rail, the vertical slide rail and the horizontal slide rail are vertically arranged to cross each other, an initial positioning point is arranged inside the intersection, an initial positioning point sensing switch is arranged at the bottom of the sliding chassis, when the initial positioning point and the initial positioning point sensing switch are on the same straight line, it means that the camera is in the initial position, the sliding chassis is connected to an external control device, moves and pauses according to the signal of the control device, the camera is connected to the control device, and transmits the collected pictures to the control device for processing, and the vertical slide rail and the horizontal slide rail are arranged as a retractable structure, which is extended or shortened according to the size of the nut tree to be picked.

[0015] Furthermore, in step 2, the nut green-skinned fruit picking cart comes in front of the nut tree, the camera identifies the external wheel library of the nut tree image, identifies the height and width of the nut tree, and then moves the camera mobile shooting device up and down so that the center position of the camera mobile shooting device is aligned with the center position of the nut tree, and then the camera moves to the initial positioning point (5) of the mobile shooting device, and then shoots the nut tree to obtain the initial image of the nut tree.

[0016] Furthermore, in step 3, the initial image is initialized, and then contour recognition is performed on objects inside the image. Contour recognition includes leaf contours, circular contours of nuts and green-skinned fruits, and branch contours. When an arc-shaped contour is identified, it is considered to be the fruit of the nut green-skinned fruit, and the positions of several points of the nut green-skinned fruit are obtained. At the same time, several points are selected at the edge and midline of the nut tree in the initial image as identification points. The identification points are used for position coordinate correction during the subsequent highlighting and fitting. Due to changes in the shooting angle, image position stretching or shrinkage correction is required after shooting, and all image recognitions are overlapped during fitting.

[0017] Furthermore, in step 5, the camera moves along the sliding frame at the center of the camera mobile shooting device, first moves to the upper part of the vertical slide rail, pauses after each movement to a fixed position, and then notifies the camera to shoot, and then records the specific position of the shot image, and then moves upward to a fixed position, and then shoots, until it moves to the top, returns to the initial point, moves downward to shoot, and then moves left and right to shoot, and all the shot directional images are collected to obtain a directional image set.

[0018] Furthermore, in step 6, the positions of the identification points of the image sets of several layers are all overlapped, and then the arc-shaped contour of the nut green-skinned fruit in each image is identified as the fruit position of the nut green-skinned fruit, and the position of the nut green-skinned fruit at each location displays detailed coordinate information;

[0019] In step 7, the specific process of summarizing the fruit position coordinates into a coordinate plane is to fuse several layers of images together, and then superimpose the same or adjacent fruit position coordinates. If there are fruits in some images and no fruits in other images, the leaves of the images without fruits are blocked and erased to display the positions of fruits in all images, and the positions of the green-skinned fruits of the nuts are obtained.

[0020] A method for automatically picking green-skinned nuts based on image recognition. The method for identifying and locating green-skinned nuts according to any one of claims 1 to 6 is used to locate nuts. A temporary correction camera for picking is then installed at the front end of a manipulator of a picking trolley. According to the position of each green-skinned nut, a control device calculates the rotational freedom required by the manipulator to pick the nut. The manipulator arrives in front of the green-skinned nut and the temporary correction camera collects the best picking position in real time. The manipulator then picks the nut. When the manipulator picks the nut and leaves, the temporary correction camera collects whether there are any unpicked nuts at the place where the nut has just been picked. If not, the nut fruit position plane is deleted. If so, the manipulator continues to be controlled to pick the nut a second time until the nut is picked and the fruit coordinates in the fruit position plane are deleted. The picking trolley then moves to the other side to continue picking the nuts. Each nut tree can be picked completely by picking on all four sides.

[0021] Furthermore, the specific process of calculating the required rotational freedom of the manipulator by the control device is as follows: H, L, and P are set to represent the height from the ground to the chassis of the manipulator, the length of the long arm, and the length of the short arm, respectively; m is the center height of the nut to be picked; the angle between the long arm and the short arm remains unchanged at 90 degrees; the long arm rotates to make the center of the camera collinear with the center of the target; after being collinear, the current base rotation angle θ and the long arm rotation angle θ1 are returned; the imaging distance measurement length S can be obtained by using trigonometric function theorem and the Pythagorean theorem; the specific derivation formulas are shown in equations (1), (2), (3), (4), and (5):

[0022] T=P·tanθ1+L (1)

[0023]

[0024]

[0025] After obtaining S, S is vector-decomposed by rotating the angle θ, and the sine and cosine function calculation formula is used to obtain the x-axis coordinate X, y-axis coordinate Y, and z-axis coordinate Z with the robot itself as the base coordinate. The specific formulas are shown in equations (6), (7), and (8):

[0026] Z=2·m (6)

[0027]

[0028] After the camera resolves the relative real coordinates of the target nut picking and the manipulator, the inverse solution algorithm is used to solve the servo rotation angle of each degree of freedom of the manipulator. A three-degree-of-freedom manipulator is used, and the trigonometric function geometry method is used to solve the inverse kinematics formula of the manipulator. The real coordinates of the target nut are known to be (X, Y, Z), the long arm of the manipulator is L1, the short arm is L2, and the gripper connection part is L3. The rotation angles of the three axes that need to be solved are

[0029] From the inverse trigonometric function, we know that in a right triangle, The formula is shown in formula (9).

[0030]

[0031] The geometric model of the robot arm is established on the YOZ plane of the robot arm. The cosine theorem in trigonometric functions can be used to obtain equations (10), (11), and (12):

[0032]

[0033] Using the Pythagorean theorem of a right triangle, we can get the expressions of T1 and T2 as shown in equations (13) and (14).

[0034]

[0035] Using the arc cosine formula in trigonometric functions, we can find The specific expressions are shown in formula (15), formula (16), formula (17) and formula (18):

[0036]

[0037] The rotation angles of the three axes are Finally, control the robot to perform picking actions.

[0038] An automatic picking device for identifying and locating green-skinned nuts based on image recognition comprises a picking vehicle body, a mobile shooting device, an identification and positioning camera, a picking manipulator, a picking correction camera and a data analysis control device. The mobile shooting device, the picking manipulator and the data analysis control device are all arranged on the picking vehicle body, the mobile shooting device is arranged at the front end, the identification and positioning camera is arranged on the mobile shooting device in a sliding manner, and the picking vehicle body, the identification and positioning camera, the picking manipulator and the picking correction camera are all connected to the data analysis control device.

[0039] An application of a green-skinned nut identification and positioning method based on image recognition, the identification and positioning method is applied to automatic orange picking identification and positioning, automatic walnut picking identification and positioning, automatic pear picking identification and positioning, or automatic apple picking identification and positioning.

[0040] The present invention has the following beneficial effects due to the adoption of the above technical solution:

[0041] (1) The present invention can identify some fruits hidden by leaves by photographing the nut tree from different angles. At the same time, by photographing at different angles, the green-skinned fruit of the nut can be identified at different angles of reflection (different angles make it easier to identify the arc-shaped outline of the fruit), and the position of the green apple of the nut can be identified more accurately. In addition, by fusion of multiple images, the fruit hidden by leaves can be displayed to avoid missing when picking.

[0042] (2) During the picking process, by adding a picking correction camera to the front end of the robot, the picking point can be corrected in real time, so that the picking gripping point is the optimal point, and more fruits can be picked at one time. At the same time, the picking point is re-identified at the moment of picking back to avoid picking missed fruits, thus achieving more thorough picking and avoiding omissions.

[0043] (3) The identification and positioning method of the present application can also be used for identification and positioning in the automatic picking of other fruits to achieve more accurate identification and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the identification and positioning method of the present invention;

[0045] Figure 2 It is a comparison diagram of images of the same point photographed from different points in the direction of the branch of the present invention;

[0046] Figure 3 It is a structural schematic diagram of the mobile shooting device of the present invention;

[0047] Figure 4 It is a model diagram of the actual distance parameter conversion of the nut position of the present invention;

[0048] Figure 5 The XOY plane of the robot arm of the present invention is used to establish the robot arm geometric model diagram;

[0049] Figure 6 It is an analysis diagram of the geometric model of the robot arm established in the YOZ plane of the robot arm of the present invention;

[0050] Figure 7 It is a functional block diagram of the automatic picking device module of the present invention.

[0051] In the attached figure, 1-leaf shielding area, 2-nut green skin fruit shape, 3-vertical slide rail, 4-horizontal slide rail, 5-initial positioning point. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are only for the purpose of enabling the reader to have a thorough understanding of one or more aspects of the present invention, and these aspects of the present invention can be implemented even without these specific details.

[0053] like Figure 1 As shown, a method for identifying and locating green-skinned nuts based on image recognition comprises the following steps:

[0054] Step 1: Install a camera mobile shooting device on the nut and green fruit picking cart. The camera mobile shooting device includes a camera, a sliding chassis and a sliding frame. The camera is set on the sliding chassis. The sliding chassis is clamped in the sliding frame and can be movably set. The sliding frame includes a vertical slide rail 3 and a horizontal slide rail 4. The vertical slide rail 3 and the horizontal slide rail 4 are vertically arranged to cross each other. An initial positioning point 5 is arranged inside the intersection. An initial positioning point sensing switch is arranged at the bottom of the sliding chassis. When the initial positioning point 5 and the initial positioning point sensing switch are on the same straight line, it means that the camera is in the initial position. The sliding chassis is connected to an external control device, and moves and pauses according to the signal of the control device. The camera is connected to the control device and transmits the collected pictures to the control device for processing. At the same time, the vertical slide rail 3 and the horizontal slide rail 4 are arranged as a retractable structure, which is extended or shortened according to the size of the nut tree to be picked. The camera mobile shooting device is as follows: Figure 3 As shown, it is set as a cross structure, the ends are closed structures, and the middle is set as a retractable guide rail structure, so that it can be retracted according to the size of the tree.

[0055] Step 2: Drive the nut green-skin fruit picking cart to the front of the nut tree, and the camera shoots the nut tree at the initial position of the mobile shooting device to obtain the initial image. When the nut green-skin fruit picking cart comes to the front of the nut tree, the camera recognizes the external wheel library of the nut tree image, identifies the height and width of the nut tree, and then moves the camera mobile shooting device up and down so that the center position of the camera mobile shooting device is aligned with the center position of the nut tree, and then the camera moves to the initial positioning point 5 of the mobile shooting device, and then shoots the nut tree to obtain the initial image of the nut tree. The initial image is a positive image. When shooting, one side of the whole tree is within the image range. Find the features of several prominent points on the edge as identification points. These identifications can basically identify the positions of several of these points after disguised shooting.

[0056] Step 3: Perform contour recognition on the initial image to obtain the position points of the nut green-skinned fruit at several points of initial recognition, and mark several points as recognition points. Initialize the initial image, and then perform contour recognition on the objects inside the image. Contour recognition includes leaf contours, circular contours of nut green-skinned fruits, and branch contours. When a circular arc contour is recognized, it is considered to be the fruit of the nut green-skinned fruit, and the positions of several points of nut green-skinned fruit are obtained. At the same time, several points are selected at the edge and midline of the nut tree in the initial image as recognition points. The recognition points are used to correct the position coordinates during the later highlighting and fitting. Due to changes in the shooting angle, the image position needs to be stretched or shrunk after shooting. When fitting, the recognition of all images is overlapped. Contour recognition uses the practical contour recognition method to obtain the contours of fruits and leaves through edge recognition, and then the circular arc is identified as fruit.

[0057] Step 4: Construct a two-dimensional coordinate plane, paste the initial image on the two-dimensional coordinate plane, obtain the coordinate position of the green-skinned fruit and the coordinates of the identification points, and set the center point. The two-dimensional coordinate plane is the XY axis plane. By setting the middle point of the XY axis plane as the origin of the plane, that is, the center point, and then identifying the features of the initial image at the center point, the features of the center point are also used as feature points for later identification and matching.

[0058] Step 5: The camera moves on the mobile shooting device, takes pictures after each fixed distance to obtain images in different directions, and then collects the directional images of all points to obtain a directional image set. The camera moves along the sliding frame at the center of the camera mobile shooting device, first moves to the upper part of the vertical slide rail 3, pauses after each fixed position, and then notifies the camera to shoot, then records the specific position of the shot image, then moves up to a fixed position, and then shoots again, until it moves to the top, returns to the initial point, moves down to shoot, and then moves left and right to shoot, and collects all the shot directional images to obtain a directional image set. Figure 2 As shown in the figure, the camera moves 10cm each time and takes a photo. When the first photo is at the center, the fruit is on the back of the leaves and is blocked. The fruit cannot be seen when the camera is shot from the front. However, after the camera moves upward, the fruit begins to slowly reveal itself. When the camera moves 20cm upward, the exposed fruit is more visible. When the three images are merged together, the position of the fruit can be seen. However, the second and third photos need to be stretched. Figure 1 By deleting the blocked position in the image, you can see the image of the naked fruit, and remove the blocked fruit leaves to achieve more accurate recognition. At the same time, the images taken at different angles have different directions of reflection. When the image in one direction cannot be recognized, the image in another direction can be recognized.

[0059] Step 6: Fit each image in the directional image set with the two-dimensional coordinate plane according to the identification point to obtain several layers of image sets, and then identify the fruit position coordinates of the nut green skin fruit in each layer of image. The positions of the identification points of several layers of image sets are all overlapped, and then the arc-shaped contour of the nut green skin fruit in each image is identified as the fruit position of the nut green skin fruit, and the detailed coordinate information of each nut green skin fruit position is displayed.

[0060] Step 7: The fruit position coordinates of all images are aggregated into one coordinate plane, and the position coordinates of each fruit remain unchanged, so as to realize the identification and positioning of green-skinned nuts. The specific process of aggregating the fruit position coordinates into one coordinate plane is to fuse several layers of images together, and then superimpose the same or adjacent fruit position coordinates. If there are fruits in some images and no fruits in other images, the leaves of the images without fruits are blocked and erased to show the positions of fruits in all images, and the positions of green-skinned nuts are obtained.

[0061] A method for automatically picking green-skinned nuts based on image recognition. The method for identifying and locating green-skinned nuts according to any one of claims 1 to 6 is used to locate nuts. A temporary correction camera for picking is then installed at the front end of a manipulator of a picking trolley. According to the position of each green-skinned nut, a control device calculates the rotational freedom required by the manipulator to pick the nut. The manipulator arrives in front of the green-skinned nut and the temporary correction camera collects the best picking position in real time. The manipulator then picks the nut. When the manipulator picks the nut and leaves, the temporary correction camera collects whether there are any unpicked nuts at the place where the nut has just been picked. If not, the nut fruit position plane is deleted. If so, the manipulator continues to be controlled to pick the nut a second time until the nut is picked and the fruit coordinates in the fruit position plane are deleted. The picking trolley then moves to the other side to continue picking the nuts. Each nut tree can be picked completely by picking on all four sides.

[0062] In the embodiment of the present invention, the specific process of the control device calculating the required rotational freedom of the manipulator is as follows: Figure 4 As shown in the figure, H, L, and P are set to represent the height from the ground to the chassis of the robot arm, the length of the long arm, and the length of the short arm, respectively. m is the center height of the nut to be picked. The angle between the long arm and the short arm remains unchanged at 90 degrees. The long arm rotates to make the center of the camera and the center of the target collinear. After collinearity, the current base rotation angle θ and the long arm rotation angle θ1 are returned. Using the trigonometric function theorem and the Pythagorean theorem, the imaging distance measurement length S can be obtained. The specific derivation formulas are shown in formulas (1), (2), (3), (4), and (5):

[0063] T=P·tanθ1+L (1)

[0064]

[0065] After obtaining S, S is vector-decomposed by rotating the angle θ, and the sine and cosine function calculation formula is used to obtain the x-axis coordinate X, y-axis coordinate Y, and z-axis coordinate Z with the robot itself as the base coordinate. The specific formulas are shown in equations (6), (7), and (8):

[0066] Z=2·m (6)

[0067]

[0068] After the camera resolves the relative real coordinates of the target nut picking and the manipulator, the inverse solution algorithm is used to solve the servo rotation angle of each degree of freedom of the manipulator. A three-degree-of-freedom manipulator is used, and the trigonometric function geometry method is used to solve the inverse kinematics formula of the manipulator. The real coordinates of the target nut are known to be (X, Y, Z), the long arm of the manipulator is L1, the short arm is L2, and the gripper connection part is L3. The rotation angles of the three axes that need to be solved are

[0069] like Figure 5 As shown, from the inverse trigonometric function, we can know that in a right triangle, The formula is shown in formula (9).

[0070]

[0071] like Figure 6 As shown, the robot arm geometric model is established based on the robot arm YOZ plane. The cosine theorem in trigonometric function can be used to obtain equations (10), (11), and (12):

[0072]

[0073]

[0074] Using the Pythagorean theorem of a right triangle, we can get the expressions of T1 and T2 as shown in equations (13) and (14).

[0075]

[0076] Using the arc cosine formula in trigonometric functions, we can find The specific expressions are shown in formula (15), formula (16), formula (17) and formula (18):

[0077]

[0078] The rotation angles of the three axes are Finally, control the robot to perform picking actions.

[0079] An automatic picking device for identifying and locating green-skinned nuts based on image recognition, such as Figure 7As shown, it includes a picking vehicle body, a mobile shooting device, an identification and positioning camera, a picking robot, a picking correction camera and a data analysis and control device. The mobile shooting device, the picking robot and the data analysis and control device are all arranged on the picking vehicle body, the mobile shooting device is arranged at the front end, the identification and positioning camera is arranged on the mobile shooting device and is slidably arranged, and the picking vehicle body, the identification and positioning camera, the picking robot and the picking correction camera are all connected to the data analysis and control device. The picking vehicle body is mainly used for moving between mountains, and at the same time carries the mobile shooting device, the identification and positioning camera, the picking robot, the picking correction camera and the data analysis and control device to move. A fruit storage frame is also arranged on the picking vehicle body to store the picked nuts. The mobile shooting device is as shown in FIG. Figure 3 The structure shown. The recognition and positioning camera uses an OpenMV camera, and the data analysis and control device mainly completes the contour recognition of the leaves and fruits inside the image, and controls the picking robot by issuing instructions. The picking correction camera realizes the positioning of the clamping point during the picking process, and realizes the secondary recognition of the clamping position after clamping to avoid missing picking.

[0080] An application of a method for identifying and locating green-skinned nuts based on image recognition, the identification and locating method is applied to the automatic picking and identification and locating of oranges, the automatic picking and identification and locating of walnuts, the automatic picking and identification and locating of snow pears, or the automatic picking and identification and locating of apples. In addition to the automatic picking of the above-mentioned fruits, the identification and locating method of this application can be used for other fruits that can be automatically picked.

[0081] Matters not covered by the present invention are known technologies.

[0082] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying and locating green-skinned nuts based on image recognition, characterized in that: The method comprises the following steps: Step 1: Install a camera mobile shooting device on the nut and green fruit picking cart; Step 2: Drive the nut green-skin fruit picking cart to the front of the nut tree, and the camera shoots the nut tree at the initial position of the mobile shooting device to obtain an initial image; Step 3: Perform object contour recognition on the initial image to obtain the position points of the nut green-skinned fruit at a number of initial recognized points, and mark a number of points as recognition points; Step 4: Construct a two-dimensional coordinate plane, paste the initial image on the two-dimensional coordinate plane, obtain the coordinate position of the nut green peel fruit and the coordinates of the identification points, and set the center point; Step 5: The camera moves on the mobile shooting device, takes pictures after each fixed distance of movement to obtain images in different directions, and then summarizes the direction images of all points to obtain a direction image set; Step 6: Fit each image in the directional image set to the two-dimensional coordinate plane according to the identification point to obtain several layers of image sets, and then identify the fruit position coordinates of the green-skinned nuts in each layer of the image; Step 7: The fruit position coordinates of all images are aggregated into a coordinate plane, and the position coordinates of each fruit remain unchanged, so as to realize the recognition and positioning of green-skinned nuts.

2. The method for identifying and locating green-skinned nuts based on image recognition according to claim 1, characterized in that: In step 1, the camera moving shooting device includes a camera, a sliding chassis and a sliding frame, the camera is arranged on the sliding chassis, the sliding chassis is clamped in the sliding frame and can be movably arranged, the sliding frame includes a vertical slide rail (3) and a horizontal slide rail (4), the vertical slide rail (3) and the horizontal slide rail (4) are arranged vertically and crosswise, an initial positioning point (5) is arranged inside the intersection, an initial positioning point sensing switch is arranged at the bottom of the sliding chassis, when the initial positioning point (5) and the initial positioning point sensing switch are on the same straight line, it means that the camera is in the initial position, the sliding chassis is connected to an external control device, moves and pauses according to the signal of the control device, the camera is connected to the control device, and transmits the collected picture to the control device for processing, and the vertical slide rail (3) and the horizontal slide rail (4) are arranged as a retractable structure, which is extended or shortened according to the size of the nut tree to be picked.

3. The method for identifying and locating green-skinned nuts based on image recognition according to claim 1, characterized in that: In step 2, the nut green-skinned fruit picking cart comes to the nut tree, and the camera identifies the external wheel library of the nut tree image, identifies the height and width of the nut tree, and then moves the camera mobile shooting device up and down so that the center position of the camera mobile shooting device is aligned with the center position of the nut tree. The camera then moves to the initial positioning point (5) of the mobile shooting device, and then shoots the nut tree to obtain the initial image of the nut tree.

4. The method for identifying and locating green-skinned nuts based on image recognition according to claim 1, characterized in that: In step 3, the initial image is initialized, and then the contours of the objects inside the image are recognized. The contour recognition includes the leaf contour, the circular contour of the nut green-skinned fruit and the branch contour. When a circular contour is recognized, it is considered to be the fruit of the nut green-skinned fruit, and the positions of several points of the nut green-skinned fruit are obtained. At the same time, several points are selected at the edge and midline of the nut tree in the initial image as identification points. The identification points are used for position coordinate correction during the later highlighting and fitting. Due to changes in the shooting angle, image position stretching or shrinkage correction is required after shooting. The recognition of all images is overlapped during fitting.

5. The method for identifying and locating green-skinned nuts based on image recognition according to claim 1, characterized in that: In step 5, the camera moves along the sliding frame at the center of the camera moving shooting device, first moves to the upper part of the vertical slide rail (3), pauses after each fixed position, and then notifies the camera to shoot, and then records the specific position of the shot image, and then moves upward to a fixed position, and then shoots again, until it moves to the top, returns to the initial point, moves downward to shoot, and then moves left and right to shoot, and all the shot directional images are collected to obtain a directional image set.

6. The method for identifying and locating green-skinned nuts based on image recognition according to claim 1, characterized in that: In step 6, the positions of the identification points of the image sets of several layers are all overlapped, and then the arc-shaped contour of the nut green-skin fruit in each image is identified as the fruit position of the nut green-skin fruit, and the detailed coordinate information of the nut green-skin fruit position at each location is displayed; In step 7, the specific process of summarizing the fruit position coordinates into a coordinate plane is to fuse several layers of images together, and then superimpose the same or adjacent fruit position coordinates. If there are fruits in some images and no fruits in other images, the leaves of the images without fruits are blocked and erased to display the positions of fruits in all images, and the positions of the green-skinned fruits of the nuts are obtained.

7. An automatic picking control method for green-skinned nuts based on image recognition, characterized in that: The nut green-skinned fruit identification and positioning method of any one of claims 1-6 is used to locate the nuts, and then a temporary correction camera for picking is installed at the front end of the manipulator of the picking cart. According to the position of each green-skinned nut, the control device calculates the rotational freedom required by the manipulator to pick the nut. The manipulator comes to the green-skinned nut, and the temporary correction camera collects the best picking position in real time, and then the manipulator picks the nut. When the manipulator picks the nut and leaves, the temporary correction camera collects whether there are any unpicked nuts at the place where they have just been picked. If not, the nut fruit position plane is deleted. If so, the manipulator continues to be controlled to pick the nut a second time until the nut is picked and the fruit coordinates in the fruit position plane are deleted. Then the picking cart moves to the other side to continue picking the nuts. Each nut tree can be picked by picking on all four sides.

8. The method for automatically picking green-skinned nuts based on image recognition according to claim 7, characterized in that: The specific process of calculating the required rotational freedom of the manipulator by the control device is as follows: H, L, and P are set to represent the height from the ground to the chassis of the manipulator, the length of the long arm, and the length of the short arm, respectively; m is the center height of the nut to be picked; the angle between the long arm and the short arm remains unchanged at 90 degrees; the long arm rotates to make the center of the camera collinear with the center of the target; after collinearity, the current base rotation angle θ and long arm rotation angle θ1 are returned; using trigonometric function theorem and the Pythagorean theorem, the imaging distance measurement length S can be obtained; the specific derivation formulas are shown in equations (1), (2), (3), (4), and (5): T=P·tanθ1+L (1) After obtaining S, S is vector-decomposed by rotating the angle θ, and the sine and cosine function calculation formula is used to obtain the x-axis coordinate X, y-axis coordinate Y, and z-axis coordinate Z with the robot itself as the base coordinate. The specific formulas are shown in equations (6), (7), and (8): Z=2·m (6) After the camera resolves the relative real coordinates of the target nut picking and the manipulator, the inverse solution algorithm is used to solve the servo rotation angle of each degree of freedom of the manipulator. A three-degree-of-freedom manipulator is used, and the trigonometric function geometry method is used to solve the inverse kinematics formula of the manipulator. The real coordinates of the target nut are known to be (X, Y, Z), the long arm of the manipulator is L1, the short arm is L2, and the gripper connection part is L3. The rotation angles of the three axes that need to be solved are From the inverse trigonometric function, we know that in a right triangle, The formula is shown in formula (9). The geometric model of the robot arm is established on the YOZ plane of the robot arm. The cosine theorem in trigonometric functions can be used to obtain equations (10), (11), and (12): Using the Pythagorean theorem of a right triangle, we can get the expressions of T1 and T2 as shown in equations (13) and (14): Using the arc cosine formula in trigonometric functions, we can find From their respective angles, the specific expressions are shown in formula (15), formula (16), formula (17), and formula (18): The rotation angles of the three axes are After that, control the robot to perform picking action.

9. An automatic picking device for identifying and locating green-skinned nuts based on image recognition, characterized in that: It includes a picking vehicle body, a mobile shooting device, an identification and positioning camera, a picking robot, a picking correction camera and a data analysis control device. The mobile shooting device, the picking robot and the data analysis control device are all arranged on the picking vehicle body, the mobile shooting device is arranged at the front end, the identification and positioning camera is arranged on the mobile shooting device in a sliding setting, and the picking vehicle body, the identification and positioning camera, the picking robot and the picking correction camera are all connected with the data analysis control device.

10. An application of a method for identifying and locating green-skinned nuts based on image recognition, characterized in that: The identification and positioning method is applied to the automatic picking and identification and positioning of oranges, the automatic picking and identification and positioning of walnuts, the automatic picking and identification and positioning of snow pears, or the automatic picking and identification and positioning of apples.

Images